SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 1.5: Case  $N(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ ,  $f(\mathbf{x}) = \mathrm{pos}(\mathbf{r}_*^\top \mathbf{x} + c_*)$ . We show that (1) also holds in this case, and therefore we use a similar approach of empirically estimating the pair and bag covariance matrices solving (1) works in principle. However, there are complications, in particular the presence of  $\boldsymbol{\mu}$  and  $c_*$  degrades the error bounds in the analysis, thus increasing the sample complexity of the algorithm. This is because the measures of  $\{\mathbf{x} \mid f(\mathbf{x}) = a\}$  for  $a = \{0,1\}$  could be highly skewed if  $\| \boldsymbol{\mu} \|_2$  and/or  $|c_*|$  is large. Moreover, the spectral algorithm only gives a solution  $\pm \hat{\mathbf{r}}$  for  $\mathbf{r}_*$ . An additional step is required to obtain an estimate of  $c_*$ . This we accomplish using the following procedure which, given a sample of  $s$  bags and any  $\mathbf{r}$  outputs a  $\hat{c}$  which has the following property: if  $s^* = \max_c \{ \text{no. of bags satisfied by } \mathrm{pos}(\mathbf{r}^\top \mathbf{x} + c) \}$ , then  $\hat{c}$  will satisfy at least  $s^* - 1$  bags. This is done by ordering the values  $\mathbf{r}^\top \mathbf{x}$  of the vectors  $\mathbf{x}$  within each bag in decreasing order, and then constructing set of the  $k$ th values of each bag. Out of these  $s$  values, the one which takes as  $c$  in  $\mathrm{pos}(\mathbf{r}^\top \mathbf{x} + c)$  satisfies the most bags, is chosen to be  $\hat{c}$ .
Due to the non-homogeneity of the Gaussian distribution and the LTF  $f$ , the application of Lemma 2.3 instead incurs an  $O\left(q^{4}((\sqrt{\lambda_{\mathrm{max}}} + \| \boldsymbol{\mu}\|_{2}) / \sqrt{\lambda_{\mathrm{min}}})^{4}\right)$  factor in the sample complexity, while a factor of  $O(\ell^2)$  is towards the estimation of  $\Sigma_B$  and  $\Sigma_D$ . Note that  $\ell$  is the distance from  $\boldsymbol{\mu}$  to the hyperplane of  $f$  normalized by the stretch induced by  $\boldsymbol{\Sigma}$ , and thus a larger value of  $\ell$  implies a lower density near the hyperplane leading to an increased sample complexity. Lastly, a further blowup by  $1 / (\Phi (\ell)(1 - \Phi (\ell)))^2$  comes from bounding the sample error from geometric bound between  $\hat{\mathbf{r}}$  and  $\mathbf{r}_{*}$ , and is required for a sufficiently accurate approximation of  $c_*$ .
Generalization Error Bounds. We prove (Thm. 2.2) bounds on the generalization of the error of a hypothesis LTF  $h$  in satisfying sampled bags to its distributional instance-level error. Using this, we are able to distinguish (for  $k \neq q / 2$ ) between the two possible solutions our principal component algorithm yields – the one which satisfies more of the sampled bags has w.h.p. low instance-level error. For proving these bounds, the first step is to use a bag-level generalization error bound shown by [26] using the techniques of [35]. Next, we show that low distributional bag satisfaction error by  $h$  implies low instance level error. This involves a fairly combinatorial analysis of two independent binomial random variables formed from the incorrectly classified labels within a random bag. Essentially, unless  $h$  closely aligns with  $f$  at the instance level, with significant probability there will be an imbalance in these two random variables leading to  $h$  not satisfying the bag.
Subgaussian concentration bounds. The standard estimation bounds for Gaussians are not directly applicable in our case, since the random vector sampled from a random bag is biased according to its label given by  $f$ , and is therefore not a Gaussian vector. To obtain sample complexity bounds linear in  $\log(1/\delta)$  we use subgaussian concentration bounds for mean and covariance estimation ([32, 31]). For this, we show  $O(\ell)$  bound on the expectation and subgaussian norm of the thresholded Gaussian given by  $\{g \sim N(0,1) \mid g > \ell\}$  for some  $\ell > 0$ . The random vectors of interest to us are (in a transformed space) distributed as a combination of thresholded Gaussians in one of the coordinates, and  $N(0,1)$  in the rest. We show that they satisfy the  $O(\ell)$  bound on their subgaussian norm and admit the corresponding subgaussian Hoeffding (for empirical mean) and empirical covariance concentration bounds. Based on this, in Sec. 3 we abstract out the procedure used in our learning algorithms for obtaining the relevant mean and covariance estimates.
Experiments. We include in Sec. 6 an experimental evaluation of our learning algorithms along with a comparison of with those of [25, 26] and random LTFs, demonstrating the effectiveness of our techniques.
We begin with some useful linear algebraic notions. Let  $\lambda_{\mathrm{max}}(\mathbf{A})$  and  $\lambda_{\mathrm{min}}(\mathbf{A})$  denote the maximum and minimum eigenvalue of a real symmetric matrix  $\mathbf{A}$ . The operator norm  $\| \mathbf{A} \|_2 \coloneqq \max_{\| \mathbf{x} \|_2 = 1} \| \mathbf{A} \mathbf{x} \|_2$  for such matrices is given by  $\lambda_{\mathrm{max}}(\mathbf{A})$ .
We shall restrict our attention to symmetric positive definite (p.d.) matrices  $\mathbf{A}$  which satisfy  $\mathbf{x}^{\mathrm{T}}\mathbf{A}\mathbf{x} > 0$  for all non-zero vectors  $\mathbf{x}$ , implying that  $\lambda_{\min}(\mathbf{A}) > 0$  and  $\mathbf{A}^{-1}$  exists and is symmetric p.d. as well. Further, for such matrices  $\mathbf{A}$ ,  $\mathbf{A}^{1/2}$  is well defined to be the unique symmetric p.d. matrix  $\mathbf{B}$  satisfying  $\mathbf{BB} = \mathbf{A}$ . The eigenvalues of  $\mathbf{A}^{1/2}$  are the square-roots of those of  $\mathbf{A}$ . We have the following lemma which is proved in Appendix B.6.
Lemma 2.1. Let  $\mathbf{A}$  and  $\mathbf{B}$  be symmetric p.d. matrices such that  $\| \mathbf{A} - \mathbf{B}\| \leq \varepsilon_1\| \mathbf{A}\| _2$ . Let  $\mathbf{r}_1,\mathbf{r}_2\in \mathbb{R}^d$  be two unit vectors such that  $\| \mathbf{r}_1 - \mathbf{r}_2\| _2\leq \varepsilon_2$ . Then,  $\left\| \frac{\mathbf{Ar}_1}{\|\mathbf{Ar}_1\|_2} -\frac{\mathbf{Br}_2}{\|\mathbf{Br}_2\|_2}\right\| _2\leq 4\frac{\lambda_{\max}(\mathbf{A})}{\lambda_{\min}(\mathbf{A})} (\varepsilon_2 + \varepsilon_1)$  when  $\frac{\lambda_{\max}(\mathbf{A})}{\lambda_{\min}(\mathbf{A})} (\varepsilon_2 + \varepsilon_1)\leq \frac{1}{2}$ .
Bag Oracle and related statistics. Let  $\mathcal{O} \coloneqq \mathsf{Ex}(f, \mathcal{D}, q, k)$  be any bag oracle with  $k \in \{1, \dots, q-1\}$  for an LTF  $f(\mathbf{x}) \coloneqq \mathbf{r}_*^{\mathrm{T}}\mathbf{x} + c_*$  in  $d$ -dimensions, and let  $\mathcal{M}$  be a collection of  $m$  bags sampled iid from the oracle. Define for any hypothesis LTF  $h$ ,
$$
\operatorname {B a g E r r} _ {\text {o r a c l e}} (h, f, \mathcal {D}, q, k) := \Pr_ {B \leftarrow 0} [ \operatorname {A v g} \{h (\mathbf {x}) \mid \mathbf {x} \in B \} \neq k / q ], \text {a n d ,} \tag {2}
$$
$$
\operatorname {B a g E r r} _ {\text {s a m p l e}} (h, \mathcal {M}) := | \{B \in \mathcal {M} \mid \operatorname {A v g} \{h (\mathbf {x}) \mid \mathbf {x} \in B \} \neq k / q \} | / m. \tag {3}
$$
We define the following statistical quantities related to  $\mathcal{O}$ . Let  $\mathbf{X}$  be a random feature-vector sampled uniformly from a random bag sampled from  $\mathcal{O}$ . Let,
$$
\boldsymbol {\mu} _ {B} := \mathbb {E} [ \mathbf {X} ] \quad \text {a n d ,} \quad \boldsymbol {\Sigma} _ {B} := \mathbb {E} \left[ \left(\mathbf {X} - \boldsymbol {\mu} _ {B}\right) \left(\mathbf {X} - \boldsymbol {\mu} _ {B}\right) ^ {\mathsf {T}} \right] = \operatorname {V a r} [ \mathbf {X} ]. \tag {4}
$$
Now, let  $\mathbf{Z} = \mathbf{X}_1 - \mathbf{X}_2$  where  $(\mathbf{X}_1,\mathbf{X}_2)$  are a random pair of feature-vectors sampled (without replacement) from a random bag sampled from  $\mathcal{O}$ . Clearly  $\mathbb{E}[\mathbf{Z}] = \mathbf{0}$ . Define
$$
\boldsymbol {\Sigma} _ {D} := \mathbb {E} \left[ \mathbf {Z} \mathbf {Z} ^ {\mathrm {T}} \right] = \operatorname {V a r} [ \mathbf {Z} ]. \tag {5}
$$
Generalization and stability bounds. We prove in Appendix D.1 the following generalization bound from bag classification error to instance classification error.
Theorem 2.2. For any  $\varepsilon < 1 / 4q$  if  $\mathrm{BagErr}_{\mathrm{sample}}(h,\mathcal{M})\leq \varepsilon$  then,
(i) if  $k \neq q/2$ ,  $\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h(\mathbf{x})] \leq 4\varepsilon$ , and
(ii) if  $k = q / 2$ ,  $\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h(\mathbf{x})] \leq 4\varepsilon$  or  $\operatorname{Pr}[f(\mathbf{x}) \neq (1 - h(\mathbf{x}))] \leq 4\varepsilon$ ,
w.p.  $1 - \delta$ , when  $m \geq C_0 d(\log q + \log (1 / \delta)) / \varepsilon^2$ , for any  $\delta > 0$  and absolute constant  $C_0 > 0$ .
In some cases we directly obtain geometric bounds on the hypothesis classifier and the following lemma (proved in Appendix D.2) allows us to straightaway bound the classification error.
Lemma 2.3. Suppose  $\| \mathbf{r} - \hat{\mathbf{r}} \|_2 \leq \varepsilon_1$  for unit vectors  $\mathbf{r}, \hat{\mathbf{r}}$ . Then,  $\operatorname*{Pr}\left[\operatorname{pos}\left(\mathbf{r}^T \mathbf{X} + c\right) \neq \operatorname{pos}\left(\hat{\mathbf{r}}^T \mathbf{X} + c\right)\right] \leq \varepsilon$  where  $\varepsilon = \varepsilon_1 (c_0 \sqrt{\lambda_{\max} / \lambda_{\min}} + c_1 \| \boldsymbol{\mu} \|_2 / \sqrt{\lambda_{\min}})$  for some absolute constants  $c_0, c_1 > 0$  and  $\lambda_{\max}, \lambda_{\min}$  are the maximum and minimum eigenvalues of  $\boldsymbol{\Sigma}$  respectively.
3 Bag distribution statistics estimation
We provide the following estimator for  $\pmb{\mu}_B$ ,  $\pmb{\Sigma}_B$  and  $\pmb{\Sigma}_D$  defined in (4) and (5). We have the following lemma - which follows from the subgaussian distribution based mean and covariance concentration bounds shown for thresholded Gaussians (see Appendix E) - whose proof is given in Appendix E.3.
Lemma 3.1. If  $m \geq O\left((d / \varepsilon^2)O(\ell^2)\log (d / \delta)\right)$  where  $\ell$  is as given in Lemma E.13 then Algorithm 1 returns  $\hat{\pmb{\mu}}_B, \hat{\pmb{\Sigma}}_B, \hat{\pmb{\Sigma}}_D$  such that  $\| \hat{\pmb{\mu}}_B - \pmb {\mu}_B\| _2\leq \varepsilon \sqrt{\lambda_{\max}} /2$ ,  $\| \hat{\pmb{\Sigma}}_B - \pmb {\Sigma}_B\| _2\leq \varepsilon \lambda_{\max}$ , and  $\| \hat{\pmb{\Sigma}}_D - \pmb {\Sigma}_D\| _2\leq \varepsilon \lambda_{\max}$ , w.p. at least  $1 - \delta$ , for any  $\varepsilon, \delta >0$ . Here  $\lambda_{\mathrm{max}}$  is the maximum eigenvalue of  $\pmb{\Sigma}$ .
Algorithm 1 MeanCovsEstimator.
Input:  $\mathsf{Ex}(f,\mathcal{D} = N(\pmb {\mu},\pmb {\Sigma}),q,k),m$  , where  $f = \mathrm{pos}\left(\mathbf{r}^{\top}\mathbf{X} + c\right)$
1. Sample  $m$  bags from  $\mathsf{E}\times (f,\mathcal{D},q,k)$ . Let  $\{B_i\}_{i = 1}^m$  be the sampled bags.
2.  $V\coloneqq \{\mathbf{x}_i\mid \mathbf{x}_i$  u.a.r.  $\leftarrow B_i,i\in \{1,\ldots ,m\} \}$
3.  $\hat{\mu}_B = \sum_{\mathbf{x}\in V}\mathbf{x} / m|$
4.  $\hat{\Sigma}_B = \Sigma_{\mathbf{x}\in V}(\mathbf{x} - \boldsymbol {\mu}_B)(\mathbf{x} - \boldsymbol {\mu}_B)^\top /m.$
5. Sample  $m$  bags from  $\mathsf{Ex}(f, \mathcal{D}, q, k)$ . Let  $\{\tilde{B}_i\}_{i=1}^m$  be the sampled bags.
6.  $\tilde{V} := \{\overline{\mathbf{x}}_i = \mathbf{x}_i - \tilde{\mathbf{x}}_i \mid (\mathbf{x}_i, \tilde{\mathbf{x}}_i) \text{ a.a.r. without replacement from } \bar{B}_i, i \in \{1, \dots, m\}\}$ .
7.  $\hat{\mathbf{S}}_D = \Sigma_{\mathbf{z}\in \tilde{V}}\mathbf{z}\mathbf{z}^{\mathsf{T}} / m.$
8. Return:  $\hat{\mu}_B, \hat{\Sigma}_B, \hat{\Sigma}_D$ .
4 Proof of Theorem 1.4
For the setting of Theorem 1.4, we provide Algorithm 2. It uses as a subroutine a polynomial time procedure PrincipalEigenVector for the principal eigen-vector of a symmetric matrix, and first computes two LTFs given by a normal vector and its negation, returning the one that has lower error on a sampled collection of bags.
Algorithm 2 PAC Learner for no-offset LTFs over  $N(\mathbf{0},\Sigma)$
Input:  $\mathcal{O} = \mathsf{Ex}(f,\mathcal{D} = N(\mathbf{0},\Sigma),q,k)$ $m,s$  where  $f(\mathbf{x}) = \mathrm{pos}\left(\mathbf{r}_*\mathsf{T}\mathbf{x}\right)$ $\| \mathbf{r}_{*}\|_{2} = 1$
1. Compute  $\hat{\Sigma}_B, \hat{\Sigma}_D$  using MeanCovsEstimator with  $m$  samples.
2.  $\overline{\mathbf{r}} = \hat{\Sigma}_B^{-1 / 2}$  PrincipalEigenVector(  $\hat{\Sigma}_B^{-1 / 2}\hat{\Sigma}_D\hat{\Sigma}_B^{-1 / 2})$  if  $\hat{\Sigma}_B^{-1 / 2}$  exists, else exit.
3. Let  $\hat{\mathbf{r}} = \overline{\mathbf{r}} / \| \overline{\mathbf{r}} \|_2$ . 4. If  $k = q/2$  return:  $h = \text{pos}(\hat{\mathbf{r}}^{\top}\mathbf{X})$ , else
a. Let  $\tilde{h} = \mathrm{pos}\left(-\hat{\mathbf{r}}^{\mathsf{T}}\mathbf{X}\right)$
b. Sample a collection  $\hat{\mathcal{M}}$  of  $s$  bags from  $\mathcal{O}$ .
c. Return  $h^* \in \{h, \bar{h}\}$  which has lower BagErrsample  $(h^*, \mathcal{M})$ .
Lemma 4.1. For any  $\varepsilon, \delta \in (0,1)$ , if  $m \geq O((d / \varepsilon^4) \log (d / \delta)(\lambda_{\max} / \lambda_{\min})^4 q^4)$ , then  $\hat{\mathbf{r}}$  computed in Step 3 of Alg. 2 satisfies  $\min \{\|\hat{\mathbf{r}} - \mathbf{r}_*\|_2, \|\hat{\mathbf{r}} + \mathbf{r}_*\|_2\} \leq \varepsilon$ , w.p.  $1 - \delta / 2$ .
The above, whose proof is deferred to Sec. 4.1, is used in conjunction with the following lemma.
Lemma 4.2. Let  $k \neq q/2$ ,  $\varepsilon, \delta \in (0,1)$  and suppose  $\hat{\mathbf{r}}$  computed in Step 3 of Alg. 2 satisfies  $\min \{ \| \hat{\mathbf{r}} - \mathbf{r}_* \|_2, \| \hat{\mathbf{r}} + \mathbf{r}_* \|_2 \} \leq \varepsilon$ . Then, with  $s \geq O\left(d(\log q + \log(1/\delta)) / \varepsilon^2\right)$ ,  $h^*$  in Step 3.c satisfies  $\operatorname{Pr}_{\mathbb{D}}[h^*(\mathbf{x}) \neq f(\mathbf{x})] \leq 16c_0q\varepsilon \sqrt{\frac{\lambda_{\max}}{\lambda_{\min}}} w.p. 1 - \delta/2$ , where constant  $c_0 > 0$  is from Lem. 2.3.
With the above we complete the proof of Theorem 1.4 as follows.
Proof. (of Theorem 1.4) Let the parameters  $\delta, \varepsilon$  be as given in the statement of the theorem.
For  $k = q / 2$ , we use  $O(\varepsilon \sqrt{\lambda_{\min} / \lambda_{\max}})$  for the error bound in Lemma 4.1 thereby taking  $m = O((d / \varepsilon^4)\log (d / \delta)(\lambda_{\max} / \lambda_{\min})^6 q^4)$  in Alg. 2, so that Lemma 4.1 along with Lemma 2.3 yields the desired misclassification error bound of  $\varepsilon$  for one of  $h, \tilde{h}$ .
For  $k \neq q/2$ , we use  $O(\varepsilon \sqrt{\lambda_{\min} / \lambda_{\max}} / q)$  for the error bound in Lemma 4.1. Taking  $m = O((d / \varepsilon^4)\log (d / \delta)(\lambda_{\max} / \lambda_{\min})^6 q^8)$  in Alg. 2 we obtain the following bound:  $\min \{ \| \hat{\mathbf{r}} - \mathbf{r}_* \|_2, \| \hat{\mathbf{r}} + \mathbf{r}_* \|_2 \} \leq \varepsilon \sqrt{\lambda_{\min} / \lambda_{\max}} / (16c_0q)$  with probability  $1 - \delta / 2$ . Using  $s \geq O\left(d(\log q + \log (1 / \delta))q^{\frac{2}{\varepsilon^2}\frac{\lambda_{\max}}{\lambda_{\min}}}\right)$ , Lemma 4.2 yields the desired misclassification error bound of  $\varepsilon$  on  $h^*$  w.p.  $1 - \delta$ .
Proof. (of Lemma 4.2) Applying Lemma 2.3 we obtain that at least one of  $h$ ,  $\tilde{h}$  has an instance misclassification error of at most  $O(\varepsilon \sqrt{\lambda_{\max} / \lambda_{\min}})$ . WLOG assume that  $h$  satisfies this error bound i.e.,  $\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h(\mathbf{x})] \leq c_0 \varepsilon \sqrt{\lambda_{\max} / \lambda_{\min}} =: \varepsilon'$ . Since the separating hyperplane of the LTF  $f$
passes through the origin, and  $\mathcal{D} = N(\mathbf{0},\boldsymbol {\Sigma})$  is centered,  $\operatorname *{Pr}_{\mathcal{D}}[f(\mathbf{x}) = 1] = \operatorname *{Pr}_{\mathcal{D}}[f(\mathbf{x}) = 0] = 1 / 2.$  Thus,
$$
\operatorname * {P r} _ {\mathcal {D}} [ h (x) \neq f (x) \mid f (\mathbf {x}) = 1 ], \operatorname * {P r} _ {\mathcal {D}} [ [ h (x) \neq f (x) \mid f (\mathbf {x}) = 0 ] \leq 2 \varepsilon^ {\prime}.
$$
Therefore, the probability that a random bag from the oracle contains a feature vector on which  $f$  and  $h$  disagree is at most  $2q\varepsilon'$ . Applying the Chernoff bound (see Appendix B.1) we obtain that with probability at least  $1 - \delta / 6$ ,  $\mathsf{BagErr}_{\mathrm{sample}}(h, \mathcal{M}) \leq 4q\varepsilon'$ . Therefore, in Step 3.c.  $h^*$  satisfies  $\mathsf{BagErr}_{\mathrm{sample}}(h^*, \mathcal{M}) \leq 4q\varepsilon'$
On the other hand, applying Theorem 2.2, except with probability  $\delta /3$ ,  $\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x})\neq h^{*}(\mathbf{x})]\leq$ $16q\varepsilon^{\prime} = 16c_{0}q\varepsilon \sqrt{\lambda_{\max} / \lambda_{\min}}$ . Therefore, except with probability  $\delta /2$ , the bound in Lemma 4.2 holds.
4.1 Proof of Lemma 4.1
We define and bound a few useful quantities depending on  $k, q, \lambda_{\min}$  and  $\lambda_{\max}$  using  $1 \leq k \leq q - 1$ .
Definition 4.3. Define, (i)  $\kappa_{1} := \left(\frac{2k}{q} - 1\right)^{2}\frac{2}{\pi}$  so that  $0 \leq \kappa_{1} \leq 2/\pi$ , (ii)  $\kappa_{2} := \frac{1}{q-1}\frac{k}{q}\left(1 - \frac{k}{q}\right)\frac{16}{\pi}$  so that  $\frac{16}{\pi q^2} \leq \kappa_{2} \leq \frac{4}{\pi(q-1)}$ , (iii)  $\kappa_{3} := \frac{\kappa_{2}}{1-\kappa_{1}}$  so that  $\frac{16}{\pi q^2} \leq \kappa_{3} \leq \frac{4}{(\pi-2)(q-1)}$ , and (iv)  $\theta := \frac{2\lambda_{\max}}{\lambda_{\min}}\left(\frac{1}{2-\max(0,2\kappa_{1}-\kappa_{2})} + \frac{1}{1-\kappa_{1}}\right)$  so that  $\frac{3\lambda_{\max}}{\lambda_{\min}} \leq \theta \leq \frac{3\lambda_{\max}}{(1-2/\pi)\lambda_{\min}}$ .
For the analysis we begin by showing in the following lemma that  $\hat{\mathbf{r}}$  in the algorithms is indeed  $\pm \mathbf{r}_{*}$  if the covariance estimates were the actual covariances.
Lemma 4.4. The ratio  $\rho (\mathbf{r})\coloneqq \mathbf{r}^{\mathsf{T}}\pmb {\Sigma}_D\mathbf{r} / \mathbf{r}^{\mathsf{T}}\pmb {\Sigma}_B\mathbf{r}$  is maximized when  $\mathbf{r} = \pm \mathbf{r}_{*}$ . Moreover,
$$
\rho (\mathbf {r}) = 2 + \frac {\gamma (\mathbf {r}) ^ {2} \kappa_ {2}}{1 - \gamma (\mathbf {r}) ^ {2} \kappa_ {1}} \qquad w h e r e \qquad \gamma (\mathbf {r}) := \frac {\mathbf {r} ^ {\top} \boldsymbol {\Sigma} \mathbf {r} _ {*}}{\sqrt {\mathbf {r} ^ {\top} \boldsymbol {\Sigma} \mathbf {r}} \sqrt {\mathbf {r} _ {*} ^ {\top} \boldsymbol {\Sigma} \mathbf {r} _ {*}}} a n d
$$
$$
\mathbf {r} ^ {\mathsf {T}} \pmb {\Sigma} _ {B} \mathbf {r} = \mathbf {r} ^ {\mathsf {T}} \pmb {\Sigma} \mathbf {r} (1 - \gamma (\mathbf {r}) ^ {2} \kappa_ {1}), \quad \mathbf {r} ^ {\mathsf {T}} \pmb {\Sigma} _ {D} \mathbf {r} = \mathbf {r} ^ {\mathsf {T}} \pmb {\Sigma} \mathbf {r} (2 - \gamma (\mathbf {r}) ^ {2} (2 \kappa_ {1} - \kappa_ {2}))
$$
Proof. Let  $\Gamma \coloneqq \Sigma^{1/2}$ , then  $\mathbf{X} \sim N(\mathbf{0}, \mathbf{\Sigma}) \Leftrightarrow \mathbf{X} = \Gamma \mathbf{Z}$  where  $\mathbf{Z} \sim N(\mathbf{0}, \mathbf{I})$ . Further,  $\text{pos}(\mathbf{r}^{\top} \mathbf{X}) = \text{pos}(\mathbf{u}^{\top} \mathbf{Z})$  where  $\mathbf{u} = \Gamma \mathbf{r} / \| \Gamma \mathbf{r} \|_2$ . Using this, we can let  $\mathbf{X}_B = \Gamma \mathbf{Z}_B$  as a random feature-vector sampled uniformly from a random bag sampled from  $\mathcal{O}$ . Also, let  $\mathbf{X}_D = \Gamma \mathbf{Z}_D$  be the difference of two random feature vectors sampled uniformly without replacement from a random bag sampled from  $\mathcal{O}$ . Observe that the ratio  $\rho(\mathbf{r}) = \text{Var}[\mathbf{r}^{\top} \mathbf{X}_D] / \text{Var}[\mathbf{r}^{\top} \mathbf{X}_B] = \text{Var}[\mathbf{u}^{\top} \mathbf{Z}_D] / \text{Var}[\mathbf{u}^{\top} \mathbf{Z}_B]$ .
Let  $\mathbf{u}_* := \mathbf{\Gamma r}_* / \| \mathbf{\Gamma r}_* \|_2$ , and  $g^* := \mathbf{u}_*^\top \mathbf{Z}$  which is  $N(0,1)$ . For  $a \in \{0,1\}$ , let  $\mathbf{Z}_a$  be  $\mathbf{Z}$  conditioned on  $\text{pos}(\mathbf{u}_*^\top \mathbf{Z}) = a$ . Let  $g_a^* := \mathbf{u}_*^\top \mathbf{Z}_a$ ,  $a \in \{0,1\}$ , be the half normal distributions satisfying  $\mathbb{E}[(g_a^*)^2] = 1$  and  $\mathbb{E}[g_a^] = (-1)^{1 - a} \sqrt{2/\pi}$ . With this setup, letting  $g_B^* := \mathbf{u}_*^\top \mathbf{Z}_B$  and  $g_D^* := \mathbf{u}_*^\top \mathbf{Z}_D$  we obtain (using Lemma B.2 in Appendix B.2)
$$
\operatorname {V a r} \left[ g _ {B} ^ {*} \right] = 1 - \kappa_ {1}, \quad \operatorname {V a r} \left[ g _ {D} ^ {*} \right] = 2 \left(1 - \kappa_ {1}\right) + \kappa_ {2}
$$
Now let  $\tilde{\mathbf{u}}$  be a unit vector orthogonal to  $\mathbf{u}_{*}$ . Let  $\tilde{g} = \tilde{\mathbf{u}}^{\mathrm{T}}\mathbf{Z}$  be  $N(0,1)$ . Also, let  $\tilde{g}_a = \tilde{\mathbf{u}}^\mathrm{T}\mathbf{Z}_a$  for  $a\in \{0,1\}$ . Since  $\mathbf{Z}_a$  are given by conditioning  $\mathbf{Z}$  only along  $\mathbf{u}_{*}$ ,  $\tilde{g}_a\sim N(0,1)$  for  $a\in \{0,1\}$ . In particular, the component along  $\tilde{u}$  of  $\mathbf{Z}_B$  (call it  $\tilde{g}_B$ ) is  $N(0,1)$  and that of  $\mathbf{Z}_D$  (call it  $\tilde{g}_D$ ) is the difference of two iid  $N(0,1)$  variables. Thus,  $\operatorname {Var}[\tilde{g}_B] = 1$  and  $\operatorname {Var}[\tilde{g}_D] = 2$ . Moreover, due to orthogonality all these gaussian variables corresponding to  $\tilde{\mathbf{u}}$  are independent of those corresponding to  $\mathbf{u}_{*}$  defined earlier. Now let  $\mathbf{u} = \alpha \mathbf{u}_{*} + \beta \tilde{\mathbf{u}}$ , where  $\beta = \sqrt{1 - \alpha^2}$  be any unit vector. From the above we have,
$$
\begin{array}{l} \frac {\mathrm {V a r} [ \mathbf {u} ^ {\mathsf {T}} \mathbf {Z} _ {D} ]}{\mathrm {V a r} [ \mathbf {u} ^ {\mathsf {T}} \mathbf {Z} _ {B} ]} = \frac {\mathrm {V a r} [ \alpha g _ {D} ^ {*} + \beta \tilde {g} _ {D} ]}{\mathrm {V a r} [ \alpha g _ {B} ^ {*} + \beta \tilde {g} _ {B} ]} = \frac {\alpha^ {2} \mathrm {V a r} [ g _ {D} ^ {*} ] + \beta^ {2} \mathrm {V a r} [ \tilde {g} _ {D} ]}{\alpha^ {2} \mathrm {V a r} [ g _ {B} ^ {*} ] + \beta^ {2} \mathrm {V a r} [ \tilde {g} _ {B} ]} = \frac {2 \alpha^ {2} (1 - \kappa_ {1}) + \alpha^ {2} \kappa_ {2} + 2 \beta^ {2}}{\alpha^ {2} (1 - \kappa_ {1}) + \beta^ {2}} \\ = 2 + \frac {\alpha^ {2} \kappa_ {2}}{1 - \alpha^ {2} \kappa_ {1}} (6) \\ \end{array}
$$
where the last equality uses  $\beta = \sqrt{1 - \alpha^2}$ . Letting  $\mathbf{u} = \mathbf{\Gamma r} / \| \mathbf{\Gamma r}\| _2$  we obtain that  $\alpha = \frac{\langle\mathbf{\Gamma r},\mathbf{\Gamma r}_{*}\rangle}{\|\mathbf{\Gamma r}\|_{2}\|\mathbf{\Gamma r}_{*}\|_{2}} = \gamma (\mathbf{r})$  completing the proof.
Lemma 4.5.  $\underset{\|\mathbf{r}\|_2 = 1}{\mathrm{argmax}} \rho(\mathbf{r}) = \pmb{\Sigma}_B^{-1/2} \mathrm{PrincipalEigenvector}(\pmb{\Sigma}_B^{-1/2} \pmb{\Sigma}_D \pmb{\Sigma}_B^{-1/2})$
Proof. This follows directly from its generalization in Appendix B.5.  $\square$
We now complete the proof of Lemma 4.1 (with  $\delta$  instead of  $\delta /2$  for convenience). By Lemma 3.1, taking  $m\geq O\left((d / \varepsilon_1^2)\log (d / \delta)\right)$  ensures that  $\| \mathbf{E}_B\| _2\leq \varepsilon_1\lambda_{\max}$  and  $\| \mathbf{E}_D\| _2\leq \varepsilon_1\lambda_{\max}$  w.p. at least  $1 - \delta$  where  $\mathbf{E}_B = \hat{\pmb{\Sigma}}_B - \pmb {\Sigma}_B$  and  $\mathbf{E}_D = \hat{\pmb{\Sigma}}_D - \pmb {\Sigma}_D$ . We start by defining  $\rho (\hat{\mathbf{r}})\coloneqq \frac{\mathbf{r}^\top\hat{\pmb{\Sigma}}_D\mathbf{r}}{\mathbf{r}^\top\hat{\pmb{\Sigma}}_B\mathbf{r}}$  which is the equivalent of  $\rho$  using the estimated matrices. Observe that it can be written as  $\hat{\rho} (\mathbf{r}) = \frac{\mathbf{r}^\top\Sigma_B\mathbf{r} + \mathbf{r}^\top\mathbf{E}_B\mathbf{r}}{\mathbf{r}^\top\Sigma_D\mathbf{r} + \mathbf{r}^\top\mathbf{E}_D\mathbf{r}}$ . Using these we can obtain the following bound on  $\hat{\rho}$ : for any  $\mathbf{r}\in \mathbb{R}^d$ ,  $|\hat{\rho} (\mathbf{r}) - \rho (\mathbf{r})|\leq \theta \varepsilon_1|\rho (\mathbf{r})|$  w.p. at least  $1 - \delta (*)$  as long as  $\varepsilon_1\leq \frac{(1 - \kappa_1)}{2}\frac{\lambda_{\min}}{\lambda_{\max}}$ , which we shall ensure (see Appendix B.4).