SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For convenience we denote the normalized projection of any vector  $\mathbf{r}$  as  $\tilde{\mathbf{r}} := \frac{\pmb{\Sigma}^{1/2}\tilde{\mathbf{r}}}{\|\pmb{\Sigma}^{1/2}\tilde{\mathbf{r}}\|_2}$ . Now let  $\tilde{\mathbf{r}} \in \mathbb{R}^d$  be a unit vector such that  $\min \left\{\|\tilde{\mathbf{r}} - \tilde{\mathbf{r}}_*\|_2, \|\tilde{\mathbf{r}} + \tilde{\mathbf{r}}_*\|_2\right\} \geq \varepsilon_2$ . Hence, using the definitions from Lemma 4.4,  $|\gamma(\mathbf{r})| \leq 1 - \varepsilon_2^2/2$  while  $\gamma(\mathbf{r}_*) = 1$  which implies  $\rho(\mathbf{r}_*) - \rho(\mathbf{r}) \geq \kappa_3\varepsilon_2^2/2$ . Note that  $\rho(\mathbf{r}) \leq \rho(\mathbf{r}_*) = 2 + \kappa_3$ . Choosing  $\varepsilon_1 < \frac{\kappa_3}{4\theta(2 + \kappa_3)}\varepsilon_2^2$ , we obtain that  $\rho(\mathbf{r}_*)(1 - \theta\varepsilon_1) > \rho(\mathbf{r})(1 + \theta\varepsilon_1)$ . Using this along with the bound (*) we obtain that w.p. at least  $1 - \delta$ ,  $\hat{\rho}(\mathbf{r}_*) > \hat{\rho}(\mathbf{r})$  when  $\varepsilon_2 > 0$ . Since our algorithm returns  $\hat{\mathbf{r}}$  as the maximizer of  $\hat{\rho}$ , w.p. at least  $1 - \delta$  we get  $\min \left\{\|\tilde{\mathbf{r}} - \tilde{\mathbf{r}}_*\|_2, \|\tilde{\mathbf{r}} + \tilde{\mathbf{r}}_*\|_2\right\} \leq \varepsilon_2$ . Using Lemma 2.1,  $\min \left\{\|\hat{\mathbf{r}} - \mathbf{r}_*\|_2, \|\hat{\mathbf{r}} + \mathbf{r}_*\|_2\right\} \leq 4\sqrt{\frac{\lambda_{\max}}{\lambda_{\min}}}\varepsilon_2$ . Substituting  $\varepsilon_2 = \frac{\varepsilon}{4}\sqrt{\frac{\lambda_{\min}}{\lambda_{\max}}}$ ,  $\|\mathbf{r} - \mathbf{r}_*\|_2 \leq \varepsilon$  w.p. at least  $1 - \delta$ . The conditions on  $\varepsilon_1$  are satisfied by taking it to be  $\leq O\left(\frac{\kappa_3\varepsilon^2\lambda_{\min}}{\theta(2 + \kappa_3)\lambda_{\max}}\right)$ , and thus we can take  $m \geq O\left((d/\varepsilon^4)\log(d/\delta)\left(\frac{\lambda_{\max}}{\lambda_{\min}}\right)^2\theta^2\left(\frac{2+\kappa_3}{\kappa_3}\right)^2\right) = O\left((d/\varepsilon^4)\log(d/\delta)\left(\frac{\lambda_{\max}}{\lambda_{\min}}\right)^4q^4\right)$ , using Defn. 4.3. This completes the proof.
5 Proof Sketches for Theorems 1.3 and 1.5
Theorem 1.3: Case  $N(\mathbf{0},\mathbf{I})$ ,  $f(\mathbf{x}) = \mathrm{pos}(\mathbf{r}_{*}^{\top}\mathbf{x})$ ,  $k \neq q/2$ . The algorithm (Algorithm 3) and the proof is in Appendix A. We argue that a vector sampled uniformly at random from a bag is distributed as  $\omega \mathbf{X}_1 + (1 - \omega)\mathbf{X}_0$  where  $\mathbf{X}_a \sim N(\mathbf{0},\mathbf{I})$  conditioned on  $f(\mathbf{X}_a) = a$  and  $\omega$  is an independent  $\{0,1\}$ -Bernoulli r.v. s.t.  $p(\omega = 1) = k/q$ . This along with the fact that uncorrelated Gaussians are independent, allows us to show that the expectation is  $\mathbf{0}$  in any direction orthogonal to  $\mathbf{r}_{*}$  and to compute the expectation in the direction of  $\mathbf{r}_{*}$ . We then use Lemma 3.1 to get the sample complexity expression.
Theorem 1.5: Case  $N(\boldsymbol{\mu}, \boldsymbol{\Sigma})$ ,  $f(\mathbf{x}) = \mathrm{pos}(\mathbf{r}_*^\top \mathbf{x} + c_*)$ . The algorithm (Algorithm 4) and the detailed proof is given in Appendix C. We start by generalizing the high probability geometric error bound in Lemma 4.1 to this case (Lemma C.1 proven in Appendix C.1) and appropriately generalize  $\kappa_1, \kappa_2, \kappa_3$  and  $\theta$ . The rest of the proof is similar to Section 4.1. An extra factor of  $O(\ell^2)$  is introduced to the sample complexity from Lemma 3.1. Next, assuming the geometric bound, we give a high probability bound on the generalization error in Lemma C.2. In this analysis, we bound the BagErrsample(h,M) where  $h(\mathbf{x}) = \mathrm{pos}(\hat{\mathbf{r}}^\top \mathbf{x} + c_*)$  with  $\hat{\mathbf{r}}$  being the geometric estimate of  $\mathbf{r}_*$  and  $\mathcal{M}$  is a sample of bags. This introducing the dependencies on  $(\sqrt{\lambda_{\max}} + \| \boldsymbol{\mu} \|_2) / \sqrt{\lambda_{\min}}$  as well as on  $\Phi(\ell)$ . Our subroutine to find  $\hat{c}$  ensures that  $h^*(\mathbf{x}) = \mathrm{pos}(\hat{\mathbf{r}}^\top \mathbf{x} + \hat{c})$  satisfies BagErrsample(h*, M) ≥ BagErrsample(h, M). We then use Theorem 2.2 to bound the generalization error of  $h^*$ . Lemmas C.1 and C.2 together imply Theorem 1.5.
General Gaussian. We empirically evaluate our algorithmic technique on centered and general Gaussian distributions for learning homogeneous LTFs. For homogeneous LTFs the general case algorithm (Alg. 4 in Appendix C) boils down to Alg. 2 in Sec. 4. The experimental LLP datasets are created using samples from both balanced as well as unbalanced bag oracles. In particular, for dimension  $d \in \{10,50\}$ , and each pair  $(q,k) \in \{(2,1),(3,1),(10,5),(10,8),(50,25),(50,35)\}$
and  $m =$  we create 25 datasets as follows: for each dataset (i) sample a random unit vector  $\mathbf{r}^*$  and let  $f(\mathbf{x})\coloneqq \mathrm{pos}\left(\mathbf{r}^{*\top}\mathbf{x}\right)$ , (ii) sample  $\pmb{\mu}$  and  $\pmb{\Sigma}$  randomly (see Appendix F for details), (iii) sample  $m = 2000$  training bags from  $\mathsf{Ex}(f,N(\pmb {\mu},\pmb {\Sigma}),q,k)$ , (iv) sample 1000 test instances  $(\mathbf{x},f(\mathbf{x}))$ ,  $\mathbf{x}\gets N(\pmb {\mu},\pmb {\Sigma})$ . We fix  $\pmb{\mu} = \mathbf{0}$  for the centered Gaussian case.
For comparison we include the random LTF algorithm in which we sample 100 random LTFs and return the one that satisfies the most bags. In addition, we evaluate the Algorithm of [25] on  $(q,k) = (2,1)$ , and the Algorithm of [26] on  $(q,k) = (3,1)$ . We measure the accuracy of each method on the test set of each dataset. The algorithms of [25, 26] are considerably slower and we use 200 training bags for them. The results for centered Gaussian are in Table 1a and for the general Gaussian are in Table 1b. We observe that our algorithms perform significantly better in terms of accuracy than the comparative methods in all the bag distribution settings. Further, our algorithms have significantly lower error bounds (see Appendix F).
Notice in Tables 1a and 1b that the test accuracy for Algorithm 2 decreases with an increase in  $q$  and  $d$ . This is consistent with the sample complexity expressions in Thm. 1.4 and Thm. 1.5. Also, notice that the test accuracy for Algorithm 2 for general Gaussian (Table 1b) is usually lesser than the same for centered Gaussian (Table 1a). This supports the theoretical result that the sample complexity increases with the increase in  $l$ .
Appendix F has additional details and further experiments for the  $N(\mathbf{0},\mathbf{I})$  with homogeneous LTFs,  $N(\pmb {\mu},\pmb {\Sigma})$  with non-homogeneous LTFs as well as on noisy label distributions.
Table 1: Algorithm A2 vs. rand. LTF (R) vs SDP algorithms (S)
d	q	k	A2	R	S
10	2	1	98.12	78.26	88.40
10	3	1	98.27	77.16	67.31
10	10	5	97.9	78.66	-
10	10	8	97.87	77.64	-
10	50	25	97.87	76.67	-
10	50	35	97.9	77.17	-
50	2	1	95.64	61.25	57.83
50	3	1	95.21	61.15	58.69
50	10	5	95.59	55.06	-
50	10	8	94.34	63.17	-
50	50	25	95.16	55.76	-
50	50	35	94.74	61.02	-
(a)  $N(\mathbf{0},\pmb {\Sigma})$  feature-vectors.
d	q	k	A2	R	S
10	2	1	98.18	78.32	90.10
10	3	1	97.92	75.14	70.80
10	10	5	97.86	70.41	-
10	10	8	97.4	69.86	-
10	50	25	97.57	70.48	-
10	50	35	97.6	62.86	-
50	2	1	94.99	58.68	61.12
50	3	1	95.6	59.8	62.39
50	10	5	95.27	57.43	-
50	10	8	94.44	61.82	-
50	50	25	94.97	53.98	-
50	50	35	94.33	56.97	-
(b)  $N(\pmb {\mu},\pmb {\Sigma})$  feature-vectors.
7 Conclusion and Future work
Our work shows that LTFs can be efficiently properly learnt in the LLP setting from random bags with given label proportion whose feature-vectors are sampled independently from a Gaussian space, conditioned on their underlying labels. For the simple case of  $N(\mathbf{0},\mathbf{I})$  distribution and bags with unbalanced labels we provide a mean estimation based algorithm. For the general scenarios we develop a more sophisticated approach using the principal component of a matrix formed from certain covariance matrices. To resolve the ambiguity between the obtained solutions we employ novel generalization error bounds from bag satisfaction to instance classification. We also show that subgaussian concentration bounds are applicable on the thresholded Gaussians, yielding efficient sample complexity bounds. Our experimental results validate the performance guarantees of our algorithmic techniques.
In future work, classes of distributions other than Gaussian could be similarly investigated. Classifiers other than LTFs are also interesting to study in the LLP setting.
[1] P. Awasthi, M. F. Balcan, and P. M. Long. The power of localization for efficiently learning linear separators with noise. J. ACM, 63(6):50:1-50:27, 2017.
[22] G. Patrini, R. Nock, T. S. Caetano, and P. Rivera. (almost) no label no cry. In Proc. Advances in Neural Information Processing Systems, pages 190-198, 2014.
[23] N. Quadrianto, A. J. Smola, T. S. Caetano, and Q. V. Le. Estimating labels from label proportions. J. Mach. Learn. Res., 10:2349-2374, 2009.
[24] S. Rueping. SVM classifier estimation from group probabilities. In Proc. ICML, pages 911-918, 2010.
[25] R. Saket. Learnability of linear thresholds from label proportions. In Proc. NeurIPS, 2021. URL https://openreview.net/forum?id=5BnaKeEwuYk.
[26] R. Saket. Algorithms and hardness for learning linear thresholds from label proportions. In Proc. NeurIPS, 2022. URL https://openreview.net/forum?id=4LZo68TuF-4.
[27] Rishi Saket, Aravindan Raghuveer, and Balaraman Ravindran. On combining bags to better learn from label proportions. In AISTATS, volume 151 of Proceedings of Machine Learning Research, pages 5913-5927. PMLR, 2022. URL https://proceedings.mlr.press/v151/saket22a.html.
[28] C. Scott and J. Zhang. Learning from label proportions: A mutual contamination framework. In Proc. NeurIPS, 2020.
[29] Leslie G. Valiant. A theory of the learnable. Commun. ACM, 27(11):1134-1142, 1984.
[30] S. Vempala. Learning convex concepts from gaussian distributions with PCA. In FOCS, pages 124-130, 2010.
[31] R. Vershynin. How close is the sample covariance matrix to the actual covariance matrix? J. Theor. Probab., 25:655-686, 2012.
[32] Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018. doi: 10.1017/9781108231596.
[33] J. Wojtusiak, K. Irvin, A. Birerdinc, and A. V. Baranova. Using published medical results and non-homogenous data in rule learning. In Proc. International Conference on Machine Learning and Applications and Workshops, volume 2, pages 84-89. IEEE, 2011.
[34] F. X. Yu, D. Liu, S. Kumar, T. Jebara, and S. F. Chang.  $\propto$ SVM for learning with label proportions. In Proc. ICML, volume 28, pages 504-512, 2013.
[35] F. X. Yu, K. Choromanski, S. Kumar, T. Jebara, and S. F. Chang. On learning from label proportions. CoRR, abs/1402.5902, 2014. URL http://arxiv.org/abs/1402.5902.
A Proof of Theorem 1.3
For the setting of Theorem 1.3 we provide Algorithm 3.
Algorithm 3 PAC Learner for homogenous LTFs from unbalanced bags over  $N(\mathbf{0},\mathbf{I})$
Input:  $\mathsf{Ex}(f,\mathcal{D},q,k),m$  where  $f(\mathbf{x}) = \mathrm{pos}\left(\mathbf{r}_*\mathsf{T}\mathbf{x}\right)$ $\| \mathbf{r}_{*}\|_{2} = 1,k\neq q / 2$
1. Compute  $\hat{\mu}_B$  using MeanCovsEstimator with  $m$  samples.
2. If  $k > q / 2$  Return:  $\hat{\mu}_B / \| \hat{\mu}_B\| _2$  else Return:  $-\hat{\mu}_B / \| \hat{\mu}_B\| _2$
Define  $\mathbf{X}_a\sim N(\mathbf{0},\mathbf{I})$  conditioned on  $f(\mathbf{X}_a) = a$  for  $a\in \{0,1\}$ , and let  $\mathbf{X}_B$  denotes the random feature-vector u.a.r sampled from a randomly sampled bag. By the definition of the bag oracle,  $\mathbf{X}_B\coloneqq \omega \mathbf{X}_1 + (1 - \omega)\mathbf{X}_0$ , where  $\omega$  is an independent  $\{0,1\}$ -Bernoulli r.v. s.t.  $p(\omega = 1) = k / q$ . Let  $g^{*}\coloneqq \mathbf{r}_{*}^{\mathrm{T}}\mathbf{X}_{B}$ .  $g^{*}\sim N(0,1)$  and let  $g_{a}^{*} = \mathbf{r}_{*}^{\mathrm{T}}\mathbf{X}_{a}$ ,  $a\in \{0,1\}$ . Since  $\mathbf{r}_*$  is a unit vector,  $g_{a}^{*}$  is a half-gaussian and by direct integration we obtain  $\mathbb{E}[g_a^*] = (-1)^{1 - a}\sqrt{2 / \pi}$  for  $a\in \{0,1\}$ . Thus,
$$
\mathbb {E} [ g ^ {*} ] = \frac {k}{q} \mathbb {E} [ g _ {1} ^ {*} ] + \left(1 - \frac {k}{q}\right) \mathbb {E} [ g _ {0} ^ {*} ] = \eta (q, k) := \left(\frac {2 k}{q} - 1\right) \sqrt {\frac {2}{\pi}}, 0 \leq \eta (q, k) \leq 1
$$
On the other hand, let  $g^{\perp} = \mathbf{r}^{\mathsf{T}}\mathbf{X}$  s.t.  $\mathbf{r}^{\mathsf{T}}\mathbf{r}_{*} = 0$  and  $\| \mathbf{r}\| _2 = 1$ , and let  $g_{a}^{\perp} = \mathbf{r}^{\mathsf{T}}\mathbf{X}_{a}$ ,  $a\in \{0,1\}$ . Since  $\mathbf{X}_a$  ( $a\in \{0,1\}$ ) is given by conditioning a standard Gaussian vector only on the component in the direction of  $\mathbf{r}_{*}$ , its component along any direction orthogonal to  $\mathbf{r}_{*}$  is a one-dimensional standard Gaussian. Therefore,  $g_{a}^{\perp}$  are iid  $N(0,1)$  ( $a\in \{0,1\}$ ), and so is  $g^{\perp}$ , implying  $\mathbb{E}\left[g^{\perp}\right] = 0$ . Thus, the value of  $\mathbf{r}^{\mathsf{T}}\mathbb{E}\left[\mathbf{X}_B\right]$  is (i)  $\eta (q,k)$  if  $\mathbf{r} = \mathbf{r}_{*}$ , and (ii) 0 if  $\mathbf{r}\perp \mathbf{r}_{*}$ . In other words,  $\boldsymbol {\mu}_B = \mathbb{E}\left[\mathbf{X}_B\right] = \eta (q,k)\mathbf{r}_{*}$ , and  $\| \boldsymbol {\mu}_B\| _2 = |\eta (q,k)|$ . Hence, if  $\eta (q,k) > 0$  then  $\mathbf{r}_{*} = \boldsymbol {\mu}_B / \| \boldsymbol {\mu}_B\| _2$  else  $\mathbf{r}_{*} = -\boldsymbol {\mu}_B / \| \boldsymbol {\mu}_B\| _2$
With  $m = O\left(\eta (q,k)^2 (d / \varepsilon^2)\log (d / \delta)\right) = O\left((d / \varepsilon^2)\log (d / \delta)\right)$ , the following lemma along with Lemma 2.3 completes the proof of Theorem 1.3.
Lemma A.1. Algorithm 3 returns a normal vector  $\hat{\mathbf{r}}$  such that  $\| \mathbf{r}_* - \hat{\mathbf{r}} \|_2 \leq \varepsilon$  w.p. at least  $1 - \delta$  when  $m \geq O\left((d / \varepsilon^2) \log (d / \delta)\right)$  for  $\varepsilon, \delta > 0$ .
Proof. First, we can assume  $\varepsilon \leq 2$  since the distance between two unit vectors is at most 2 and therefore the lemma is trivially true for  $\varepsilon > 2$ . By Lemma 3.1, taking  $m \geq = O\left((d / \varepsilon^2) \log (d / \delta)\right) = O\left(\eta (q,k)^2 (d / \varepsilon^2) \log (d / \delta)\right)$  ensures  $\|\hat{\pmb{\mu}}_B - \pmb{\mu}_B\|_2 \leq \varepsilon |\eta (q,k)| / 4 = \varepsilon \| \pmb{\mu}_B\|_2 / 4$  w.p.  $1 - \delta$ . Therefore, by triangle inequality,  $\|\hat{\pmb{\mu}}_B\|_2 - \| \pmb{\mu}_B\|_2| \leq \varepsilon \| \pmb{\mu}_B\|_2 / 4 \Rightarrow \| \hat{\pmb{\mu}}_B\|_2 / \| \pmb{\mu}_B\|_2 \in [1 - \frac{\varepsilon}{4}, 1 + \frac{\varepsilon}{4}]$ . Now,  $\mathbf{r}_* = \mathrm{sign}(\eta (q,k)) \pmb{\mu}_B / \| \pmb{\mu}_B\|_2$  and the algorithm returns  $\hat{\mathbf{r}} := \mathrm{sign}(\eta (q,k)) \hat{\pmb{\mu}}_B / \| \hat{\pmb{\mu}}_B\|_2$ .
$$
\begin{array}{l} \| \hat {\mathbf {r}} - \mathbf {r} _ {*} \| _ {2} = \left\| \frac {\hat {\boldsymbol {\mu}} _ {B}}{\| \hat {\boldsymbol {\mu}} _ {B} \| _ {2}} - \frac {\boldsymbol {\mu} _ {B}}{\| \boldsymbol {\mu} _ {B} \| _ {2}} \right\| _ {2} \leq \frac {\| \hat {\boldsymbol {\mu}} _ {B} \| \boldsymbol {\mu} _ {B} \| _ {2} - \boldsymbol {\mu} _ {B} \| \boldsymbol {\mu} _ {B} \| _ {2} + \boldsymbol {\mu} _ {B} \| \boldsymbol {\mu} _ {B} \| _ {2} - \boldsymbol {\mu} _ {B} \| \hat {\boldsymbol {\mu}} _ {B} \| _ {2} \| _ {2}}{\| \hat {\boldsymbol {\mu}} _ {B} \| _ {2} \| \boldsymbol {\mu} _ {B} \| _ {2}} \\ \leq \frac {\| \hat {\boldsymbol {\mu}} _ {B} - \boldsymbol {\mu} _ {B} \| _ {2} \| \boldsymbol {\mu} _ {B} \| _ {2} + \| \boldsymbol {\mu} _ {B} \| _ {2} \| \boldsymbol {\mu} _ {B} \| _ {2} - \| \hat {\boldsymbol {\mu}} _ {B} \| _ {2} \|}{\| \hat {\boldsymbol {\mu}} _ {B} \| _ {2} \| \boldsymbol {\mu} _ {B} \| _ {2}} \\ \leq \frac {\frac {\varepsilon}{2} \| \boldsymbol {\mu} _ {B} \| _ {2}}{\| \boldsymbol {\mu} _ {B} \| _ {2} - \frac {\varepsilon}{4} \| \boldsymbol {\mu} _ {B} \| _ {2}} \leq \frac {2 \varepsilon}{4 - \varepsilon} \leq \varepsilon \quad \text {f o r} \varepsilon \leq 2. \\ \end{array}
$$
B Useful Tools
B.1 Chernoff Bound
We state the well known Chernoff Bound.
Theorem B.1. Let  $X_{1},\ldots ,X_{n}$  be iid  $\{0,1\}$ -valued random variables. Let  $S$  be their sum and  $\mu = \mathbb{E}[S]$ . Then, for any  $\delta >0$
$$
\Pr \left[ S \geq (1 + \delta) \mu \right] \leq \exp \left(- \delta^ {2} \mu / (2 + \delta)\right).
$$
B.2 Relationship between  $\Sigma_B$  and  $\Sigma_D$
Lemma B.2. If  $\Sigma_B, \Sigma_D$  are as defined in the preliminaries and  $\mathbf{X}_a \coloneqq \mathbf{X}|f(\mathbf{X}) = a$ , then
$$
\boldsymbol {\Sigma} _ {D} = 2 \boldsymbol {\Sigma} _ {B} + \frac {2}{q - 1} \left(\frac {k}{q}\right) \left(1 - \frac {k}{q}\right) (\mathbb {E} [ \mathbf {X} _ {1} ] - \mathbb {E} [ \mathbf {X} _ {0} ]) (\mathbb {E} [ \mathbf {X} _ {1} ] - \mathbb {E} [ \mathbf {X} _ {0} ]) ^ {T}
$$
Proof. Let  $\mathbf{X}_B$  be a random-feature vector sampled uniformly from a random bag sampled from  $\mathcal{O}$ . Hence, with probability  $k / q$  it is sampled from  $\mathbf{X}_1$  and with a probability of  $1 - k / q$  it is sampled from  $\mathbf{X}_0$ . Hence,
$$
\boldsymbol {\mu} _ {B} = \mathbb {E} [ \mathbf {X} _ {B} ] = \frac {k}{q} \mathbb {E} [ \mathbf {X} _ {1} ] + \left(1 - \frac {k}{q}\right) \mathbb {E} [ \mathbf {X} _ {0} ]
$$
$$
\boldsymbol {\Sigma} _ {B} = \mathbb {E} \left[ \mathbf {X} _ {B} \mathbf {X} _ {B} ^ {T} \right] - \mathbb {E} \left[ \mathbf {X} _ {B} \right] \mathbb {E} \left[ \mathbf {X} _ {B} \right] ^ {T} \text {w h e r e} \mathbb {E} \left[ \mathbf {X X} ^ {T} \right] = \frac {k}{q} \mathbb {E} \left[ \mathbf {X} _ {1} \mathbf {X} _ {1} ^ {T} \right] + \left(1 - \frac {k}{q}\right) \mathbb {E} \left[ \mathbf {X} _ {0} \mathbf {X} _ {0} ^ {T} \right]
$$
Let  $\mathbf{X}_D = \mathbf{X} - \mathbf{X}'$  where  $(\mathbf{X},\mathbf{X}')$  are a random pair of feature-vectors sampled (without replacement) from a random bag sampled from  $\mathcal{O}$ . Hence, with probability  $\binom{k}{2}/\binom{q}{2}$  it is the difference of two vectors sampled independently from  $\mathbf{X}_1$ , with probability  $\binom{q-k}{2}/\binom{q}{2}$  it is the difference of two vectors sampled independently from  $\mathbf{X}_0$ , with probability  $k(q-k)/2\binom{q}{2}$ , it is the difference of one vector sampled from  $X_1$  and another sampled independently from  $X_0$  and with probability  $k(q-k)/2\binom{q}{2}$ , it is the difference of one vector sampled from  $X_0$  and another sampled independently from  $X_1$ . Then,
$$
\begin{array}{l} \boldsymbol {\Sigma} _ {D} = \mathbb {E} [ \mathbf {X} _ {D} \mathbf {X} _ {D} ^ {T} ] = \frac {1}{\binom {q} {2}} \left[ \binom {k} {2} \mathbb {E} [ (\mathbf {X} _ {1} - \mathbf {X} _ {1} ^ {\prime}) (\mathbf {X} _ {1} - \mathbf {X} _ {1} ^ {\prime}) ^ {T} ] + \binom {q - k} {2} \mathbb {E} [ (\mathbf {X} _ {0} - \mathbf {X} _ {0} ^ {\prime}) (\mathbf {X} _ {0} - \mathbf {X} _ {0} ^ {\prime}) ^ {T} ] \right] \\ + \frac {k (q - k)}{2} \mathbb {E} [ (\mathbf {X} _ {1} - \mathbf {X} _ {0} ^ {\prime}) (\mathbf {X} _ {1} - \mathbf {X} _ {0} ^ {\prime}) ^ {T} ] + \frac {k (q - k)}{2} \mathbb {E} [ (\mathbf {X} _ {0} - \mathbf {X} _ {1} ^ {\prime}) (\mathbf {X} _ {0} - \mathbf {X} _ {1} ^ {\prime}) ^ {T} ] \Biggr ] \\ \end{array}
$$
Due to independence, we obtain for  $a \in \{0,1\}$
$$
\begin{array}{l} \mathbb {E} \left[ \left(\mathbf {X} _ {a} - \mathbf {X} _ {a} ^ {\prime}\right) \left(\mathbf {X} _ {a} - \mathbf {X} _ {a} ^ {\prime}\right) ^ {T} \right] = 2 \mathbb {E} \left[ \mathbf {X} _ {a} \mathbf {X} _ {a} ^ {T} \right] - 2 \mathbb {E} \left[ \mathbf {X} _ {a} \right] \mathbb {E} \left[ \mathbf {X} _ {a} \right] ^ {T} \\ \mathbb {E} \big [ (\mathbf {X} _ {a} - \mathbf {X} _ {1 - a} ^ {\prime}) (\mathbf {X} _ {a} - \mathbf {X} _ {1 - a} ^ {\prime}) ^ {T} \big ] = \mathbb {E} [ \mathbf {X} _ {1} \mathbf {X} _ {1} ^ {T} ] + \mathbb {E} [ \mathbf {X} _ {0} \mathbf {X} _ {0} ^ {T} ] - \mathbb {E} [ \mathbf {X} _ {1} ] \mathbb {E} [ \mathbf {X} _ {0} ] ^ {T} - \mathbb {E} [ \mathbf {X} _ {0} ] \mathbb {E} [ \mathbf {X} _ {1} ] ^ {T} \\ \end{array}
$$
$$
\begin{array}{l} \boldsymbol {\Sigma} _ {D} = \frac {1}{\binom {q} {2}} \left[ \left(2 \binom {k} {2} + k (q - k)\right) \mathbb {E} [ \mathbf {X} _ {1} \mathbf {X} _ {1} ^ {T} ] + \left(2 \binom {q - k} {2} + k (q - k)\right) \mathbb {E} [ \mathbf {X} _ {0} \mathbf {X} _ {0} ^ {T} ] \right. \\ \left. - \left(k (k - 1) \mathbb {E} [ \mathbf {X} _ {1} ] \mathbb {E} [ \mathbf {X} _ {1} ] ^ {T} + (q - k) (q - k - 1) \mathbb {E} [ \mathbf {X} _ {0} ] \mathbb {E} [ \mathbf {X} _ {0} ] ^ {T}\right) \right. \\ \left. + \left(k (q - k) \left(\mathbb {E} [ \mathbf {X} _ {1} ] \mathbb {E} [ \mathbf {X} _ {0} ] ^ {T} + \mathbb {E} [ \mathbf {X} _ {0} ] \mathbb {E} [ \mathbf {X} _ {1} ] ^ {T}\right)\right) \right] \\ \end{array}
$$
Simplifying  $\Sigma_D - 2\Sigma_B$ , we get
$$
\boldsymbol {\Sigma} _ {D} - 2 \boldsymbol {\Sigma} _ {B} = \frac {2}{q - 1} \left(\frac {k}{q}\right) \left(1 - \frac {k}{q}\right) \left[ (\mathbb {E} [ \mathbf {X} _ {1} ] - \mathbb {E} [ \mathbf {X} _ {0} ]) (\mathbb {E} [ \mathbf {X} _ {1} ] - \mathbb {E} [ \mathbf {X} _ {0} ]) ^ {T} \right]
$$
B.3 Bound on  $\gamma (\mathbf{r})$  when  $\mathbf{r}^T\Sigma_B\mathbf{r}_* = 0$
Lemma B.3. If  $\mathbf{r}^T\pmb{\Sigma}_B\mathbf{r}_* = 0$  then  $\gamma (\mathbf{r})\leq 1 - \left(\frac{\lambda_{\min}}{\lambda_{\max}}\right)^2\frac{1 - \max(0,\kappa_1(q,k,\ell))}{1 - \min(0,\kappa_1(q,k,\ell))}$ . Further if  $\ell = 0$ , then  $|\gamma (\mathbf{r})|\leq 1 - \left(\frac{\lambda_{\min}}{\lambda_{\max}}\right)^2 (1 - \kappa_1(q,k))$ .
Proof. We begin by observing the for any  $\mathbf{r} \in \mathbb{R}^d$ ,  $\|\pmb{\Sigma}^{1/2}\mathbf{r}\|_2 = 1$ ,  $\mathrm{Var}[\mathbf{r}^T\mathbf{Z}_B] = 1 - \kappa_1(q,k,\ell)$ . Hence,  $\lambda_{\min}(\pmb{\Sigma}_B) \geq (1 - \max(0,\kappa_1(q,k,\ell)))\lambda_{\min}$  and  $\lambda_{\max}(\pmb{\Sigma}_B) \leq (1 - \min(0,\kappa_1(q,k,\ell)))\lambda_{\max}$ .
Thus, by Cautchy-Schwartz inequality,
$$
\| \boldsymbol {\Sigma} _ {B} ^ {1 / 2} \boldsymbol {\Sigma} ^ {- 1 / 2} \| _ {2} \leq \sqrt {1 - \min (0 , \kappa_ {1} (q , k , \ell))} \sqrt {\frac {\lambda_ {\max}}{\lambda_ {\min}}}
$$
$$
\| \boldsymbol {\Sigma} ^ {1 / 2} \boldsymbol {\Sigma} _ {B} ^ {- 1 / 2} \| _ {2} \leq \frac {1}{\sqrt {1 - \max (0 , \kappa_ {1} (q , k , \ell))}} \sqrt {\frac {\lambda_ {\operatorname* {m a x}}}{\lambda_ {\operatorname* {m i n}}}}
$$
Define  $\tilde{\mathbf{r}} = \frac{\pmb{\Sigma}^{1/2}\mathbf{r}}{\|\pmb{\Sigma}^{1/2}\mathbf{r}\|_2}$  for any  $\mathbf{r} \in \mathbb{R}^d$ . Observe that  $\gamma(\mathbf{r}) = \tilde{\mathbf{r}}^T\tilde{\mathbf{r}}_*$ . Now since  $\mathbf{r}^T\pmb{\Sigma}_B\mathbf{r}_* = 0$ , by substitution  $(\pmb{\Sigma}_B^{1/2}\pmb{\Sigma}^{-1/2}\tilde{\mathbf{r}})^T(\pmb{\Sigma}_B^{1/2}\pmb{\Sigma}^{-1/2}\tilde{\mathbf{r}}_*) = 0$ . Thus, using Cautchy-Schwartz again
$$
\begin{array}{l} \| \boldsymbol {\Sigma} _ {B} ^ {1 / 2} \boldsymbol {\Sigma} ^ {- 1 / 2} \tilde {\mathbf {r}} - \boldsymbol {\Sigma} _ {B} ^ {1 / 2} \boldsymbol {\Sigma} ^ {- 1 / 2} \tilde {\mathbf {r}} _ {*} \| _ {2} = \sqrt {\| \boldsymbol {\Sigma} _ {B} ^ {1 / 2} \boldsymbol {\Sigma} ^ {- 1 / 2} \tilde {\mathbf {r}} \| _ {2} ^ {2} + \| \boldsymbol {\Sigma} _ {B} ^ {1 / 2} \boldsymbol {\Sigma} ^ {- 1 / 2} \tilde {\mathbf {r}} _ {*} \| _ {2} ^ {2}} \\ \geq \frac {\sqrt {\| \tilde {\mathbf {r}} \| _ {2} ^ {2} + \| \tilde {\mathbf {r}} _ {*} \| _ {2} ^ {2}}}{\| \boldsymbol {\Sigma} ^ {1 / 2} \boldsymbol {\Sigma} _ {B} ^ {- 1 / 2} \| _ {2}} \geq \sqrt {2 (1 - \max  (0 , \kappa_ {1} (q , k , \ell)))} \sqrt {\frac {\lambda_ {\min }}{\lambda_ {\max }}} \\ \end{array}
$$
Again using Cautchy-Schwartz,