SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anand Brahmbhatt*
Google Research India anandpareshb@google.com
Rishi Saket*
Google Research India rishisaket@google.com
Aravindan Raghuveer
Google Research India araghuveer@gmail.com
Learning from label proportions (LLP) is a generalization of supervised learning in which the training data is available as sets or bags of feature-vectors (instances) along with the average instance-label of each bag. The goal is to train a good instance classifier. While most previous works on LLP have focused on training models on such training data, computational learnability of LLP was only recently explored by [25, 26] who showed worst case intractability of properly learning linear threshold functions (LTFs) from label proportions. However, their work did not rule out efficient algorithms for this problem on natural distributions.
In this work we show that it is indeed possible to efficiently learn LTFs using LTFs when given access to random bags of some label proportion in which feature-vectors are, conditioned on their labels, independently sampled from a Gaussian distribution  $N(\pmb{\mu}, \pmb{\Sigma})$ . Our work shows that a certain matrix - formed using covariances of the differences of feature-vectors sampled from the bags with and without replacement - necessarily has its principal component, after a transformation, in the direction of the normal vector of the LTF. Our algorithm estimates the means and covariance matrices using subgaussian concentration bounds which we show can be applied to efficiently sample bags for approximating the normal direction. Using this in conjunction with novel generalization error bounds in the bag setting, we show that a low error hypothesis LTF can be identified. For some special cases of the  $N(\mathbf{0}, \mathbf{I})$  distribution we provide a simpler mean estimation based algorithm. We include an experimental evaluation of our learning algorithms along with a comparison with those of [25, 26] and random LTFs, demonstrating the effectiveness of our techniques.
In learning from label proportions (LLP), the training data is aggregated into sets or bags of feature-vectors (instances). For each bag we are given its constituent feature-vectors along with only the sum or average of their labels. The goal is a to obtain a good instance-level classifier – one that minimizes the classification error on a test set of instances or bags. In this work we study the LLP learnability over Gaussian distributions of linear threshold functions (LTFs), also called linear classifiers or halfspaces, given by  $f(\mathbf{x}) = \mathrm{pos}\left(\mathbf{r}^{\mathsf{T}}\mathbf{x} + c\right)$  where  $\mathrm{pos}(a) \coloneqq 1$  if  $a > 0$  and 0 otherwise.
The probably approximately correct (PAC) model of [29] states that a concept class  $\mathcal{C}$  of  $\{0,1\}$ -valued functions can be learnt by a hypothesis class  $\mathcal{H}$  if there is an algorithm to efficiently obtain, using iid samples from a distribution on  $(\mathbf{x},f(\mathbf{x}))$ , a hypothesis  $h\in \mathcal{H}$  of arbitrarily high accuracy on that distribution, for any unknown  $f\in \mathcal{C}$ . If  $\mathcal{H} = \mathcal{C}$  we say that  $\mathcal{C}$  is properly learnable, for
\* - equal contribution
e.g. LTFs are known to be properly learnable using linear programming ([3]). This notion can be extended to the LLP setting - which for brevity we call PAC-LLP - as follows: distribution  $D$  is over bags and their label proportions  $(B, \sigma(B, f))$  where  $B = \{\mathbf{x}_1, \ldots, \mathbf{x}_q\}$  is a bag of feature vectors and  $\sigma(B, f) = \mathrm{Avg}\{f(\mathbf{x}) \mid \mathbf{x} \in B\}$ . A bag  $(B, \sigma(B, f))$  is said to be satisfied by  $h$  iff  $\sigma(B, h) = \sigma(B, f)$ , and the accuracy of  $h$  is the fraction of bags satisfied by it.
With the above notion of PAC-LLP, [25] studied the learnability of LTFs and rather disturbingly showed that for any constant  $\varepsilon > 0$  it is NP-hard to PAC-LLP learn an LTF using an LTF which satisfies  $(1/2 + \varepsilon)$ -fraction of the bags when all bags are of size at most 2, which was subsequently strengthened to a  $(4/9 + \varepsilon)$ -factor hardness by [26] who also proved  $(1/q + \varepsilon)$ -factor hardness when the bag size is at most  $q$ , for any  $q \geq 3$ . This is in contrast to the supervised learning (i.e., with unit-sized bags) in which an LTF can be efficiently learnt by an LTF using linear programming. On the algorithmic side, [25] gave a semi-definite programming (SDP) based algorithm to find an LTF satisfying  $(2/5)$ -fraction of bags of size  $\leq 2$ , which was extended by [26] to a fairly involved SDP yielding  $(1/12)$ -approximation for bags of size  $\leq 3$ , while no non-trivial algorithms for bags of size  $>3$  are known. These results show that PAC-LLP learning LTFs using LTFs is intractable on hard bag distributions, and even the non-trivial algorithms for bag sizes  $\leq 3$  are via complicated convex programming techniques. A natural question therefore, is whether the problem is tractable on natural distributions that may arise out of real world scenarios.
We answer the above question in the affirmative when the feature-vectors are distributed according to some (unknown) Gaussian distribution  $\mathcal{D} = N(\boldsymbol{\mu},\boldsymbol{\Sigma})$  in  $d$ -dimensions. Gaussian distributions are ubiquitous in machine learning and in many applications the input data distribution is modeled as multivariate Gaussians, and several previous works [8, 30] have studied learnability in Gaussian distributions. An unknown target LTF is given by  $f(\mathbf{x}) \coloneqq \mathrm{pos}\left(\mathbf{r}_*\mathsf{T}\mathbf{x} + c_*\right)$  where  $\|\mathbf{r}_*\|_2 = 1$ . Let  $\mathcal{D}_a$  be the distribution of  $\mathbf{x} \gets \mathcal{D}$  conditioned on  $f(\mathbf{x}) = a$ , for  $a \in \{0,1\}$ . Using this we formalize the notion of a distribution  $\mathcal{O}$  on bags of size  $q$  and average label  $k/q$ : a random bag  $B$  sampled from  $\mathcal{O}$  consists of  $k$  iid samples from  $\mathcal{D}_1$  and  $(q - k)$  iid samples from  $\mathcal{D}_0$ . The case of  $k \in \{0,q\}$  is uninteresting as all instances in such bags are either labeled 0 or 1 and traditional PAC-learning for LTFs can be employed directly. Unlike [25, 26] our objective is to directly maximize the instance-level level accuracy on  $\mathcal{D}$ . With this setup we informally describe our main result.
Our PAC-LLP LTF Learner (Informal): Assuming mild conditions on  $\Sigma, \mu$  and  $c_*$ , for any  $q, k \in \mathbb{Z}^+$  s.t.  $1 \leq k \leq q - 1$  and  $\varepsilon, \delta > 0$ , there is an algorithm that samples at most  $m$  bags from  $\mathcal{O}$  and runs in time  $O(t + m)$  and with probability  $1 - \delta$  produces an LTF  $h$  s.t.
$\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h(\mathbf{x})] \leq \varepsilon$  if  $k \neq q/2$ , and
$\operatorname*{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h(\mathbf{x})] \leq \varepsilon$  or  $\operatorname*{Pr}[f(\mathbf{x}) \neq (1 - h(\mathbf{x}))] \leq \varepsilon$  if  $k = q/2$ ,
where  $t, m$  are fixed polynomials in  $d, q, (1 / \varepsilon), \log(1 / \delta)$ . We also obtain a more efficient algorithm when  $k \neq q / 2$ ,  $\pmb{\mu} = \mathbf{0}$ ,  $c_* = 0$  and  $\pmb{\Sigma} = \mathbf{I}$ . The ambiguity in the case of  $k = q / 2$  is inherent since bags of label proportion  $1 / 2$  consistent with an LTF  $f(\mathbf{x})$  are also consistent with  $(1 - f(\mathbf{x}))$ .
Remark 1.1 (Mixtures of  $(q, k)$ ). The training data could consist of bags of different sizes and label proportions, however typically the maximum size of bags is bounded by (say)  $Q$ , and in a large enough sample we would have at least  $(1 / Q^2)$ -fraction of bags of a particular size and label proportion and we can apply our PAC-LLP LTF Learner above to that subsample.
The LLP problem is motivated by many real applications where labels are available not for each feature-vector but only as the average labels of bags of feature-vectors. This may occur because of privacy and legal ([24, 33]) reasons, supervision cost ([5]) or lack of labeling instrumentation ([10]). Previous works ([9, 15, 20, 24]) on LLP have applied techniques such as such as clustering, and linear classifiers and MCMC. Specifically for LLP, assuming class conditional independence of bags, [23] gave an algorithm to learn an exponential generative model, which was further generalized by [22]. On the other hand, the work of [34] proposed a novel proportional SVM based algorithms which optimized the SVM loss over instance-labels which were constrained by bag-level loss w.r.t the given label-proportions. Subsequently, approaches based on deep neural nets for large-scale and multi-class data ([18, 11, 19, 21]), as well as bag pre-processing techniques ([28, 27]) have been developed. Recently, [4, 6] have proposed model training methods for either random or curated bags.
The LLP framework (as an analogue of PAC learning) was first formalized in the work of [35]. They bounded the generalization error of a trained classifier when taking the (bag, label-proportion)-pairs
as instances sampled iid from some distribution. Their loss function was different – a weaker notion than the strict bag satisfaction predicate of [25, 26]. A single-bag variant – class ratio estimation – of LLP was studied by [13] in which learning LTFs has a simple algorithm (see Appendix G). Nevertheless, the study of computational learning in the LLP framework has been fairly limited, apart from the works of [25, 26] whose results of learning LTFs in the LLP setting have been described earlier in this section.
In the fully supervised setting [3] showed that LTFs can be learnt using LTFs via linear programming without any distributional assumptions. Adversarial label noise makes the problem NP-hard to approximate beyond the trivial  $(1/2)$ -factor even using constant degree polynomial thresholds as hypothesis ([12, 14, 2]). However, under distributional assumptions a series of results ([16, 17, 1, 7]) have given efficient algorithms to learn adversarially noisy LTFs.
Next, Sec. 1.2 mathematically defines our problem statement. Sec. 1.3 states the main results of this paper. Sec. 1.4 provides an overview of our techniques. Sec. 2 mentions some preliminary results which are used in our proofs. Sec. 3 defines and analyses a subroutine which we use in all our algorithms. Sec. 4 provides a complete proof for one of our main results. Sec 5 gives brief proof sketches of our other results. Sec. 6 mentions some experiments which support of our results.
1.2 Problem Definition
Definition 1.2 (Bag Oracle). Given distribution  $\mathcal{D}$  over  $\mathbb{R}^d$  and a target concept  $f: \mathbb{R} \to \{0,1\}$ , the bag oracle for size  $q$  and label proportion  $k/q$  ( $1 \leq k \leq q-1$ ), denoted by  $\mathsf{Ex}(f,\mathcal{D},q,k)$ , generates a bag  $\{\mathbf{x}^{(i)}\}_{i=1}^q$  such that  $\mathbf{x}^{(i)}$  is independently sampled from (i)  $\mathcal{D}_{f,1}$  for  $i = \{1,\dots,k\}$ , and (ii)  $\mathcal{D}_{f,0}$  for  $i = \{k+1,\dots,q\}$ , where  $\mathcal{D}_{f,a}$  is  $\mathbf{x} \gets \mathcal{D}$  conditioned on  $f(\mathbf{x}) = a$ , for  $a \in \{0,1\}$ .
1.3 Our results
We first state our result (proved in Appendix A) for the case of standard  $d$ -dimensional Gaussian distribution  $N(\mathbf{0}, \mathbf{I})$ , homogeneous target LTF and unbalanced bags.
Theorem 1.3. For  $q > 2$  and  $k \in \{1, \ldots, q - 1\}$  s.t.  $k \neq q / 2$  and LTF  $f(\mathbf{x}) \coloneqq \mathrm{pos}(\mathbf{r}_*^\top \mathbf{x})$ , there is an algorithm that samples  $m$  iid bags from  $\mathsf{Ex}(f, N(\mathbf{0}, \mathbf{I}), q, k)$  and runs in  $O(m)$  time to produce a hypothesis  $h(\mathbf{x}) \coloneqq \mathrm{pos}(\hat{\mathbf{r}}^\top \mathbf{x})$  s.t. w.p. at least  $1 - \delta$  over the sampling,  $\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h(\mathbf{x})] \leq \varepsilon$ , for any  $\varepsilon, \delta > 0$ , when  $m \geq O\left((d / \varepsilon^2) \log (d / \delta)\right)$ .
The above algorithm is based on estimating the mean of the bag vectors, which unfortunately does not work when  $k = q / 2$  or for a general covariance matrix  $\boldsymbol{\Sigma}$ . We instead use a covariance estimation based approach – albeit with a worse running time – for our next result which is proved in Sec. 4.  $\lambda_{\mathrm{min}}$  and  $\lambda_{\mathrm{max}}$  denote the minimum and maximum eigenvalues of the covariance matrix  $\boldsymbol{\Sigma}$ .
Theorem 1.4. For  $q > 2$ ,  $k \in \{1, \ldots, q - 1\}$ ,  $f(\mathbf{x}) \coloneqq \mathrm{pos}(\mathbf{r}_*^\top \mathbf{x})$ , and positive definite  $\boldsymbol{\Sigma}$  there is an algorithm that samples  $m$  iid bags from  $\mathsf{Ex}(f, N(\mathbf{0}, \boldsymbol{\Sigma}), q, k)$  and runs in  $\mathrm{poly}(m)$  time to produce a hypothesis  $h(\mathbf{x}) \coloneqq \mathrm{pos}(\hat{\mathbf{r}}^\top \mathbf{x})$  s.t. w.p. at least  $1 - \delta$  over the sampling
- if  $k \neq q/2$ ,  $\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h(\mathbf{x})] \leq \varepsilon$ , and
- if  $k = q/2$ ,  $\min \{\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h(\mathbf{x})], \operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq \tilde{h}(\mathbf{x})]\} \leq \varepsilon$ , where  $\tilde{h}(\mathbf{x}) \coloneqq \operatorname{pos}(-\hat{\mathbf{r}}^{\top}\mathbf{x})$
for any  $\varepsilon, \delta > 0$ , when  $m \geq O((d / \varepsilon^4) \log (d / \delta) (\lambda_{\max} / \lambda_{\min})^6 q^8)$ .
Our general result stated below (proved in Appendix C), extends our algorithmic methods to the case of non-centered Gaussian space and non-homogeneous LTFs.
Theorem 1.5. For  $q > 2$ ,  $k \in \{1, \ldots, q - 1\}$ ,  $f(\mathbf{x}) \coloneqq \mathrm{pos}(\mathbf{r}_*^\top \mathbf{x} + c_*)$ , and positive definite  $\boldsymbol{\Sigma}$  there is an algorithm that samples  $m$  iid bags from  $\mathsf{Ex}(f, N(\boldsymbol{\mu}, \boldsymbol{\Sigma}), q, k)$  and runs in  $\mathrm{poly}(m)$  time to produce a hypothesis  $h(\mathbf{x}) \coloneqq \mathrm{pos}(\hat{\mathbf{r}}^\top \mathbf{x} + \hat{c})$  s.t. w.p. at least  $1 - \delta$  over the sampling
- if  $k \neq q/2$ ,  $\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h(\mathbf{x})] \leq \varepsilon$ , and
- if  $k = q/2$ , min  $\left\{\operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq h(\mathbf{x})], \operatorname{Pr}_{\mathcal{D}}[f(\mathbf{x}) \neq \tilde{h}(\mathbf{x})]\right\} \leq \varepsilon, \tilde{h}(\mathbf{x}) := \operatorname{pos}(-\hat{\mathbf{r}}^{\mathsf{T}}\mathbf{x} - \hat{c})$
for any  $\varepsilon, \delta > 0$ , when  $m \geq O\left((d / \varepsilon^4) \frac{O(\ell^2)}{(\Phi(\ell)(1 - \Phi(\ell)))^2} \log(d / \delta) \left( \frac{\lambda_{\max}}{\lambda_{\min}} \right)^4 \left( \frac{\sqrt{\lambda_{\max}} + \| \boldsymbol{\mu} \|_2}{\sqrt{\lambda_{\min}}} \right)^4 q^8\right)$  where  $\Phi(.)$  is the standard Gaussian cdf and  $\ell = -\frac{c_* + \mathbf{r}_*^\mathrm{T} \boldsymbol{\mu}}{\|\boldsymbol{\Sigma}^{1/2} \mathbf{r}_* \|_2}$
The value of  $\hat{\mathbf{r}}$  output by our algorithms is a close estimate of  $\mathbf{r}_*$  (or possibly  $-\mathbf{r}_*$  in the case of balanced bags). Note that our algorithms do not require knowledge of  $\mu$  or  $\boldsymbol{\Sigma}$ , and only the derived parameters in Thms. 1.4 and 1.5 are used for the sample complexity bounds. They are based on the certain properties of the empirical mean-vectors and covariance matrices formed by sampling vectors or pairs of vectors from random bags of the bag oracle. An empirical mean-based approach has been previously developed by [23] in the LLP setting to estimate the parameters of an exponential generative model, when bag distributions satisfy the so-called class conditioned independencei.e., given its label, the feature-vector distribution is same for all the bags. These techniques were extended by [22] to linear classifiers with loss functions satisfying certain smoothness conditions. While the bag oracle in our setup satisfies such conditioned independence, we aim to minimize the instance classification error on which the techniques of [23, 22] are not applicable.
For the case when  $q = 1$  (ordinary classification), the sample complexity is  $O(d / \varepsilon \log (d / \delta))$  as one can solve a linear program to obtain an LTF and then use uniform convergence to bound the generalization error. The sample complexity expressions in Theorems 1.3, 1.4 and 1.5 have the same dependence on  $d$  and  $\delta$ . However, they have higher powers of  $1 / \varepsilon$ . They also include other parameters like the bag size ( $q$ ), condition number of  $\Sigma (\lambda_{\mathrm{max}} / \lambda_{\mathrm{min}})$  and the normalized distance of mean of the Gaussian to the LTF ( $l$ ). The origins and significance of these discrepancies are discussed in Sec. 1.4.
1.4 Our Techniques
Theorem 1.3: Case  $N(0,\mathbf{I})$ ,  $f(\mathbf{x}) = \mathrm{pos}(\mathbf{r}_*^\top \mathbf{x})$ ,  $k \neq q/2$ . Assume that  $k > q/2$ . A randomly sampled bag with label proportion  $k/q$  has  $k$  vectors iid sampled from the positive side of the separating hyperplane passing through origin, and  $(q - k)$  iid sampled from its negative side. It is easy to see that the expected sum of the vectors vanishes in all directions orthogonal to the normal vector  $\mathbf{r}_*$ , and in the direction of  $\mathbf{r}_*$  it has a constant magnitude. The case of  $k < q/2$  is analogous with the direction of the expectation opposite to  $\mathbf{r}_*$ . Sampling a sufficient number of bags and a random vector from each of them, and taking their normalized expectation (negating if  $k < q/2$ ) yields the a close estimate  $\hat{\mathbf{r}}$  of  $\mathbf{r}_*$ , which in turn implies low classification error. The sample complexity is the same as that for mean estimation bag-vectors (see Section 3) and thus the power of  $1/\varepsilon$  is 2.
This simple approach however does not work when  $r = q / 2$ , in which case the expectation vanishes completely, or for general Gaussian distributions which (even if centered) could be skewed in arbitrary directions. We present our variance based method to handle these cases.
Theorem 1.4: Case  $N(\mathbf{0}, \mathbf{\Sigma})$ ,  $f(\mathbf{x}) = \mathrm{pos}(\mathbf{r}_*^\top \mathbf{x})$ . To convey the main idea of our approach, consider two different ways of sampling two feature-vectors from a random bag of the oracle. The first way is to sample two feature-vectors  $\mathbf{Z}_1, \mathbf{Z}_2$  independently and u.a.r from a random bag. In this case, the probability that they have different labels (given by  $f$ ) is  $2k(q - k) / q^2$ . The second way is to sample a random pair  $\tilde{\mathbf{Z}}_1, \tilde{\mathbf{Z}}_2$  of feature-vectors i.e., without replacement. In this case, the probability of different labels is  $2k(q - k) / (q(q - 1))$  which is strictly greater than  $2k(q - k) / q^2$ . Since the labels are given by thresholding in the direction of  $\mathbf{r}_*$ , this suggests that the variance of  $(\tilde{\mathbf{Z}}_1 - \tilde{\mathbf{Z}}_2)$  w.r.t. that of  $(\mathbf{Z}_1 - \mathbf{Z}_2)$  is maximized in the direction of  $\mathbf{r}_*$ . Indeed, let  $\boldsymbol{\Sigma}_D := \operatorname{Var}[\tilde{\mathbf{Z}}_1 - \tilde{\mathbf{Z}}_2]$  be the pair covariance matrix and let  $\boldsymbol{\Sigma}_B := \operatorname{Var}[\mathbf{Z}_1] = (1/2)\operatorname{Var}[\mathbf{Z}_1 - \mathbf{Z}_2]$  be the bag covariance matrix. Then we show that  $\pm \mathbf{r}_* = \operatorname{argmax}_{\mathbf{r}} \rho(\mathbf{r})$  where  $\rho(\mathbf{r}) := \frac{\mathbf{r}^\top \boldsymbol{\Sigma}_D \mathbf{r}}{\mathbf{r}^\top \boldsymbol{\Sigma}_B \mathbf{r}}$ . A simple transformation gives us that
$$
\pm \mathbf {r} _ {*} = \boldsymbol {\Sigma} _ {B} ^ {- 1 / 2} \text {P r i n c i p a l E i g e n V e c t o r} \left(\boldsymbol {\Sigma} _ {B} ^ {- 1 / 2} \boldsymbol {\Sigma} _ {D} \boldsymbol {\Sigma} _ {B} ^ {- 1 / 2}\right) \tag {1}
$$
This suggests the following algorithm: sample enough bags to construct the corresponding empirical estimates  $\hat{\Sigma}_D$  and  $\hat{\Sigma}_B$  and then compute the empirical proxy of the RHS of (1). We show that using close enough empirical estimates w.h.p the algorithm computes a vector  $\hat{\mathbf{r}}$  s.t. one of  $\pm \hat{\mathbf{r}}$  is close to  $\mathbf{r}_{*}$ , and via a geometric stability argument this implies that one of  $\pm \hat{\mathbf{r}}$  yields an LTF that has small instance-level classification error.
At this point, if  $k = q / 2$ , there is no way to identify the correct solution from  $\pm \hat{\mathbf{r}}$ , since a balanced bag, if consistent with an LTF, is also consistent with its complement. On the other hand, if  $k \neq q / 2$  we can obtain the correct solution as follows. It is easy to show that since  $f$  is homogeneous and the instance distribution is a centered Gaussian, the measure of  $\{\mathbf{x} \mid f(\mathbf{x}) = a\}$  is  $1 / 2$  for  $a = \{0,1\}$ . Thus, one of  $h(\mathbf{x}) := \mathrm{pos}(\hat{\mathbf{r}}^{\top}\mathbf{x})$ ,  $\tilde{h}(\mathbf{x}) = \mathrm{pos}(-\hat{\mathbf{r}}^{\top}\mathbf{x})$  will have a high bag satisfaction accuracy. Thus, a large enough sample of bags can be used to identify one of  $h$ ,  $\tilde{h}$  having a high bag satisfaction
accuracy. Lastly, we use a novel generalization error bound (see below) to show that the identified LTF also has a high instance classification accuracy.
The algorithm incurs a sample complexity of  $O\left((q\lambda_{\max} / (\varepsilon \lambda_{\min}))^4\right)$  to estimate  $\Sigma_D$  and  $\Sigma_B$  accurately so that the distance between  $\hat{\mathbf{r}}$  (or  $-\hat{\mathbf{r}}$ ) and  $\mathbf{r}_*$  is sufficiently bounded. (see Lemmas 3.1 and 4.1). In fact, the distance is required to be  $O\left((\varepsilon / q)\sqrt{\lambda_{\min} / \lambda_{\max}}\right)$  to translate the geometric bound into an  $\varepsilon$ -misclassification error bound, thereby incurring a further factor of  $O\left(q^4 (\lambda_{\max} / \lambda_{\min})^2\right)$  in the sample complexity (see Lemma 2.3). The higher powers in the dependencies are mainly due to the second moment estimates which degrade with larger values of  $(\lambda_{\max} / \lambda_{\min})$ . Note that the sample complexity explicitly depends on the bag size  $q$  (and not just the label proportion  $k / q$ ). This is because the probability of sampling (without replacement from a bag) a pair of differently labeled feature-vectors is  $2(k / q)(1 - k / q) / (1 - 1 / q)$ . Keeping  $k / q$  the same, this probability decreases with increasing bag size, thereby increasing the sample complexity for larger bags.