Title: PAC Learning Linear Thresholds from Label Proportions
Venue: NeurIPS 2023
Accepted: yes
Keywords: PAC learning

Learning from label proportions

Linear thresholds
Abstract: Learning from label proportions (LLP) is a generalization of supervised learning in which the training data is available as sets or bags of feature-vectors (instances) along with the average instance-label of each bag. The goal is to train a good instance classifier. While most previous works on LLP have focused on training models on such training data, computational learnability of LLP was only
recently explored by Saket (2021, 2022) who showed worst case intractability of properly learning linear threshold functions (LTFs) from label proportions. However, their work did not rule out efficient algorithms for this problem for natural distributions.

In this work we show that it is indeed possible to efficiently learn LTFs using LTFs when given access to random bags of some label proportion in which feature-vectors are, conditioned on their labels, independently sampled from a Gaussian distribution $N(µ, Σ)$. Our work shows that a certain matrix – formed using covariances of the differences of feature-vectors sampled from the bags with and without replacement – necessarily has its principal component, after a transformation, in the direction of the normal vector of the LTF. Our algorithm estimates the means and covariance matrices using subgaussian concentration bounds which we show can be applied to efficiently sample bags for approximating the normal direction. Using this in conjunction with novel generalization error bounds in the bag setting, we show that a low error hypothesis LTF can be identified. For some special cases of the $N(0, I)$ distribution we provide a simpler mean estimation based algorithm. We include an experimental evaluation of our learning algorithms along with a comparison with those of Saket (2021, 2022) and random LTFs, demonstrating the effectiveness of our techniques.

PAC Learning Linear Thresholds from Label Proportions

Abstract

Learning from label proportions (LLP) is a generalization of supervised learning in which the training data is available as sets or bags of feature-vectors (instances) along with the average instance-label of each bag. The goal is to train a good instance classifier. While most previous works on LLP have focused on training models on such training data, computational learnability of LLP was only recently explored by [25, 26] who showed worst case intractability of properly learning linear threshold functions (LTFs) from label proportions. However, their work did not rule out efficient algorithms for this problem on natural distributions.

In this work we show that it is indeed possible to efficiently learn LTFs using LTFs when given access to random bags of some label proportion in which featurevectors are, conditioned on their labels, independently sampled from a Gaussian distribution N ( µ , Σ ) . Our work shows that a certain matrix - formed using covariances of the differences of feature-vectors sampled from the bags with and without replacement - necessarily has its principal component, after a transformation, in the direction of the normal vector of the LTF. Our algorithm estimates the means and covariance matrices using subgaussian concentration bounds which we show can be applied to efficiently sample bags for approximating the normal direction. Using this in conjunction with novel generalization error bounds in the bag setting, we show that a low error hypothesis LTF can be identified. For some special cases of the N ( 0 , I ) distribution we provide a simpler mean estimation based algorithm. We include an experimental evaluation of our learning algorithms along with a comparison with those of [25, 26] and random LTFs, demonstrating the effectiveness of our techniques.

1 Introduction

In learning from label proportions (LLP), the training data is aggregated into sets or bags of featurevectors (instances). For each bag we are given its constituent feature-vectors along with only the sum or average of their labels The goal is a to obtain a good instance-level classifier - one that minimizes the classification error on a test set of instances or bags. In this work we study the LLP learnability over Gaussian distributions of linear threshold functions (LTFs), also called linear classifiers or halfspaces , given by f ( x ) = pos ( r T x + c ) where pos ( a ) := 1 if a > 0 and 0 otherwise.

The probably approximately correct (PAC) model of [29] states that a concept class C of { 0 , 1 } -valued functions can be learnt by a hypothesis class H if there is an algorithm to efficiently obtain, using iid samples from a distribution on ( x , f ( x )) , a hypothesis h ∈ H of arbitrarily high accuracy on that distribution, for any unknown f ∈ C . If H = C we say that C is properly learnable, for

e.g. LTFs are known to be properly learnable using linear programming ([3]). This notion can be extended to the LLP setting - which for brevity we call PAC-LLP - as follows: distribution D is over bags and their label proportions ( B,σ ( B,f )) where B = { x 1 , . . . , x q } is a bag of feature vectors and σ ( B,f ) = Avg { f ( x ) | x ∈ B } . A bag ( B,σ ( B,f )) is said to be satisfied by h iff σ ( B,h ) = σ ( B,f ) , and the accuracy of h is the fraction of bags satisfied by it.

With the above notion of PAC-LLP, [25] studied the learnability of LTFs and rather disturbingly showed that for any constant ε > 0 it is NP-hard to PAC-LLP learn an LTF using an LTF which satisfies (1 / 2 + ε ) -fraction of the bags when all bags are of size at most 2 , which was subsequently strengthened to a (4 / 9 + ε ) -factor hardness by [26] who also proved (1 /q + ε ) -factor hardness when the bag size is at most q , for any q ≥ 3 . This is in contrast to the supervised learning (i.e, with unit-sized bags) in which an LTF can be efficiently learnt by an LTF using linear programming. On the algorithmic side, [25] gave a semi-definite programming (SDP) based algorithm to find an LTF satisfying (2 / 5) -fraction of bags of size ≤ 2 , which was extended by [26] to a fairly involved SDP yielding (1 / 12) -approximation for bags of size ≤ 3 , while no non-trivial algorithms for bags of size > 3 are known. These results show that PAC-LLP learning LTFs using LTFs is intractable on hard bag distributions, and even the non-trivial algorithms for bag sizes ≤ 3 are via complicated convex programming techniques. A natural question therefore, is whether the problem is tractable on natural distributions that may arise out of real world scenarios.

We answer the above question in the affirmative when the feature-vectors are distributed according to some (unknown) Gaussian distribution D = N ( µ , Σ ) in d -dimensions. Gaussian distributions are ubiquitous in machine learning and in many applications the input data distribution is modeled as multivariate Gaussians, and several previous works [8, 30] have studied learnability in Gaussian distributions. An unkown target LTF is given by f ( x ) := pos ( r T ∗ x + c ∗ ) where ‖ r ∗ ‖ 2 = 1 . Let D a be the distribution of x ← D conditioned on f ( x ) = a , for a ∈ { 0 , 1 } . Using this we formalize the notion of a distribution O on bags of size q and average label k/q : a random bag B sampled from O consists of k iid samples from D 1 and ( q -k ) iid samples from D 0 . The case of k ∈ { 0 , q } is uninteresting as all instances in such bags are either labeled 0 or 1 and traditional PAC-learning for LTFs can be employed directly. Unlike [25, 26] our objective is to directly maximize the instance-level level accuracy on D . With this setup we informally describe our main result.

Our PAC-LLP LTF Learner (Informal): Assuming mild conditions on Σ , µ and c ∗ , for any q, k ∈ Z + s.t. 1 ≤ k ≤ q -1 and ε, δ > 0 , there is an algorithm that samples at most m bags from O and runs in time O ( t + m ) and with probability 1 -δ produces an LTF h s.t.

glyph[negationslash]

Pr D [ f ( x ) = h ( x )] ≤ ε if k = q/ 2 , and glyph[negationslash]

glyph[negationslash]

Pr D [ f ( x ) = h ( x )] ≤ ε or Pr[ f ( x ) = (1 -h ( x ))] ≤ ε if k = q/ 2 , glyph[negationslash]

glyph[negationslash]

where t, m are fixed polynomials in d, q, (1 /ε ) , log(1 /δ ) . We also obtain a more efficient algorithm when k = q/ 2 , µ = 0 , c ∗ = 0 and Σ = I . The ambiguity in the case of k = q/ 2 is inherent since bags of label proportion 1 / 2 consistent with an LTF f ( x ) are also consistent with (1 -f ( x )) .

Remark 1.1 (Mixtures of ( q, k ) ) . The training data could consist of bags of different sizes and label proportions, however typically the the maximum size of bags is bounded by (say) Q , and in a large enough sample we would have at least (1 /Q 2 ) -fraction of bags of a particular size and label proportion and we can apply our PAC-LLP LTF Learner above to that subsample.

1.1 Related Work

The LLP problem is motivated by many real applications where labels are available not for each feature-vector but only as the average labels of bags of feature-vectors. This may occur because of privacy and legal ([24, 33])) reasons, supervision cost ([5]) or lack of labeling instrumentation ([10]). Previous works ([9, 15, 20, 24]) on LLP have applied techniques such as such as clustering, and linear classifiers and MCMC. Specifically for LLP, assuming class conditional independence of bags, [23] gave an algorithm to learn an exponential generative model, which was further generalized by [22]. On the other hand, the work of [34] proposed a novel proportional SVM based algorithms which optimized the SVM loss over instance-labels which were constrained by bag-level loss w.r.t the given label-proportions. Subsequently, approaches based on deep neural nets for large-scale and multi-class data ([18, 11, 19, 21]), as well as bag pre-processing techniques ([28, 27]) have been developed. Recently, [4, 6] have proposed model training methods for either random or curated bags.

The LLP framework (as an analogue of PAC learning) was first formalized in the work of [35]. They bounded the generalization error of a trained classifier when taking the ( bag, label-proportion ) -pairs

as instances sampled iid from some distribution. Their loss function was different - a weaker notion than the strict bag satisfaction predicate of [25, 26]. A single-bag variant class ratio estimation - of LLP was studied by [13] in which learning LTFs has a simple algorithm (see Appendix G). Nevertheless, the study of computational learning in the LLP framework has been fairly limited, apart from the works of [25, 26] whose results of learning LTFs in the LLP setting have been described earlier in this section.

In the fully supervised setting [3] showed that LTFs can be learnt using LTFs via linear programming without any distributional assumptions. Adversarial label noise makes the problem NP-hard to approximate beyond the trivial ( 1 / 2 ) -factor even using constant degree polynomial thresholds as hypothesis ([12, 14, 2]). However, under distributional assumptions a series of results ([16, 17, 1, 7]) have given efficient algorithms to learn adversarially noisy LTFs.

Next, Sec. 1.2 mathematically defines our problem statement. Sec. 1.3 states the main results of this paper. Sec. 1.4 provides an overview of our techniques. Sec. 2 mentions some preliminary results which are used in our proofs. Sec. 3 defines and analyses a subroutine which we use in all our algorithms. Sec. 4 provides a complete proof for one of our main results. Sec 5 gives brief proof sketches of our other results. Sec. 6 mentions some experiments which support of our results.

1.2 Problem Definition

Definition 1.2 (Bag Oracle) . Given distribution D over R d and a target concept f : R →{ 0 , 1 } , the bag oracle for size q and label proportion k/q ( 1 ≤ k ≤ q -1 ), denoted by Ex ( f, D , q, k ) , generates a bag { x ( i ) } q i =1 such that x ( i ) is independently sampled from (i) D f, 1 for i = { 1 , . . . , k } , and (ii) D f, 0 for i = { k +1 , . . . , q } , where D f,a is x ← D conditioned on f ( x ) = a , for a ∈ { 0 , 1 } .

1.3 Our results

We first state our result (proved in Appendix A) for the case of standard d -dimensional Gaussian distribution N ( 0 , I ) , homogeneous target LTF and unbalanced bags.

glyph[negationslash]

Theorem 1.3. For q > 2 and k ∈ { 1 , . . . , q -1 } s.t. k = q/ 2 and LTF f ( x ) := pos ( r T ∗ x ) , there is an algorithm that samples m iid bags from Ex ( f, N ( 0 , I ) , q, k ) and runs in O ( m ) time to produce a hypothesis h ( x ) := pos (ˆ r T x ) s.t. w.p. at least 1 -δ over the sampling, Pr D [ f ( x ) = h ( x )] ≤ ε , for any ε, δ > 0 , when m ≥ O ( ( d/ε 2 ) log( d/δ ) ) .

glyph[negationslash]

The above algorithm is based on estimating the mean of the bag vectors, which unfortunately does not work when k = q/ 2 or for a general covariance matrix Σ . We instead use a covariance estimation based approach - albeit with a worse running time - for our next result which is proved in Sec. 4. λ min and λ max denote the minimum and maximum eigenvalues of the covariance matrix Σ .

Theorem 1.4. For q > 2 , k ∈ { 1 , . . . , q -1 } , f ( x ) := pos ( r T ∗ x ) , and positive definite Σ there is an algorithm that samples m iid bags from Ex ( f, N ( 0 , Σ ) , q, k ) and runs in poly ( m ) time to produce a hypothesis h ( x ) := pos (ˆ r T x ) s.t. w.p. at least 1 -δ over the sampling glyph[negationslash]

if k = q/ 2 , Pr D [ f ( x ) = h ( x )] ≤ ε , and

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

for any ε, δ > 0 , when m ≥ O (( d/ε 4 ) log( d/δ )( λ max /λ min ) 6 q 8 ) .

Our general result stated below (proved in Appendix C), extends our algorithmic methods to the case of non-centered Gaussian space and non-homogeneous LTFs.

Theorem 1.5. For q > 2 , k ∈ { 1 , . . . , q -1 } , f ( x ) := pos ( r T ∗ x + c ∗ ) , and positive definite Σ there is an algorithm that samples m iid bags from Ex ( f, N ( µ , Σ ) , q, k ) and runs in poly ( m ) time to produce a hypothesis h ( x ) := pos (ˆ r T x +ˆ c ) s.t. w.p. at least 1 -δ over the sampling glyph[negationslash]

if k = q/ 2 , Pr D [ f ( x ) = h ( x )] ≤ ε , and

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

The value of ˆ r output by our algorithms is a close estimate of r ∗ (or possibly -r ∗ in the case of balanced bags). Note that our algorithms do not require knowledge of µ or Σ , and only the derived parameters in Thms. 1.4 and 1.5 are used for the sample complexity bounds. They are based on the certain properties of the empirical mean-vectors and covariance matrices formed by sampling vectors or pairs of vectors from random bags of the bag oracle. An empirical mean based approach has been previously developed by [23] in the LLP setting to estimate the parameters of an exponential generative model, when bag distributions satisfy the so-called class conditioned independencei.e., given its label, the feature-vector distribution is same for all the bags. These techniques were extended by [22] to linear classifiers with loss functions satisfying certain smoothness conditions. While the bag oracle in our setup satisfies such conditioned independence, we aim to minimize the instance classification error on which the techniques of [23, 22] are not applicable.

For the case when q = 1 (ordinary classification), the sample complexity is O ( d/ε log( d/δ )) as one can solve a linear program to obtain an LTF and then use uniform convergence to bound the generalization error. The sample complexity expressions in Theorems 1.3, 1.4 and 1.5 have the same dependence on d and δ . However, they have higher powers of 1 /ε . They also include other parameters like the bag size ( q ), condition number of Σ ( λ max /λ min ) and the normalized distance of mean of the Gaussian to the LTF ( l ). The origins and significance of these discrepancies are discussed in Sec. 1.4.

1.4 Our Techniques

Theorem 1.3: Case N ( 0 , I ) , f ( x ) = pos ( r T ∗ x ) , k = q/ 2 . Assume that k > q/ 2 . A randomly sampled bag with label proportion k/q has k vectors iid sampled from the positive side of the separating hyperplane passing through origin, and ( q -k ) iid sampled from its negative side. It is easy to see that the expected sum of the vectors vanishes in all directions orthogonal to the normal vector r ∗ , and in the direction of r ∗ it has a constant magnitude. The case of k < q/ 2 is analogous with the direction of the expectation opposite to r ∗ . Sampling a sufficient number of bags and a random vector from each of them, and taking their normalized expectation (negating if k < q/ 2 ) yields the a close estimate ˆ r of r ∗ , which in turn implies low classification error. The sample complexity is the same as that for mean estimation bag-vectors (see Section 3) and thus the power of 1 /ε is 2 .

glyph[negationslash]

This simple approach however does not work when r = q/ 2 , in which case the expectation vanishes completely, or for general Gaussian distributions which (even if centered) could be skewed in arbitrary directions. We present our variance based method to handle these cases.

Theorem 1.4: Case N ( 0 , Σ ) , f ( x ) = pos ( r T ∗ x ) . To convey the main idea of our approach, consider two different ways of sampling two feature-vectors from a random bag of the oracle. The first way is to sample two feature-vectors Z 1 , Z 2 independently and u.a.r from a random bag. In this case, the probability that they have different labels (given by f ) is 2 k ( q -k ) /q 2 . The second way is to sample a random pair ˜ Z 1 , ˜ Z 2 of feature-vectors i.e., without replacement. In this case, the probability of different labels is 2 k ( q -k ) / ( q ( q -1)) which is strictly greater than 2 k ( q -k ) /q 2 . Since the labels are given by thresholding in the direction of r ∗ , this suggests that the variance of ( ˜ Z 1 -˜ Z 2 ) w.r.t. that of ( Z 1 -Z 2 ) is maximized in the direction of r ∗ . Indeed, let Σ D := Var [ ˜ Z 1 -˜ Z 2 ] be the pair covariance matrix and let Σ B := Var [ Z 1 ] = (1 / 2) Var [ Z 1 -Z 2 ] be the bag covariance matrix. Then we show that ± r ∗ = argmax r ρ ( r ) where ρ ( r ) := r T Σ D r r T Σ B r . A simple transformation gives us that

This suggests the following algorithm: sample enough bags to construct the corresponding empirical estimates ˆ Σ D and ˆ Σ B and then compute the empirical proxy of the RHS of (1). We show that using close enough empirical estimates w.h.p the algorithm computes a vector ˆ r s.t. one of ± ˆ r is close to r ∗ , and via a geometric stability argument this implies that one of ± ˆ r yields an LTF that has small instance-level classification error.

glyph[negationslash]

At this point, if k = q/ 2 , there is no way to identify the correct solution from ± ˆ r , since a balanced bag, if consistent with an LTF, is also consistent with its complement. On the other hand, if k = q/ 2 we can obtain the correct solution as follows. It is easy to show that since f is homogeneous and the instance distribution is a centered Gaussian, the measure of { x | f ( x ) = a } is 1 / 2 for a = { 0 , 1 } . Thus, one of h ( x ) := pos (ˆ r T x ) , ˜ h ( x ) = pos ( -ˆ r T x ) will have a high bag satisfaction accuracy. Thus, a large enough sample of bags can be used to identify one of h, ˜ h having a high bag satisfaction

accuracy. Lastly, we use a novel generalization error bound (see below) to show that the identified LTF also has a high instance classification accuracy.

The algorithm incurs a sample complexity of O ( ( qλ max / ( ελ min )) 4 ) to estimate Σ D and Σ B accurately so that the distance between ˆ r (or -ˆ r ) and r ∗ is sufficiently bounded. (see Lemmas 3.1 and 4.1). In fact, the distance is required to be O ( ( ε/q ) √ λ min /λ max ) to translate the geometric bound into an ε -misclassification error bound, thereby incurring a further factor of O ( q 4 ( λ max /λ min ) 2 ) in the sample complexity (see Lemma 2.3). The higher powers in the dependencies are mainly due to the second moment estimates which degrade with larger values of ( λ max /λ min ) . Note that the sample complexity explicitly depends on the bag size q (and not just the label proportion k/q ). This is because the probability of of sampling (without replacement from a bag) a pair of differently labeled feature-vectors is 2( k/q )(1 -k/q ) / (1 -1 /q ) . Keeping k/q the same, this probability decreases with increasing bag size, thereby increasing the sample complexity for larger bags.

Theorem 1.5: Case N ( µ , Σ ) , f ( x ) = pos ( r T ∗ x + c ∗ ) . We show that (1) also holds in this case, and therefore we use a similar approach of empirically estimating the pair and bag covariance matrices solving (1) works in principle. However, there are complications, in particular the presence of µ and c ∗ degrades the error bounds in the analysis, thus increasing the sample complexity of the algorithm. This is because the measures of { x | f ( x ) = a } for a = { 0 , 1 } could be highly skewed if ‖ µ ‖ 2 and/or | c ∗ | is large. Moreover, the spectral algorithm only gives a solution ± ˆ r for r ∗ . An additional step is required to obtain an estimate of c ∗ . This we accomplish using the following procedure which, given a sample of s bags and any r outputs a ˆ c which has the following property: if s ∗ = max c { no. of bags satisfied by pos ( r T x + c ) } , then ˆ c will satisfy at least s ∗ -1 bags. This is done by ordering the values r T x of the vectors x within each bag in decreasing order, and then constructing set of the k th values of each bag. Out of these s values, the one which taken as c in pos ( r T x + c ) satisfies the most bags, is chosen to be ˆ c .

Due to the non-homogeneity of the Gaussian distribution and the LTF f , the application of Lemma 2.3 instead incurs an O ( q 4 (( √ λ max + ‖ µ ‖ 2 ) / √ λ min ) 4 ) factor in the sample complexity, while a factor of O ( glyph[lscript] 2 ) is towards the estimation of Σ B and Σ D . Note that glyph[lscript] is the distance from µ to the hyperplane of f normalized by the stretch induced by Σ , and thus a larger value of glyph[lscript] implies a lower density near the hyperplane leading to an increased sample complexity. Lastly, a further blowup by 1 / (Φ( glyph[lscript] )(1 -Φ( glyph[lscript] ))) 2 comes from bounding the sample error from geometric bound between ˆ r and r ∗ , and is required for a sufficiently accurate approximation of c ∗ .

glyph[negationslash]

Generalization Error Bounds. We prove (Thm. 2.2) bounds on the generalization of the error of a hypothesis LTF h in satisfying sampled bags to its distributional instance-level error. Using this, we are able to distinguish (for k = q/ 2 ) between the two possible solutions our principal component algorithm yields - the one which satisfies more of the sampled bags has w.h.p. low instance-level error. For proving these bounds, the first step is to use a bag-level generalization error bound shown by [26] using the techniques of [35]. Next, we show that low distributional bag satisfaction error by h implies low instance level error. This involves a fairly combinatorial analysis of two independent binomial random variables formed from the incorrectly classified labels within a random bag. Essentially, unless h closely aligns with f at the instance level, with significant probability there will be an imbalance in these two random variables leading to h not satisfying the bag.

Subgaussian concentration bounds. The standard estimation bounds for Gaussians are not directly applicable in our case, since the random vector sampled from a random bag is biased according to its label given by f , and is therefore not a Gaussian vector. To obtain sample complexity bounds linear in log(1 /δ ) we use subgaussian concentration bounds for mean and covariance estimation ([32, 31]). For this, we show O ( glyph[lscript] ) bound on the expectation and subgaussian norm of the thresholded Gaussian given by { g ∼ N (0 , 1) | g > glyph[lscript] } for some glyph[lscript] > 0 . The random vectors of interest to us are (in a transformed space) distributed as a combination of thresholded Gaussians in one of the coordinates, and N (0 , 1) in the rest. We show that they satisfy the O ( glyph[lscript] ) bound on their subgaussian norm and admit the corresponding subgaussian Hoeffding (for empirical mean) and empirical covariance concentration bounds. Based on this, in Sec. 3 we abstract out the procedure used in our learning algorithms for obtaining the relevant mean and covariance estimates.

Experiments. We include in Sec. 6 an experimental evaluation of our learning algorithms along with a comparison of with those of [25, 26] and random LTFs, demonstrating the effectiveness of our techniques.

2 Preliminaries

We begin with some useful linear algebraic notions. Let λ max ( A ) and λ min ( A ) denote the maximum and minimum eigenvalue of a real symmetric matrix A . The operator norm ‖ A ‖ 2 := max ‖ x ‖ 2 =1 ‖ Ax ‖ 2 for such matrices is given by λ max ( A ) .

We shall restrict our attention to symmetric positive definite (p.d.) matrices A which satisfy x T Ax > 0 for all non-zero vectors x , implying that λ min ( A ) > 0 and A -1 exists and is symmetric p.d. as well. Further, for such matrices A , A 1 / 2 is well defined to be the unique symmetric p.d. matrix B satisfying BB = A . The eigenvalues of A 1 / 2 are the square-roots of those of A . We have the following lemma which is proved in Appendix B.6.

Bag Oracle and related statistics. Let O := Ex ( f, D , q, k ) be any bag oracle with k ∈ { 1 , . . . , q -1 } for an LTF f ( x ) := r T ∗ x + c ∗ in d -dimensions, and let M be a collection of m bags sampled iid from the oracle. Define for any hypothesis LTF h ,

We define the following statistical quantities related to O . Let X be a random feature-vector sampled uniformly from a random bag sampled from O . Let,

Now, let Z = X 1 -X 2 where ( X 1 , X 2 ) are a random pair of feature-vectors sampled (without replacement) from a random bag sampled from O . Clearly E [ Z ] = 0 . Define

Generalization and stability bounds. We prove in Appendix D.1 the following generalization bound from bag classification error to instance classification error.

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

In some cases we directly obtain geometric bounds on the hypothesis classifier and the following lemma (proved in Appendix D.2) allows us to straightaway bound the classification error.

glyph[negationslash]

Lemma 2.3. Suppose ‖ r -ˆ r ‖ 2 ≤ ε 1 for unit vectors r , ˆ r . Then, Pr [ pos ( r T X + c ) = pos ( ˆ r T X + c )] ≤ ε where ε = ε 1 ( c 0 √ λ max /λ min + c 1 ‖ µ ‖ 2 / √ λ min ) for some absolute constants c 0 , c 1 > 0 and λ max , λ min are the maximum and minimum eigenvalues of Σ respectively.

3 Bag distribution statistics estimation

We provide the following estimator for µ B , Σ B and Σ D defined in (4) and (5). We have the following lemma - which follows from the subgaussian distribution based mean and covariance concentration bounds shown for thresholded Gaussians (see Appendix E) - whose proof is given in Appendix E.3.

Lemma 3.1. If m ≥ O ( ( d/ε 2 ) O ( glyph[lscript] 2 ) log( d/δ ) ) where glyph[lscript] is as given in Lemma E.13 then Algorithm 1 returns ˆ µ B , ˆ Σ B , ˆ Σ D such that ‖ ˆ µ B -µ B ‖ 2 ≤ ε √ λ max / 2 , ‖ ˆ Σ B -Σ B ‖ 2 ≤ ελ max , and ‖ ˆ Σ D -Σ D ‖ 2 ≤ ελ max , w.p. at least 1 -δ , for any ε, δ > 0 . Here λ max is the maximum eigenvalue of Σ .

Algorithm 1 MeanCovsEstimator .

Input: Ex ( f, D = N ( µ , Σ ) , q, k ) , m , where f = pos ( r T X + c )

.

Sample m bags from Ex ( f, D , q, k ) . Let { B i } i m =1 be the sampled bags.

V := { x i | x i u.a.r. ← B i , i ∈ { 1 , . . . , m }} .

ˆ µ B = ∑ x ∈ V x /m | .

ˆ Σ B = Σ x ∈ V ( x -µ B )( x -µ B ) T /m .

Sample m bags from Ex ( f, D , q, k ) . Let { ˜ B i } i m =1 be the sampled bags.

˜ V := { x i = x i -˜ x i | ( x i , ˜ x i ) u.a.r. without replacement from ˜ B i , i ∈ { 1 , . . . , m }} .

ˆ Σ D = Σ z ∈ ˜ V zz T /m .

Return: ˆ µ B , ˆ Σ B , ˆ Σ D .

4 Proof of Theorem 1.4

For the setting of Theorem 1.4, we provide Algorithm 2. It uses as a subroutine a polynomial time procedure PrincipalEigenVector for the principal eigen-vector of a symmetric matrix, and first computes two LTFs given by a normal vector and its negation, returning the one that has lower error on a sampled collection of bags.

Algorithm 2 PAC Learner for no-offset LTFs over N ( 0 , Σ )

Input: O = Ex ( f, D = N ( 0 , Σ ) , q, k ) , m, s , where f ( x ) = pos ( r T x ) , ‖ r ∗ ‖ 2 = 1 .

∗ 1. Compute ˆ Σ B , ˆ Σ D using MeanCovsEstimator with m samples. 2. r = ˆ Σ -1 / 2 B PrincipalEigenVector ( ˆ Σ -1 / 2 B ˆ Σ D ˆ Σ -1 / 2 B ) if ˆ Σ -1 / 2 B exists, else exit . 3. Let ˆ r = r / ‖ r ‖ 2 . 4. If k = q/ 2 return: h = pos ( ˆ r T X ) , else a. Let ˜ h = pos ( -ˆ r T X ) . b. Sample a collection M of s bags from O . c. Return h ∗ ∈ { h, ˜ h } which has lower BagErr sample ( h ∗ , M ) .

Lemma 4.1. For any ε, δ ∈ (0 , 1) , if m ≥ O (( d/ε 4 ) log( d/δ )( λ max /λ min ) 4 q 4 ) , then ˆ r computed in Step 3 of Alg. 2 satisfies min {‖ ˆ r -r ∗ ‖ 2 , ‖ ˆ r + r ∗ ‖ 2 } ≤ ε, w.p. 1 -δ/ 2 .

The above, whose proof is deferred to Sec. 4.1, is used in conjunction with the following lemma.

glyph[negationslash]

Lemma 4.2. Let k = q/ 2 , ε, δ ∈ (0 , 1) and suppose ˆ r computed in Step 3 of Alg. 2 satisfies min {‖ ˆ r -r ∗ ‖ 2 , ‖ ˆ r + r ∗ ‖ 2 } ≤ ε, . Then, with s ≥ O ( d (log q +log(1 /δ )) /ε 2 ) , h ∗ in Step. 3.c satisfies Pr D [ h ∗ ( x ) = f ( x )] ≤ 16 c 0 qε √ λ max λ min w.p. 1 -δ/ 2 , where constant c 0 > 0 is from Lem. 2.3.

glyph[negationslash]

With the above we complete the proof of Theorem 1.4 as follows.

Proof. (of Theorem 1.4) Let the parameters δ, ε be as given in the statement of the theorem.

For k = q/ 2 , we use O ( ε √ λ min /λ max ) for the error bound in Lemma 4.1 thereby taking m = O (( d/ε 4 ) log( d/δ )( λ max /λ min ) 6 q 4 ) in Alg. 2, so that Lemma 4.1 along with Lemma 2.3 yields the desired misclassification error bound of ε for one of h , ˜ h .

glyph[negationslash]

For k = q/ 2 , we use O ( ε √ λ min /λ max /q ) for the error bound in Lemma 4.1. Taking m = O (( d/ε 4 ) log( d/δ )( λ max /λ min ) 6 q 8 ) in Alg. 2 we obtain the following bound: min {‖ ˆ r -r ∗ ‖ 2 , ‖ ˆ r + r ∗ ‖ 2 } ≤ ε √ λ min /λ max / (16 c 0 q ) with probability 1 -δ/ 2 . Using s ≥ O ( d (log q +log(1 /δ )) q 2 λ max ε 2 λ min ) , Lemma 4.2 yields the desired misclassification error bound of ε on h ∗ w.p. 1 -δ .

glyph[negationslash]

Proof. (of Lemma 4.2) Applying Lemma 2.3 we obtain that at least one of h , ˜ h has an instance misclassification error of at most O ( ε √ λ max /λ min ) . WLOG assume that h satisfies this error bound i.e., Pr D [ f ( x ) = h ( x )] ≤ c 0 ε √ λ max /λ min =: ε ′ . Since the separating hyperplane of the LTF f

passes through the origin, and D = N ( 0 , Σ ) is centered, Pr D [ f ( x ) = 1] = Pr D [ f ( x ) = 0] = 1 / 2 . Thus, glyph[negationslash]

glyph[negationslash]

Therefore, the probability that a random bag from the oracle contains a feature vector on which f and h disagree is at most 2 qε ′ . Applying the Chernoff bound (see Appendix B.1) we obtain that with probability at least 1 -δ/ 6 , BagErr sample ( h, M ) ≤ 4 qε ′ . Therefore, in Step 3.c. h ∗ satisfies BagErr sample ( h ∗ , M ) ≤ 4 qε ′

glyph[negationslash]

On the other hand, applying Theorem 2.2, except with probability δ/ 3 , Pr D [ f ( x ) = h ∗ ( x )] ≤ 16 qε ′ = 16 c 0 qε √ λ max /λ min. Therefore, except with probability δ/ 2 , the bound in Lemma 4.2 holds.

4.1 Proof of Lemma 4.1

We define and bound a few useful quantities depending on k, q, λ min and λ max using 1 ≤ k ≤ q -1 .

Definition 4.3. Define, (i) κ 1 := ( 2 k q -1 ) 2 2 π so that 0 ≤ κ 1 ≤ 2 /π , (ii) κ 2 := 1 q -1 k q ( 1 -k q ) 16 π so that 16 πq 2 ≤ κ 2 ≤ 4 π ( q -1) , (iii) κ 3 := κ 2 1 -κ 1 so that 16 πq 2 ≤ κ 3 ≤ 4 ( π -2)( q -1) , and (iv) θ := 2 λ max λ min ( 1 2 -max(0 , 2 κ 1 -κ 2 ) + 1 1 -κ 1 ) so that 3 λ max λ min ≤ θ ≤ 3 λ max (1 -2 /π ) λ min .

For the analysis we begin by showing in the following lemma that ˆ r in the algorithms is indeed ± r ∗ if the covariance estimates were the actual covariances.

Lemma 4.4. The ratio ρ ( r ) := r T Σ D r / r T Σ B r is maximized when r = ± r ∗ . Moreover,

Proof. Let Γ := Σ 1 / 2 , then X ∼ N ( 0 , Σ ) ⇔ X = ΓZ where Z ∼ N ( 0 , I ) . Further, pos ( r T X ) = pos ( u T Z ) where u = Γr / ‖ Γr ‖ 2 . Using this, we can let X B = ΓZ B as a random feature-vector sampled uniformly from a random bag sampled from O . Also, let X D = ΓZ D be the difference of two random feature vectors sampled uniformly without replacement from a random bag sampled from O . Observe that the ratio ρ ( r ) = Var[ r T X D ] / Var[ r T X B ] = Var[ u T Z D ] / Var[ u T Z B ] .

Let u ∗ := Γr ∗ / ‖ Γr ∗ ‖ 2 , and g ∗ := u T ∗ Z which is N (0 , 1) . For a ∈ { 0 , 1 } , let Z a be Z conditioned on pos ( u T ∗ Z ) = a . Let g ∗ a := u T ∗ Z a , a ∈ { 0 , 1 } , be the half normal distributions satisfying E [( g ∗ a ) 2 ] = 1 and E [ g ∗ a ] = ( -1) 1 -a √ 2 /π . With this setup, letting g ∗ B := u T ∗ Z B and g ∗ D := u T ∗ Z D we obtain (using Lemma B.2 in Appendix B.2)

Now let ˜ u be a unit vector orthogonal to u ∗ . Let ˜ g = ˜ u T Z be N (0 , 1) . Also, let ˜ g a = ˜ u T Z a for a ∈ { 0 , 1 } . Since Z a are given by conditioning Z only along u ∗ , ˜ g a ∼ N (0 , 1) for a ∈ { 0 , 1 } . In particular, the component along ˜ u of Z B (call it ˜ g B ) is N (0 , 1) and that of Z D (call it ˜ g D ) is the difference of two iid N (0 , 1) variables. Thus, Var[˜ g B ] = 1 and Var[˜ g D ] = 2 . Moreover, due to orthogonality all these gaussian variables corresponding to ˜ u are independent of those corresponding to u ∗ defined earlier. Now let u = α u ∗ + β ˜ u , where β = √ 1 -α 2 be any unit vector. From the above we have,

where the last equality uses β = √ 1 -α 2 . Letting u = Γr / ‖ Γr ‖ 2 we obtain that α = 〈 Γr , Γr ∗ 〉 ‖ Γr ‖ 2 ‖ Γr ∗ ‖ 2 = γ ( r ) completing the proof.

We now complete the proof of Lemma 4.1 (with δ instead of δ/ 2 for convenience). By Lemma 3.1, taking m ≥ O ( ( d/ε 2 1 ) log( d/δ ) ) ensures that ‖ E B ‖ 2 ≤ ε 1 λ max and ‖ E D ‖ 2 ≤ ε 1 λ max w.p. at least 1 -δ where E B = ˆ Σ B -Σ B and E D = ˆ Σ D -Σ D . We start by defining ˆ ρ ( r ) := r T ˆ Σ D r r T ˆ Σ B r which is the equivalent of ρ using the estimated matrices. Observe that it can be written as ˆ ρ ( r ) = r T Σ B r + r T E B r r T Σ D r + r T E D r . Using these we can obtain the following bound on ˆ ρ : for any r ∈ R d , | ˆ ρ ( r ) -ρ ( r ) | ≤ θε 1 | ρ ( r ) | w.p. at least 1 -δ (*) as long as ε 1 ≤ (1 -κ 1 ) 2 λ min λ max , which we shall ensure (see Appendix B.4).

For convenience we denote the normalized projection of any vector r as ˜ r := Σ 1 / 2 r ‖ Σ 1 / 2 r ‖ 2 . Now let ˜ r ∈ R d be a unit vector such that min {‖ ˜ r -˜ r ∗ ‖ 2 , ‖ ˜ r +˜ r ∗ ‖ 2 } ≥ ε 2 . Hence, using the definitions from Lemma 4.4, | γ ( r ) | ≤ 1 -ε 2 2 / 2 while γ ( r ∗ ) = 1 which implies ρ ( r ∗ ) -ρ ( r ) ≥ κ 3 ε 2 2 / 2 . Note that ρ ( r ) ≤ ρ ( r ∗ ) = 2 + κ 3 . Choosing ε 1 < κ 3 4 θ (2+ κ 3 ) ε 2 2 , we obtain that ρ ( r ∗ )(1 -θε 1 ) > ρ ( r )(1 + θε 1 ) . Using this along with the bound (*) we obtain that w.p. at least 1 -δ , ˆ ρ ( r ∗ ) > ˆ ρ ( r ) when ε 2 > 0 . Since our algorithm returns ˆ r as the maximizer of ˆ ρ , w.p. at least 1 -δ we get min {‖ ˜ r -˜ r ∗ ‖ 2 , ‖ ˜ r +˜ r ∗ ‖ 2 } ≤ ε 2 . Using Lemma 2.1, min {‖ ˆ r -r ∗ ‖ 2 , ‖ ˆ r + r ∗ ‖ 2 } ≤ 4 √ λ max λ min ε 2 . Substituting ε 2 = ε 4 √ λ min λ max , ‖ r -r ∗ ‖ 2 ≤ ε w.p. at least 1 -δ . The conditions on ε 1 are satisfied by taking it to be ≤ O ( κ 3 ε 2 λ min θ (2+ κ 3 ) λ max ) , and thus we can take m ≥ O ( ( d/ε 4 ) log( d/δ ) ( λ max λ min ) 2 θ 2 ( 2+ κ 3 κ 3 ) 2 ) = O ( ( d/ε 4 ) log( d/δ ) ( λ max λ min ) 4 q 4 ) , using Defn. 4.3. This completes the proof.

5 Proof Sketches for Theorems 1.3 and 1.5

glyph[negationslash]

Theorem 1.3: Case N ( 0 , I ) , f ( x ) = pos ( r T ∗ x ) , k = q/ 2 . The algorithm (Algorithm 3) and the proof is in Appendix A. We argue that a vector sampled uniformly at random from a bag is distributed as ω X 1 +(1 -ω ) X 0 where X a ∼ N ( 0 , I ) conditioned on f ( X a ) = a and ω is an independent { 0 , 1 }-Bernoulli r.v. s.t. p ( ω = 1) = k/q . This along with the fact that uncorrelated Gaussians are independent, allows us to show that the expectation is 0 in any direction orthogonal to r ∗ and to compute the expectation in the direction of r ∗ . We then use Lemma 3.1 to get the sample complexity expression.

Theorem 1.5: Case N ( µ , Σ ) , f ( x ) = pos ( r T ∗ x + c ∗ ) . The algorithm (Algorithm 4) and the detailed proof is given in Appendix C. We start by generalizing the high probability geometric error bound in Lemma 4.1 to this case (Lemma C.1 proven in Appendix C.1) and appropriately generalize κ 1 , κ 2 , κ 3 and θ . The rest of the proof is similar to Section 4.1. An extra factor of O ( glyph[lscript] 2 ) is introduced to the sample complexity from Lemma 3.1. Next, assuming the geometric bound, we give a high probability bound on the generalization error in Lemma C.2. In this analysis, we bound the BagErr sample ( h, M ) where h ( x ) = pos (ˆ r T x + c ∗ ) with ˆ r being the geometric estimate of r ∗ and M is a sample of bags. This introducing the dependencies on ( √ λ max + ‖ µ ‖ 2 ) / √ λ min as well as on Φ( glyph[lscript] ) . Our subroutine to find ˆ c ensures that h ∗ ( x ) = pos (ˆ r T x +ˆ c ) satisfies BagErr sample ( h ∗ , M ) ≥ BagErr sample ( h, M ) . We then use Theorem 2.2 to bound the generalization error of h ∗ . Lemmas C.1 C.2 together imply Theorem 1.5.

6 Experimental Results

General Gaussian. We empirically evaluate our algorithmic technique on centered and general Gaussian distributions for learning homogeneous LTFs. For homogeneous LTFs the general case algorithm (Alg. 4 in Appendix C) boils down to Alg. 2 in Sec. 4. The experimental LLP datasets are created using samples from both balanced as well as unbalanced bag oracles. In particular, for dimension d ∈ { 10 , 50 } , and each pair ( q, k ) ∈ { (2 , 1) , (3 , 1) , (10 , 5) , (10 , 8) , (50 , 25) , (50 , 35) }

and m = we create 25 datasets as follows: for each dataset (i) sample a random unit vector r ∗ and let f ( x ) := pos ( r ∗ T x ) , (ii) sample µ and Σ randomly (see Appendix F for details), (iii) sample m = 2000 training bags from Ex ( f, N ( µ , Σ ) , q, k ) , (iv) sample 1000 test instances ( x , f ( x )) , x ← N ( µ , Σ ) . We fix µ = 0 for the centered Gaussian case.

For comparison we include the random LTF algorithm in which we sample 100 random LTFs and return the one that satisfies the most bags. In addition, we evaluate the Algorithm of [25] on ( q, k ) = (2 , 1) , and the Algorithm of [26] on ( q, k ) = (3 , 1) . We measure the accuracy of each method on the test set of each dataset. The algorithms of [25, 26] are considerably slower and we use 200 training bags for them. The results for centered Gaussian are in Table 1a and for the general Gaussian are in Table 1b. We observe that our algorithms perform significantly better in terms of accuracy than the comparative methods in all the bag distribution settings. Further, our algorithms have significantly lower error bounds (see Appendix F).

Notice in Tables 1a and 1b that the test accuracy for Algorithm 2 decreases with an increase in q and d . This is consistent with the sample complexity expressions in Thm. 1.4 and Thm. 1.5. Also, notice that the test accuracy for Algorithm 2 for general Gaussian (Table 1b) is usually lesser than the same for centered Gaussian (Table 1a). This supports the theoretical result that the sample complexity increases with the increase in l .

Appendix F has additional details and further experiments for the N ( 0 , I ) with homogeneous LTFs, N ( µ , Σ ) with non-homogeneous LTFs as well as on noisy label distributions.

(a) N ( 0 , Σ ) feature-vectors.

7 Conclusion and Future work

Our work shows that LTFs can be efficiently properly learnt in the LLP setting from random bags with given label proportion whose feature-vectors are sampled independently from a Gaussian space, conditioned on their underlying labels. For the simple case of N ( 0 , I ) distribution and bags with unbalanced labels we provide a mean estimation based algorithm. For the general scenarios we develop a more sophisticated approach using the principal component of a matrix formed from certain covariance matrices. To resolve the ambiguity between the obtained solutions we employ novel generalization error bounds from bag satisfaction to instance classification. We also show that subgaussian concentration bounds are applicable on the thresholded Gaussians, yielding efficient sample complexity bounds. Our experimental results validate the performance guarantees of our algorithmic techniques.

In future work, classes of distributions other than Gaussian could be similarly investigated. Classifiers other than LTFs are also interesting to study in the LLP setting.

References

P. Awasthi, M. F. Balcan, and P. M. Long. The power of localization for efficiently learning linear separators with noise. J. ACM , 63(6):50:1-50:27, 2017.

(b) N ( µ , Σ ) feature-vectors.

A. Bhattacharyya, S. Ghoshal, and R. Saket. Hardness of learning noisy halfspaces using polynomial thresholds. In Proc. COLT , volume 75, pages 876-917. PMLR, 2018. URL http://proceedings.mlr.press/v75/bhattacharyya18a.html .

A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth. Learnability and the vapnikchervonenkis dimension. J. ACM , 36(4):929-965, 1989.

Robert Istvan Busa-Fekete, Heejin Choi, Travis Dick, Claudio Gentile, and Andres Munoz medina. Easy learning from label proportions. arXiv , 2023. URL https://arxiv.org/abs/ 2302.03115 .

L. Chen, Z. Huang, and R. Ramakrishnan. Cost-based labeling of groups of mass spectra. In Proc. ACM SIGMOD International Conference on Management of Data , pages 167-178, 2004.

Lin Chen, Thomas Fu, Amin Karbasi, and Vahab Mirrokni. Learning from aggregated data: Curated bags versus random bags. arXiv , 2023. URL https://arxiv.org/abs/2305. 09557 .

Amit Daniely. A PTAS for agnostically learning halfspaces. In Proceedings of The 28th Conference on Learning Theory, COLT 2015, Paris, France, July 3-6, 2015 , volume 40 of JMLR Workshop and Conference Proceedings , pages 484-502, 2015.

S. Dasgupta. Learning mixtures of Gaussians. In FOCS , pages 634-644, 1999.

N. de Freitas and H. Kück. Learning about individuals from group statistics. In Proc. UAI , pages 332-339, 2005.

L. M. Dery, B. Nachman, F. Rubbo, and A. Schwartzman. Weakly supervised classification in high energy physics. Journal of High Energy Physics , 2017(5):1-11, 2017.

G. Dulac-Arnold, N. Zeghidour, M. Cuturi, L. Beyer, and J. P. Vert. Deep multi-class learning from label proportions. CoRR , abs/1905.12909, 2019. URL http://arxiv.org/abs/1905. 12909 .

V. Feldman, P. Gopalan, S. Khot, and A. K. Ponnuswami. On agnostic learning of parities, monomials, and halfspaces. SIAM J. Comput. , 39(2):606-645, 2009.

Benjamin Fish and Lev Reyzin. On the complexity of learning a class ratio from unlabeled data. J. Artif. Intell. Res. , 69:1333-1349, 2020.

V. Guruswami and P. Raghavendra. Hardness of learning halfspaces with noise. In Proc. FOCS , pages 543-552, 2006.

J. Hernández-González, I. Inza, and J. A. Lozano. Learning bayesian network classifiers from label proportions. Pattern Recognit. , 46(12):3425-3440, 2013.

A. T. Kalai, A. R. Klivans, Y. Mansour, and R. A. Servedio. Agnostically learning halfspaces. In 46th Annual IEEE Symposium on Foundations of Computer Science (FOCS 2005), 23-25 October 2005, Pittsburgh, PA, USA, Proceedings , pages 11-20. IEEE Computer Society, 2005.

A. R. Klivans, P. M. Long, and R. A. Servedio. Learning halfspaces with malicious noise. J. Mach. Learn. Res. , 10:2715-2740, 2009.

D. Kotzias, M. Denil, N. de Freitas, and P. Smyth. From group to individual labels using deep features. In Proc. SIGKDD , pages 597-606, 2015.

J. Liu, B. Wang, Z. Qi, Y. Tian, and Y. Shi. Learning from label proportions with generative adversarial networks. In Proc. NeurIPS , pages 7167-7177, 2019.

D. R. Musicant, J. M. Christensen, and J. F. Olson. Supervised learning by training on aggregate outputs. In Proc. ICDM , pages 252-261. IEEE Computer Society, 2007.

J. Nandy, R. Saket, P. Jain, J. Chauhan, B. Ravindran, and A. Raghuveer. Domain-agnostic contrastive representations for learning from label proportions. In Proc. CIKM , pages 15421551, 2022.

G. Patrini, R. Nock, T. S. Caetano, and P. Rivera. (almost) no label no cry. In Proc. Advances in Neural Information Processing Systems , pages 190-198, 2014.

N. Quadrianto, A. J. Smola, T. S. Caetano, and Q. V. Le. Estimating labels from label proportions. J. Mach. Learn. Res. , 10:2349-2374, 2009.

S. Rueping. SVM classifier estimation from group probabilities. In Proc. ICML , pages 911-918, 2010.

R. Saket. Learnability of linear thresholds from label proportions. In Proc. NeurIPS , 2021. URL https://openreview.net/forum?id=5BnaKeEwuYk .

R. Saket. Algorithms and hardness for learning linear thresholds from label proportions. In Proc. NeurIPS , 2022. URL https://openreview.net/forum?id=4LZo68TuF-4 .

Rishi Saket, Aravindan Raghuveer, and Balaraman Ravindran. On combining bags to better learn from label proportions. In AISTATS , volume 151 of Proceedings of Machine Learning Research , pages 5913-5927. PMLR, 2022. URL https://proceedings.mlr.press/v151/ saket22a.html .

C. Scott and J. Zhang. Learning from label proportions: A mutual contamination framework. In Proc. NeurIPS , 2020.

Leslie G. Valiant. A theory of the learnable. Commun. ACM , 27(11):1134-1142, 1984.

S. Vempala. Learning convex concepts from gaussian distributions with PCA. In FOCS , pages 124-130, 2010.

R. Vershynin. How close is the sample covariance matrix to the actual covariance matrix? J. Theor. Probab. , 25:655-686, 2012.

Roman Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018. doi: 10.1017/9781108231596.

J. Wojtusiak, K. Irvin, A. Birerdinc, and A. V. Baranova. Using published medical results and non-homogenous data in rule learning. In Proc. International Conference on Machine Learning and Applications and Workshops , volume 2, pages 84-89. IEEE, 2011.

F. X. Yu, D. Liu, S. Kumar, T. Jebara, and S. F. Chang. ∝ SVM for learning with label proportions. In Proc. ICML , volume 28, pages 504-512, 2013.

F. X. Yu, K. Choromanski, S. Kumar, T. Jebara, and S. F. Chang. On learning from label proportions. CoRR , abs/1402.5902, 2014. URL http://arxiv.org/abs/1402.5902 .

A Proof of Theorem 1.3

For the setting of Theorem 1.3 we provide Algorithm 3.

Algorithm 3 PAC Learner for homogenous LTFs from unbalanced bags over N ( 0 , I )

glyph[negationslash]

Input: Ex ( f, D , q, k ) , m , where f ( x ) = pos ( r T ∗ x ) , ‖ r ∗ ‖ 2 = 1 , k = q/ 2 .

1. Compute ˆ µ B using MeanCovsEstimator with m samples.

2. If k > q/ 2 Return: ˆ µ B / ‖ ˆ µ B ‖ 2 else Return: - ˆ µ B / ‖ ˆ µ B ‖ 2

Define X a ∼ N ( 0 , I ) conditioned on f ( X a ) = a for a ∈ { 0 , 1 } , and let X B denotes the random feature-vector u.a.r sampled from a randomly sampled bag. By the definition of the bag oracle, X B := ω X 1 +(1 -ω ) X 0 , where ω is an independent { 0 , 1 } -Bernoulli r.v. s.t. p ( ω = 1) = k/q . Let g ∗ := r T ∗ X B . g ∗ ∼ N (0 , 1) and let g ∗ a = r T ∗ X a , a ∈ { 0 , 1 } . Since r ∗ is a unit vector, g ∗ a is a half-gaussian and by direct integration we obtain E [ g ∗ a ] = ( -1) 1 -a √ 2 /π for a ∈ { 0 , 1 } . Thus,

On the other hand, let g ⊥ = r T X s.t. r T r ∗ = 0 and ‖ r ‖ 2 = 1 , and let g ⊥ a = r T X a , a ∈ { 0 , 1 } . Since X a ( a ∈ { 0 , 1 } ) is given by conditioning a standard Gaussian vector only on the component in the direction of r ∗ , its component along any direction orthogonal to r ∗ is a one-dimensional standard Gaussian. Therefore, g ⊥ a are iid N (0 , 1) ( a ∈ { 0 , 1 } ), and so is g ⊥ , implying E [ g ⊥ ] = 0 . Thus, the value of r T E [ X B ] is (i) η ( q, k ) if r = r ∗ , and (ii) 0 if r ⊥ r ∗ . In other words, µ B = E [ X B ] = η ( q, k ) r ∗ , and ‖ µ B ‖ 2 = | η ( q, k ) | . Hence, if η ( q, k ) > 0 then r ∗ = µ B / ‖ µ B ‖ 2 else r ∗ = -µ B / ‖ µ B ‖ 2

With m = O ( η ( q, k ) 2 ( d/ε 2 ) log( d/δ ) ) = O ( ( d/ε 2 ) log( d/δ ) ) , the following lemma along with Lemma 2.3 completes the proof of Theorem 1.3.

Lemma A.1. Algorithm 3 returns a normal vector ˆ r such that ‖ r ∗ -ˆ r ‖ 2 ≤ ε w.p. at least 1 -δ when m ≥ O ( ( d/ε 2 ) log( d/δ ) ) for ε, δ > 0 .

Proof. First, we can assume ε ≤ 2 since the distance between two unit vectors is at most 2 and therefore the lemma is trivially true for ε > 2 . By Lemma 3.1, taking m ≥ = O ( ( d/ε 2 ) log( d/δ ) ) = O ( η ( q, k ) 2 ( d/ε 2 ) log( d/δ ) ) ensures ‖ ˆ µ B -µ B ‖ 2 ≤ ε | η ( q, k ) | / 4 = ε ‖ µ B ‖ 2 / 4 w.p. 1 -δ . Therefore, by triangle inequality, |‖ ˆ µ B ‖ 2 -‖ µ B ‖ 2 | ≤ ε ‖ µ B ‖ 2 / 4 ⇒ ‖ ˆ µ B ‖ 2 / ‖ µ B ‖ 2 ∈ [1 -ε 4 , 1 + ε 4 ] . Now, r ∗ = sign ( η ( q, k )) µ B / ‖ µ B ‖ 2 and the algorithm returns ˆ r := sign ( η ( q, k )) ˆ µ B / ‖ ˆ µ B ‖ 2 .

B Useful Tools

B.1 Chernoff Bound

Theorem B.1. Let X 1 , . . . , X n be iid { 0 , 1 } -valued random variables. Let S be their sum and µ = E [ S ] . Then, for any δ > 0 ,

B.2 Relationship between Σ B and Σ D

Lemma B.2. If Σ B , Σ D are as defined in the preliminaries and X a := X | f ( X ) = a , then

Proof. Let X B be a random-feature vector sampled uniformly from a random bag sampled from O . Hence, with probability k/q it is sampled from X 1 and with a probability of 1 -k/q it is sampled from X 0 . Hence,

Let X D = X -X ′ where ( X , X ′ ) are a random pair of feature-vectors sampled (without replacement) from a random bag sampled from O . Hence, with probability ( k 2 ) / ( q 2 ) it is the difference of two vectors sampled independently from X 1 , with probability ( q -k 2 ) / ( q 2 ) it is the difference of two vectors sampled independently from X 0 , with probability k ( q -k ) / 2 ( q 2 ) , it is the difference of one vector sampled from X 1 and another sampled independently from X 0 and with probability k ( q -k ) / 2 ( q 2 ) , it is the difference of one vector sampled from X 0 and another sampled independently from X 1 . Then,

Due to independence, we obtain for a ∈ { 0 , 1 }

Hence,

Simplifying Σ D -2 Σ B , we get

B.3 Bound on γ ( r ) when r T Σ B r ∗ = 0

Lemma B.3. If r T Σ B r ∗ = 0 then γ ( r ) ≤ 1 -( λ min λ max ) 2 1 -max(0 ,κ 1 ( q,k,glyph[lscript] )) 1 -min(0 ,κ 1 ( q,k,glyph[lscript] )) . Further if glyph[lscript] = 0 , then | γ ( r ) | ≤ 1 -( λ min λ max ) 2 (1 -κ 1 ( q, k )) .

Proof. We begin by observing the for any r ∈ R d , ‖ Σ 1 / 2 r ‖ 2 = 1 , Var[ r T Z B ] = 1 -κ 1 ( q, k, glyph[lscript] ) . Hence, λ min ( Σ B ) ≥ (1 -max(0 , κ 1 ( q, k, glyph[lscript] ))) λ min and λ max ( Σ B ) ≤ (1 -min(0 , κ 1 ( q, k, glyph[lscript] ))) λ max.

Thus, by Cautchy-Schwartz inequality,

Define ˜ r = Σ 1 / 2 r ‖ Σ 1 / 2 r ‖ 2 for any r ∈ R d . Observe that γ ( r ) = ˜ r T ˜ r ∗ . Now since r T Σ B r ∗ = 0 , by substitution ( Σ 1 / 2 B Σ -1 / 2 ˜ r ) T ( Σ 1 / 2 B Σ -1 / 2 ˜ r ∗ ) = 0 . Thus, using Cautchy-Schwartz again

Again using Cautchy-Schwartz,

Thus,

If glyph[lscript] = 0 , then κ 1 ( q, k, 0) = κ 1 ( q, k ) ≥ 0 . Hence,

B.4 Bounding error in ˆ ρ

LemmaB.4. If ‖ E B ‖ 2 ≤ ελ max and ‖ E D ‖ 2 ≤ ελ max where E B = ˆ Σ B -ˆ Σ B and E D = ˆ Σ D -ˆ Σ D then,

for θ as defined in Defn. 4.3 or Defn. C.3.

Proof. Let a ( r ) := r T Σ D r , b ( r ) := r T Σ B r , e ( r ) := r T E D r , f ( r ) := r T E B r . Using the bounds on the spectral norms of E B and E D , we get that | e ( r ) | ≤ ελ max and | f ( r ) | ≤ ελ max. Also, using variances in Lemma C.4, a ( r ) ≥ λ min (2 -max(0 , 2 κ 1 -κ 2 )) ≥ 0 and b ( r ) ≥ λ min (1 -max(0 , κ 1 )) ≥ 0 . Hence, we can conclude that

where θ is as defined in Defn. C.3. If we substitute glyph[lscript] = 0 , we get that κ 1 ≥ 0 and we get θ as defined in Defn. 4.3.

B.5 Ratio maximisation as a PCA problem

Theorem B.5. If A and B are positive definite matrices, then for all r ∈ R d

Proof. The first statement comes from the substitution. Now, r T Ar r T Br is homogeneoous in r . Let a glyph[similarequal] b ⇒ a = kb, k ∈ R

B.6 Proof of Lemma 2.1

Proof. Let r ′ 1 = Ar 1 and r ′ 2 = Br 2 . As r 1 and r 2 are unit vectors, 1 ‖ A -1 ‖ 2 ≤ ‖ r ′ 1 ‖ 2 ≤ ‖ A ‖ 2 .

C Proof of Theorem 1.5

For the setting of Theorem 1.5, we provide Algorithm 4. It uses as a subroutine a polynomial time procedure PrincipalEigenVector for the principal eigen-vector of a symmetric matrix. For notation, let Γ := Σ 1 / 2 , then by Lemma E.13, we can write our linear threshold function pos ( r T ∗ X + c ∗ ) = pos ( u T ∗ Z -glyph[lscript] ) where Z ∼ N ( 0 , I ) where glyph[lscript] := -c ∗ + r T ∗ µ ‖ Γr ∗ ‖ 2 , u ∗ := Γr ∗ / ‖ Γr ∗ ‖ 2 . For any r ∈ R d , define u := Γr / ‖ Γr ‖ 2 . Let φ and Φ be the standard gaussian pdf and cdf respectively.

The following geometric bound is obtained by the algorithm.

Algorithm 4 PAC Learner for LTFs in over N ( µ , Σ )

Input: O = Ex ( f, D = N ( µ , Σ ) , q, k ) , m, s , where f ( x ) = pos ( r T ∗ x + c ∗ ) , ‖ r ∗ ‖ 2 = 1 .

Compute ˆ Σ B , ˆ Σ D using MeanCovsEstimator with m samples.

r = ˆ Σ -1 / 2 B PrincipalEigenVector ( ˆ Σ -1 / 2 B ˆ Σ D ˆ Σ -1 / 2 B ) if ˆ Σ -1 / 2 B exists, else exit .

Let ˆ r = r / ‖ r ‖ 2 .

If k = q/ 2

Sample a collection M of s bags from O .

For each bag B j in M

Project each vector x ( i ) j in B j on ˆ r .

Order these projections in a descending order ˆ r T x (1) j > ... > ˆ r T x ( q ) j .

Define h j = pos (ˆ r T x +ˆ r T x ( k ) j )

Return h ∗ ∈ { h 1 , ..., h s } which has lower BagErr sample ( h ∗ , M ) . 5. else

Sample a collection M of s bags from O .

For each bag B j in M

Project each vector x ( i ) j in B j on ˆ r .

Order these projections in a descending order ˆ r T x (1) j > ... > ˆ r T x ( q ) j .

Define h j = pos (ˆ r T x +ˆ r T x ( k ) j )

For each bag B j in M

Project each vector x ( i ) j in B j on -ˆ r .

Order these projections in a descending order -ˆ r T x (1) j > ... > -ˆ r T x ( q ) j .

Define ˜ h j = pos ( -ˆ r T x -ˆ r T x ( k ) j )

Return h ∗ ∈ { h 1 , ..., h s , ˜ h 1 , ..., ˜ h s } which has lower BagErr sample ( h ∗ , M ) .

The above lemma, whose proof is deferred to Sec. C.1, is used in conjunction with the following lemma.

glyph[negationslash]

Lemma C.2. Let ε, δ ∈ (0 , 1) and suppose that ˆ r computed in Step 3 of Alg. 4 satisfies if k = q/ 2 min {‖ ˆ r -r ∗ ‖ 2 , ‖ ˆ r + r ∗ ‖ 2 } ≤ ε, . Then, with s ≥ O ( d (log q +log(1 /δ )) /ε 2 ) , h ∗ computed in Step 4.c or Step 5.d satisfies glyph[negationslash]

and if k = q/ 2 ,

glyph[negationslash]

glyph[negationslash]

w.p. 1 -δ/ 2 , where c 0 , c 1 are the constants from Lemma 2.3.

With the above we complete the proof of Theorem 1.5 as follows.

Proof. (of Theorem 1.5) Let the parameters δ, ε be as given in the statement of the theorem. We use O ( ε (Φ( glyph[lscript] )(1 -Φ( glyph[lscript] ))) √ λ min q ( √ λ max + ‖ µ ‖ 2 ) ) for the error bound in Lemma C.1. We now take m to be of m = O ( ( d/ε 4 ) O ( glyph[lscript] 2 ) (Φ( glyph[lscript] )(1 -Φ( glyph[lscript] ))) 2 log( d/δ ) ( λ max λ min ) 4 ( √ λ max + ‖ µ ‖ 2 √ λ min ) 4 q 8 ) in Alg. 4 we obtain the following bound: min {‖ ˆ r -r ∗ ‖ 2 , ‖ ˆ r + r ∗ ‖ 2 } ≤ ε (Φ( glyph[lscript] )(1 -Φ( glyph[lscript] ))) √ λ min q ( c 0 √ λ max + c 1 ‖ µ ‖ 2 ) with probability 1 -δ/ 2 . Using s ≥ O ( d (log( q ) + log(1 /δ )) q 2 ( √ λ max + ‖ µ ‖ 2 ) 2 ε 2 (Φ( glyph[lscript] )(1 -Φ( glyph[lscript] )) 2 λ min ) , Lemma C.2 yields the desired misclassification error bound of ε on h ∗ w.p. 1 -δ .

Proof. (of Lemma C.2) Define h ( x ) := pos (ˆ r T X + c ∗ ) and ˜ h ( x ) := pos ( -ˆ r T X + c ∗ ) . Applying Lemma 2.3, we obtain that at least one of h , ˜ h has an instance misclassification error of at most

glyph[negationslash]

O ( ε ( √ λ max /λ min + ‖ µ ‖ 2 /λ min )) . WLOG assume that h satisfies this error bound i.e., Pr D [ f ( x ) = h ( x )] ≤ ε ( c 0 √ λ max /λ min + c 1 ‖ µ ‖ 2 / √ λ min ) =: ε ′ . Note that, Pr D [ f ( x ) = 1] = Φ( glyph[lscript] ) , Pr D [ f ( x ) = 0] = 1 -Φ( glyph[lscript] ) . Thus, glyph[negationslash]

glyph[negationslash]

Therefore, taking a union bound we get that the probability that a random bag from the oracle contains a feature vector on which f and h disagree is at most qε ′ / Φ( glyph[lscript] )(1 -Φ( glyph[lscript] )) . Applying Chernoff bound (see Appendix B.1) we obtain that with probability at least 1 -δ/ 6 , BagErr sample ( h, M ) ≤ 2 qε ′ / Φ( glyph[lscript] )(1 -Φ( glyph[lscript] )) . Therefore, h satisfies BagErr sample ( h ∗ , M ) ≤ 2 qε ′ / Φ( glyph[lscript] )(1 -Φ( glyph[lscript] )) . Hence, there exists at least one h j ( x ) = pos (ˆ r T x + ˆ r T x ( k ) j ) will satisfy BagErr sample ( h j , M ) ≤ 2 qε ′ / Φ( glyph[lscript] )(1 -Φ( glyph[lscript] )) . Since, h ∗ ( x ) has the minimum sample bag error among all h j ( x ) , BagErr sample ( h ∗ , M ) ≤ 2 qε ′ / Φ( glyph[lscript] )(1 -Φ( glyph[lscript] )) .

glyph[negationslash]

On the other hand, applying Theorem 2.2, except with probability δ/ 3 , Pr D [ f ( x ) = h ∗ ( x )] ≤ 8 qε ′ / Φ( glyph[lscript] )(1 -Φ( glyph[lscript] )) = 8 qε Φ( glyph[lscript] )(1 -Φ( glyph[lscript] )) ( c 0 √ λ max /λ min + c 1 ‖ µ ‖ 2 / √ λ min ) if k = q/ 2 and min(Pr D [ f ( x ) = h ∗ ( x )] , Pr D [ f ( x ) = (1 -h ∗ ( x ))]) ≤ 8 qε ′ / Φ( glyph[lscript] )(1 -Φ( glyph[lscript] )) = 8 qε Φ( glyph[lscript] )(1 -Φ( glyph[lscript] )) ( c 0 √ λ max /λ min + c 1 ‖ µ ‖ 2 / √ λ min ) if k = q/ 2 . Therefore, except with probability δ/ 2 , the bound in Lemma C.2 holds.

glyph[negationslash]

C.1 Proof of Lemma C.1

We use these to generalize a few quantities we had defined earlier. We obtain their bounds using E.2, Definition C.3. Define:

Where K i 's are some finite functions of K .

Similar to Lemma 4.4, we again show in the following lemma that ˆ r in the algorithms is indeed ± r ∗ if the covariance estimates were the actual covariances.

Lemma C.4. The ratio ρ ( r ) := r T Σ D r / r T Σ B r is maximized when r = ± r ∗ . Moreover,

Proof. Using the transformations to Z , we can let X B = ΓZ B as a random feature-vector sampled uniformly from a random bag sampled from O . Also, let X D = ΓZ D be the difference of two random feature vectors sampled uniformly without replacement from a random bag sampled from O . Observe that the ratio ρ ( r ) = Var[ r T X D ] / Var[ r T X B ] = Var[ u T Z D ] / Var[ u T Z B ] .

Let g ∗ := u T ∗ Z which is N (0 , 1) . For a ∈ { 0 , 1 } , let Z a be Z conditioned on pos ( u T ∗ Z -glyph[lscript] ) = a . Let g ∗ a := u T ∗ Z a , a ∈ { 0 , 1 } . g ∗ 0 lower-tailed one-sided truncated normal distributions truncated at glyph[lscript] and g ∗ 1 upper-tailed one-sided truncated normal distributions truncated at glyph[lscript] . Hence, E [ g ∗ 1 ] = φ ( glyph[lscript] ) / (1 -Φ( glyph[lscript] )) , E [ g ∗ 0 ] = -φ ( glyph[lscript] ) / Φ( glyph[lscript] ) , E [ g ∗ 2 1 ] = 1 + glyph[lscript]φ ( glyph[lscript] ) / (1 -Φ( glyph[lscript] )) , E [ g ∗ 2 0 ] = 1 -glyph[lscript]φ ( glyph[lscript] ) / Φ( glyph[lscript] ) .

glyph[negationslash]

glyph[negationslash]

With this setup, letting g ∗ B := u T ∗ Z B and g ∗ D := u T ∗ Z D we obtain (using Lemma B.2 in Appendix B.2)

Now let ˜ u be a unit vector orthogonal to u ∗ . Let ˜ g = ˜ u T Z be N (0 , 1) . Also, let ˜ g a = ˜ u T Z a for a ∈ { 0 , 1 } . Since Z a are given by conditioning Z only along u ∗ , ˜ g a ∼ N (0 , 1) for a ∈ { 0 , 1 } . In particular, the component along ˜ u of Z B (call it ˜ g B ) is N (0 , 1) and that of Z D (call it ˜ g D ) is the difference of two iid N (0 , 1) variables. Thus, Var[˜ g B ] = 1 and Var[˜ g D ] = 2 . Moreover, due to orthogonality all these gaussian variables corresponding to ˜ u are independent of those corresponding to u ∗ defined earlier.

u = α u + β ˜ u β = √ 1 -α 2

where the last equality uses β = √ 1 -α 2 . Letting u = Γr / ‖ Γr ‖ 2 we obtain that α = 〈 Γr , Γr ∗ 〉 ‖ Γr ‖ 2 ‖ Γr ∗ ‖ 2 = γ ( r ) completing the proof.

Lemma 4.5 shows that ratio maximization can be treated as an eigenvalue decomposition problem of the matrix Σ -1 / 2 B Σ D Σ -1 / 2 B . We now prove that the error in the estimate of ˆ r given to us by the algorithm is bounded if the error in the covariance estimates are bounded. The sample complexity of computing these estimates gives the sample complexity of our algorithm.

We now complete the proof of Lemma C.1 (with δ instead of δ/ 2 for convenience). By Lemma 3.1, taking m ≥ O ( ( d/ε 2 1 ) O ( glyph[lscript] 2 ) log( d/δ ) ) ensures that ‖ E B ‖ 2 ≤ ε 1 λ max and ‖ E D ‖ 2 ≤ ε 1 λ max w.p. at least 1 -δ where E B = ˆ Σ B -Σ B and E D = ˆ Σ D -Σ D . We start by defining ˆ ρ ( r ) := r T ˆ Σ D r r T ˆ Σ B r which is the equivalent of ρ using the estimated matrices. Observe that it can be written as ˆ ρ ( r ) = r T Σ B r + r T E B r r T Σ D r + r T E D r . Using these we can obtain the following bound on ˆ ρ : for any r ∈ R d , | ˆ ρ ( r ) -ρ ( r ) | ≤ θε 1 | ρ ( r ) | w.p. at least 1 -δ (*) as long as ε 1 ≤ (1 -max(0 ,κ 1 )) 2 λ min λ max , which we shall ensure (see Appendix B.4).

For convenience we denote the normalized projection of any vector r as ˜ r := Σ 1 / 2 r ‖ Σ 1 / 2 r ‖ 2 . Now let ˜ r ∈ R d be a unit vector such that min {‖ ˜ r -˜ r ∗ ‖ 2 , ‖ ˜ r +˜ r ∗ ‖ 2 } ≥ ε 2 . Hence, using the definitions from Lemma C.4, | γ ( r ) | ≤ 1 -ε 2 2 / 2 while γ ( r ∗ ) = 1 which implies ρ ( r ∗ ) -ρ ( r ) ≥ κ 3 ε 2 2 / 2 . Note that ρ ( r ) ≤ ρ ( r ∗ ) = 2 + κ 3 (1 -max(0 , κ 1 )) . Choosing ε 1 < κ 3 2 θ (2+ κ 3 (1 -max(0 ,κ 1 )) ε 2 2 , we obtain that ρ ( r ∗ )(1 -θε 1 ) > ρ ( r )(1 + θε 1 ) . Using this along with the bound (*) we obtain that w.p. at least 1 -δ , ˆ ρ ( r ∗ ) > ˆ ρ ( r ) when ε 2 > 0 . Since our algorithm returns ˆ r as the maximizer of ˆ ρ , w.p. at least 1 -δ we get min {‖ ˜ r -˜ r ∗ ‖ 2 , ‖ ˜ r + ˜ r ∗ ‖ 2 } ≤ ε 2 . Using Lemma 2.1, min {‖ ˆ r -r ∗ ‖ 2 , ‖ ˆ r + r ∗ ‖ 2 } ≤ 4 √ λ max λ min ε 2 . Substituting ε 2 = ε 4 √ λ min λ max , ‖ r -r ∗ ‖ 2 ≤ ε w.p. at least 1 -δ . The conditions on ε 1 are satisfied by taking it to be ≤ O ( κ 3 ε 2 λ min θ (2+ κ 3 (1 -max(0 ,κ 1 )) λ max ) , and thus we can take m ≥ O ( ( d/ε 4 ) O ( glyph[lscript] 2 ) log( d/δ ) ( λ max λ min ) 2 θ 2 ( 2+ κ 3 (1 -max(0 ,κ 1 ) κ 3 ) 2 ) = O ( ( d/ε 4 ) O ( glyph[lscript] 2 ) log( d/δ ) ( λ max λ min ) 4 q 4 ) since 1 /κ 3 ≤ q 2 (1 + glyph[lscript] 2 / 4) /glyph[lscript] 2 whenever glyph[lscript] > 1 (Defn. C.3). This completes the proof of Lemma C.1.

D Generalization error Bounds

We show that if a hypothesis LTF h satisfies close to 1 fraction of sufficient number of bags sampled from a bag oracle then with high probability h is a good approximator for the target LTF f (or its

complement in the case of balanced bags). The first step is to prove a generalization bound from the sample bag-level accuracy to the oracle bag-level accuracy.

Theorem D.1. Let O := Ex ( f, D , q, k ) be any bag oracle for an LTF f in d -dimensions, and let M be a collection of m bags sampled iid from the oracle. Then, there is an absolute constant C 0 ≤ 1000 s.t. w.p. at least 1 -δ ,

when m ≥ C 0 d (log q +log(1 /δ )) /ε 2 , for any δ, ε > 0 .

Proof. The proof follows from the arguments similar to the ones used in Appendix M of [26] to prove bag satisfaction generalization bounds. Consider a bag loss function of the form glyph[lscript] ( φ ζ ( h, B ) , ( φ ζ ( f, B )) , where φ ζ ( g, B ) := ζ ( y g,B ) where y g,B is the vector of ( g ( x )) x ∈ B , for ζ : { 0 , 1 } q → R . The result of [35] showed generalization error bounds when (i) ζ is 1 -Lipschitz w.r.t. to ∞ -norm, and (ii) glyph[lscript] is 1 -Lipschitz in the first coordinate. We can thus apply their results using the bound of ( d +1) on the VC-dimension of LTFs in d -dimensions to show that the above bound on m holds (with C 0 / 8 instead of C 0 ) for the generalization of the following bag error:

for t ∈ { 0 , . . . , q -1 } . We can bound our bag satisfaction error by the sum of the generalization errors of | γ ( B,f,k ) -γ ( B,h,k ) | , and | γ ( B,f,k -1) -γ ( B,h,k -1) | , which can each be bounded by ε/ 2 thus completing the proof.

Next we show that if the oracle-level bag accuracy of h is high then this translates to h is being a low error instance-level approximator for the target LTF f (or its complement in the case of balanced bags). With the setup as used in the previous theorem, let us define define the regions S a := { x s.t f ( x ) = a } for a ∈ { 0 , 1 } , and S ab := { x s.t f ( x ) = a, h ( x ) = b } . Let µ be the measure induced by D , µ a and µ ab be the respectively conditional measures induced on S a and S ab . The oracle O for a random bag B , samples k points iid from ( S 1 , µ 1 ) and ( q -k ) points from ( S 0 , µ 0 ) .

we have the following theorem.

Theorem D.2. Suppose k ∈ { 1 , . . . , q } , and 0 < BagAcc oracle ( h, f, D , q, k ) ≤ ε ′ < 1 / (4 q ) , then glyph[negationslash]

glyph[negationslash]

(i) Pr [ f ( x ) = h ( x )] ≤ ε if k = q/ 2 , and

glyph[negationslash]

glyph[negationslash]

where ε = 4 ε ′ .

Before we prove the above them, we need the following lemma bounding the probability that two independent binomial random variables take the same value. Let Binomial ( n, p ) be the sum of n iid { 0 , 1 } random variables each with expectation p .

glyph[negationslash]

Lemma D.3. Let u, v ∈ { 1 , . . . , q } , and X 1 ∼ Binomial ( u, p 1 ) and X 2 ∼ Binomial ( v, p 2 ) be independent binomial distributions for some probabilities p 1 and p 2 . Let p ∗ = min { max { p 1 , (1 -p 1 ) } , max { p 2 , (1 -p 2 ) }} , then, Pr[ X 1 = X 2 ] ≥ 1 - √ p ∗ .

Proof. We first begin by bounding a binomial coefficient by the sum of its adjacent binomial coefficients. Let n ≥ 1 and n > r > 0 . We begin with the standard identity and proceed further:

where

Moreover, it is trivially true that ( n 0 ) ≤ ( n 1 ) and ( n n ) ≤ ( n n -1 ) , therefore (11) holds even for r ∈ { 0 , n } whenever the binomial coefficients exist. Now, for some probability p define ν ( p, n, r ) := ( n r ) p r (1 -p ) 1 -r which is pdf at r of Binomial ( n, p ) . The above implies the following:

where p ′ = max { p/ (1 -p ) , (1 -p ) /p } . Using the fact that ν ( p, n, r -1)+ ν ( p, n, r )+ ν ( p, n, r +1) ≤ 1 , we obtain that

The above allows us to complete the proof of the lemma as follows.

where we use Cauchy-Schwarz for the first inequality.

Proof. (of Theorem D.2) We now consider three cases, for each one we shall prove points (i) and (ii) of the theorem.

glyph[negationslash]

Case Pr D [ f ( x ) = h ( x )] ≥ 1 -ε . This condition means that µ ( S 10 ) + µ ( S 01 ) ≥ 1 -ε , which implies that at least one of µ 1 ( S 10 ) , µ 0 ( S 01 ) is ≥ 1 -ε . Assume that µ 0 ( S 01 ) ≥ 1 -ε (the other case is analogous).

glyph[negationslash]

Consider unbalanced bags i.e., k = q/ 2 . Now in case that µ 1 ( S 10 ) ≥ 1 -ε , all the points sampled from S 1 are sampled from S 10 w.p. (1 -kε ) and those sampled from S 0 are all sampled from S 01 w.p. (1 -( q -k ) ε ) . Therefore, with probability at least (1 -qε ) all the points are sampled from S 10 ∪ S 01 . Since k = q/ 2 this implies that h does not satisfy the random bags with probability at least (1 -qε ) > ε ′ . If µ 1 ( S 10 ) ≤ ε , we can show with similar arguments that with probability ≥ (1 -qε ) > ε ′ no points are sampled from S 10 and q -k points are sampled from S 01 which means (since k ≥ 1 ) that h does not satisfy the bag. Finally, let µ 1 ( S 10 ) ∈ ( ε, 1 -ε ) . In this case, the number of points sampled from S 10 is distributed as Binomial ( k, µ 1 ( S 10 )) and those sampled from S 01 is independently distributed as Binomial ( k, µ 0 ( S 01 )) . If these two numbers are different then h does not satisfy the bag. We can apply Lemma D.3 and using the bounds on µ 1 ( S 10 ) , we get that p ∗ ≤ 1 -ε . Therefore, the probability of h not satisfying a randomly sampled bag is at least

glyph[negationslash]

For balanced bags, the case Pr D [ f ( x ) = h ( x )] ≥ 1 -ε implies that Pr[ f ( x ) = (1 -h ( x ))] ≤ ε , so the condition (ii) of the theorem holds.

glyph[negationslash]

glyph[negationslash]

Case ε < Pr D [ f ( x ) = h ( x )] < 1 -ε . This case violates the conditions (i) and (ii). First observe that, µ 1 ( S 10 ) and µ 0 ( S 01 ) both cannot be ≥ (1 -ε ) or ≤ ε , otherwise µ ( S 10 ) + µ ( S 01 ) is either ≥ 1 -ε or ≤ ε violating the assumption of this case. For the subcase that µ 1 ( S 10 ) ≤ ε and µ 0 ( S 01 ) ≥ (1 -ε ) , glyph[negationslash]

we can show using arguments similar to the previous case that w.p. ≥ (1 -qε ) no points are sampled from S 10 and q -k points are sampled from S 01 , and thus, h will not satisfy a random bag with probability at least (1 -qε ) > ε ′ . The subcase when µ 1 ( S 01 ) ≤ ε and µ 0 ( S 10 ) ≥ (1 -ε ) is analogous. Finally, we have the subcase that µ 1 ( S 10 ) or µ 0 ( S 01 ) both lie in the range ( ε, 1 -ε ) . Now, the number of points sampled from S 10 is distributed as Binomial ( k, µ 1 ( S 10 )) and those sampled from S 01 is independently distributed as Binomial ( k, µ 0 ( S 01 )) . If these two numbers are different then h does not satisfy the bag. We can apply Lemma D.3 and using the bounds on one of µ 1 ( S 10 ) or µ 0 ( S 01 ) , we get that p ∗ ≤ 1 -ε . Therefore, the probability of h not satisfying a randomly sampled bag is at least 1 - √ 1 -ε ≥ ε/ 2 > ε ′ , using (15).

D.1 Proof of Theorem 2.2

The proof follows directly from Theorems D.1 and D.2.

D.2 Proof of Lemma 2.3

Proof. We have,

glyph[negationslash]

Now, the RHS of (16) can be bounded as, glyph[negationslash]

glyph[negationslash]

Now, g := a T Z ∼ N (0 , 1) . Thus, the first term in the above is bounded by the probability that g lies in a range of length | ζ -ˆ ζ | = ‖ µ ‖ 2 ‖ r -ˆ r ‖ 2 / ‖ Γr ‖ 2 ≤ ε ‖ µ ‖ 2 / √ λ min. This probability is at most ε ‖ µ ‖ 2 / √ 2 πλ min.

glyph[negationslash]

For the second term, that is at most the probability that pos ( a T Z ) = pos (ˆ a T Z ) . Now ‖ ˆ a -a ‖ 2 ≤ 4 √ λ max λ min ε ⇒ ∠ ˆ a , a ≤ π 4 √ λ max λ min ε ⇒ Pr[ pos ( a T Z ) = pos (ˆ a T Z )] ≤ 2 √ λ max λ min ε . Hence, glyph[negationslash]

glyph[negationslash]

E Subgaussian concentration with thresholded Gaussian random variables

Let Φ( . ) be the standard Gaussian cdf i.e., Φ( t ) := Pr X ∼ N (0 , 1) [ X ≥ t ] . We also define Φ( t ) := Pr[ X > t ] = 1 -Φ( t ) . We begin by defining the subgaussian norm of a random variable.

Definition E.1. The subgaussian norm of a random variable X denoted by ‖ X ‖ ψ 2 and is defined as: ‖ X ‖ ψ 2 := inf { t > 0 : E [ exp ( X 2 /t 2 )] ≤ 2 } . Further, there is an absolute constant K 0 such that ‖ X ‖ ψ 2 ≤ K 0 K if X satisfies,

Let X ∼ N (0 , 1) . It is easy to see that E [ exp ( X 2 / 2 2 ) ] = (1 / √ 2 π ) ∫ ∞ -∞ exp ( -x 2 / 4) dx = √ 2(1 / √ 2 π ) ∫ ∞ -∞ exp ( -z 2 / 2) dz = √ 2 . Thus, X ∼ N (0 , 1) is subgaussian with subgaussian norm ≤ 2 . In the analysis of this section, we shall use the following proposition from [32].

glyph[negationslash]

glyph[negationslash]

Proposition E.2 (Prop. 2.1.2 of [32]) . Let X ∼ N (0 , 1) . Then, for any t > 0 ,

In particular for t ≥ 1 ,

Using the above, and by symmetry it is easy to see that for t ≥ 1 , Pr[ | X | > t ] ≤ ( √ 2 /π ) exp ( -t 2 / 2 ) ≤ 2 · exp ( -t 2 / 2 ) . On the other hand, 2 · exp ( -t 2 / 2 ) ≥ 1 for 0 ≤ t < 1 . Thus,

Consider the normal distribution conditioned on a threshold defined by letting D glyph[lscript] be the distribution of { X ∼ N (0 , 1) | X > glyph[lscript] } . We shall show that ˜ X ∼ D glyph[lscript] is a subgaussian random variable. Let us handle the (relatively) easy case of glyph[lscript] ≤ 0 first.

Lemma E.3. Let ˜ X ∼ D glyph[lscript] for some glyph[lscript] ≤ 0 . Then, Pr [ | ˜ X | > t ] ≤ 2 · exp ( -t 2 / 2 ) , for any t > 0 .

Proof. Let E be the event that ˜ X ≥ 0 . Conditioned on E , ˜ X is distributed as | X | where X ∼ N (0 , 1) and (19) implies that

On the other hand, conditioned on E , ˜ X is sampled as -| Z | where { Z ∼ N (0 , 1) | | Z | ≤ -glyph[lscript] = | glyph[lscript] |} . Thus,

Now, if t > | glyph[lscript] | then Pr [ | ˜ X | > t | E ] = 0 , otherwise if t ≤ glyph[lscript] the LHS of (20) is,

which is bounded by 2 · exp ( -t 2 / 2 ) and combining the probability bounds conditioned on E and E we complete the proof.

The case of glyph[lscript] > 0 is proved below.

Lemma E.4. Let ˜ X ∼ D glyph[lscript] for some glyph[lscript] > 0 . Then, Pr [ | ˜ X | > t ] ≤ 2 · exp ( -t 2 /K 2 1 ) , for any t > 0 , where K 1 = max { √ 20 , | glyph[lscript] | √ 10 } .

Proof. Let us first explicitly define the pdf of ˜ X as:

where f N (0 , 1) is the pdf of N (0 , 1) . We prove this in two cases.

Case glyph[lscript] ≤ 2 . In this case, one can use the lower bound from Prop. E.2 to show that Φ( glyph[lscript] ) ≥ Φ(2) > 1 / 50 by explicit calculation. Thus, f D glyph[lscript] ( x ) ≤ 50 f N (0 , 1) ( x ) for x > 0 . We can now obtain the desired bound as follows. Letting X ∼ N (0 , 1) , for any t > 0 ,

using (19). Now, it is easy to check that for t ≥ 3 ,

On the other hand, for 0 ≤ t < 3 , 2 exp ( -t 2 / 20 ) > 1 . Thus,

Case glyph[lscript] > 2 . In this case using the easily verifiable facts that hold for glyph[lscript] > 2 :

Prop. E.2 yields

, then this probability vanishes.

Observe that Pr [∣ ∣ ∣ ˜ X ∣ ∣ ∣ > t ] = Pr Z ∼ N (0 , 1) [ Z > t | Z ≥ glyph[lscript] ] . If t < glyph[lscript]

Otherwise t ≥ glyph[lscript] > 2 , and from Prop. E.2, Pr[ Z > t ] ≤ exp ( -t 2 / 2) and therefore Pr [∣ ∣ ∣ ˜ X ∣ ∣ ∣ > t ] can be bounded by

using (24). Now, if t 2 = 5 glyph[lscript] 2 + κ for some κ ≥ 0 , then using glyph[lscript] > 2 we have

Therefore, 2 exp ( -t 2 / 2 + 2 glyph[lscript] 2 ) ≤ 2 exp ( -t 2 / (10 glyph[lscript] 2 ) ) for t 2 ≥ 5 glyph[lscript] 2 . On the other hand,

Thus, in this case the following holds for all t > 0 :

completing the proof.

The above results also apply to "complements" of the thresholded Gaussians. In particular, let D glyph[lscript] be the distribution of { X ∼ N (0 , 1) | X ≤ glyph[lscript] ⇔-X ≥ -glyph[lscript] } which is equivalently {-X ∼ N (0 , 1) | X ≥ -glyph[lscript] } to which the above analysis can be directly be applied. This yields, that if ˜ X ∼ D glyph[lscript] then for any t > 0 ,

where K 1 = max { √ 20 , | glyph[lscript] | √ 10 } .

E.1 Bag mean and convariance estimation error bounds

In this section we shall be concerned with random variables ˜ X which are sampled from D glyph[lscript] with probability p and from D glyph[lscript] with probability (1 -p ) . Let us denote this distribution by ˆ D ( p, glyph[lscript] ) . Using Lemmas E.3, E.4 and (28), (29), we obtain the following lemma.

Lemma E.5. Let ˜ X ∼ ˆ D ( p, glyph[lscript] ) for some p ∈ (0 , 1) . Then for any t > 0 ,

where K 1 = max 20 , | glyph[lscript] 10 } . In particular, ‖ ˜ X ‖ ψ 2 = O ( | glyph[lscript] | )

We however, shall also require similar bounds for the mean-zero version of such distributions. To begin with we state an easy lemma bounding the mean of ˜ X .

Lemma E.6. Let ˜ X ∼ ˆ D ( p, glyph[lscript] ) for some p ∈ (0 , 1) . Then, ∣ ∣ ∣ E [ ˜ X ]∣ ∣ ∣ ≤ γ glyph[lscript] = max { γ 0 , 2 glyph[lscript] } , where γ 0 > 0 is some constant.

Proof. Let us consider the case of glyph[lscript] > 0 (and glyph[lscript] ≤ 0 follows analogously). When glyph[lscript] < 2 , it is easy to see that desired expectation is O (1) . Further, the expectation over D glyph[lscript] is O ( glyph[lscript] ) for any glyph[lscript] > 0 , since it is a convex combination of the expectation of a half gaussian which has O (1) expectation, and a gaussian truncated from below at 0 and above at glyph[lscript] , which has O ( glyph[lscript] ) expectation. To complete the argument we need to bound the expectation over D glyph[lscript] . Using (21) and (1 / √ 2 π ) ∫ ∞ glyph[lscript] x exp ( -x 2 / 2) dx = (1 / √ 2 π ) exp ( -glyph[lscript] 2 / 2) , we obtain E X ∼ D glyph[lscript] [ X ] = (1 / √ 2 π ) exp ( -glyph[lscript] 2 / 2)Φ( glyph[lscript] ) -1 and the lower bound from Prop. E.2 along with glyph[lscript] ≥ 2 yields an upper bound of 2 glyph[lscript] on the expectation.

With the setup as in Lemma E.5 define ̂ X := ˜ X -E [ ˜ X ] . Clearly, E [ ̂ X ] = 0 . Further,

Therefore,

Let K 2 = max { 2 K 1 , 2 γ glyph[lscript] } . Now,

On the other hand, when 0 ≤ t < 2 γ glyph[lscript] , t 2 /K 2 2 ≥ 1 / 2 , and thus

for all t > 0 , where K 2 = O ( | glyph[lscript] | ) .

E.1.1 Concentration of mean estimate using Hoeffding's Bound

Let us first state the Hoeffding's concentration bound for subgaussian random variables.

Theorem E.7 (Theorem 2.6.2 of [32]) . Let X 1 , . . . , X N be independent, mean-zero, sub-gaussian random variables. Then, for every ε ≥ 0 ,

where c > 0 is some absolute constant.

Thus,

For the rest of this section we shall fix glyph[lscript] ∈ R and p ∈ (0 , 1) . Consider vector valued random variable X = ( X (1) , . . . , X ( d ) ) with independent coordinates where

X (1) = ˜ X -E [ ˜ X ] where ˜ X ∼ ˆ D ( p, glyph[lscript] ) . From the previous subsection, we have ‖ X (1) ‖ ψ 2 = O ( | glyph[lscript] | ) .

For i = 2 , . . . , d , X ( i ) ∼ N (0 , 1) and therefore ‖ X ( i ) ‖ ψ 2 = O (1) .

Using the above bounds on the subgaussian norms, and applying Theorem E.7 to bound the error in each coordinate by ε/ √ d and taking a union bound we obtain the following lemma.

Lemma E.8. Let X 1 , . . . , X N be N iid samples of X . Then for every ε ≥ 0 ,

for some absolute constant c 0 > 0 . In particular, if N > O ( ( d/ε 2 ) O ( glyph[lscript] 2 ) log( d/δ ) ) ,

for any δ > 0 .

E.1.2 Concentration of covariance estimate

Consider the vector random variable X defined in the previous subsection. It is mean-zero and so by Defn. 3.4.1 and Lemma 3.4.2 of [32],

using the bounds on the subgaussian norms of the coordinates of X given in the previous subsection. Using this we can directly apply Proposition 2.1 of [31] to obtain the following lemma.

Lemma E.9. Let X 1 , . . . , X N be N iid samples of X , then if N > O ( ( d/ε 2 ) O ( glyph[lscript] 2 ) log(1 /δ ) ) ,

for any ε, δ > 0 .

E.1.3 Mean and covariance estimate bounds for non-centered vector r.v.s

Distribution D asymvec ( p, glyph[lscript] ) . Werevisit the definition of X in Sec. E.1.1 and instead define distribution D asymvec ( p, glyph[lscript] ) over Z = ( Z (1) , . . . , Z ( d ) ) with independent coordinates by taking Z (1) = ˜ X where ˜ X ∼ ˆ D ( p, glyph[lscript] ) and for i = 2 , . . . , d , Z ( i ) ∼ N (0 , 1) .

Clearly, X = Z -E [ Z ] . For convenience, we shall use the following notation:

where Z i is an iid sample of Z and X i = Z i -µ Z , for i = 1 , . . . , N . We have the following lemma. ( 2 2 )

LemmaE.10. For any ε, δ ∈ (0 , 1) , if N > O ( d/ε ) O ( glyph[lscript] ) log( d/δ ) , then w.p. 1 -δ the following hold simultaneously,

Proof. We begin by applying Lemmas E.8 and E.9 to X and the iid samples X 1 , . . . , X N so that their conditions hold with ε/ 2 and δ/ 2 . Taking a union bound we get that the following simultaneously hold with probability at least 1 -δ :

By the definitions above, (42) directly implies (40).

Now, observe that Σ Z = E [ XX T ] . On the other hand, letting ζ := ˆ µ Z -µ Z = ( ∑ N i =1 X i ) /N we simplify ˆ Σ Z as,

Thus, the LHS of (41) is at most,

since we have shown that ‖ ζ ‖ 2 = ‖ ˆ µ Z -µ Z ‖ 2 ≤ ε/ 2

Finally, we prove a version of the above lemma under a symmteric psd transformation. Let A be a psd matrix s.t. λ max is the maximum eigenvalue of A 2 = AA i.e., √ λ max is the maximum eigenvalue of A . Then, if we define ˜ Z := AZ and ˜ Z i as iid samples of ˜ Z , i = 1 , . . . , N and analogous to (38) and (39), define µ ˜ Z , ˆ µ ˜ Z , Σ ˜ Z and ˆ Σ ˜ Z , we have the following lemma which follows directly from Lemma E.10 the √ λ max upper bound on the operator norm of A .

LemmaE.11. For any ε, δ ∈ (0 , 1) , if N > O ( ( d/ε 2 ) O ( glyph[lscript] 2 ) log( d/δ ) ) , then w.p. 1 -δ the following hold simultaneously,

E.2 Estimating Covariance of differences

First we begin this subsection with a simple observation. If X and Y are two random variables such that ,

then X -Y is a random variable such that,

It is easy to see that,

when t 2 ≥ 8 K 2 1 . On the other hand, when t 2 < 8 K 2 1 , 2 · exp ( -t 2 / (4 K 1 ) 2 ) > 1 . Thus,

In this subsection shall consider ˜ X ∼ D ( p, q, glyph[lscript] ) to be defined as follows, for some glyph[lscript] ∈ R , p, q ∈ [0 , 1) s.t. p + q < 1 . ˜ X = U -V where:

with probability p , U ∼ D glyph[lscript] and V ∼ D glyph[lscript] independently,

with probability q , U ∼ D glyph[lscript] and V ∼ D glyph[lscript] independently,

with probability (1 -p -q ) / 2 , U ∼ D glyph[lscript] and V ∼ D glyph[lscript] independently,

with probability (1 -p -q ) / 2 , U ∼ D glyph[lscript] and V ∼ D glyph[lscript] independently.

From the above it is clear that E [ ˜ X ] = 0 . Further, from (49) and Lemmas E.3, E.4 and (28), (29),

for t > 0 , where K 1 = max { 4 √ 20 , 4 | glyph[lscript] | √ 10 } . In particular, ‖ ˜ X ‖ ψ 2 = O ( glyph[lscript] ) .

Let us now define a distribution D diffvec ( p, q, glyph[lscript] ) vector valued random variable with independent coordinates X = ( X (1) , . . . , X ( d ) ) , where

X (1) ∼ D ( p, q, glyph[lscript] ) , for some glyph[lscript] ∈ R , p, q ∈ [0 , 1) s.t. p + q < 1 .

X ( j ) is the difference of two iid N (0 , 1) random variables, for j = 2 , . . . , d . In particular, the subgaussian norm of these coordinates is O (1) .

From, the above it is clear that E [ X ] = 0 , and by Defn. 3.4.1 and Lemma 3.4.2 of [32], (36) is applicable to X as defined above. Thus, letting ˆ X := AX , where A is as used in the previous subsection we have,

Lemma E.12. For X defined above, the statement of Lemma E.9 is applicable, and (37) implies that the following holds:

with probability 1 -δ .

E.3 Proof of Lemma 3.1

Our proof shall utilize the following normalization of LTFs in a Gaussian space.

Lemma E.13. Suppose f ( X ) = pos ( r T X + c ) , where ‖ r ‖ 2 > 0 and X ∼ N ( µ , Σ ) s.t. Σ is positive definite. Let Γ := Σ 1 / 2 be symmetric p.d., and U be any orthonormal transformation satisfying UΓr / ‖ Γr ‖ 2 = e 1 where e 1 is the vector with 1 in the first coordinate and 0 in the rest. Then, letting Z ∼ N ( 0 , I ) so that X = ΓU T Z + µ ,

where glyph[lscript] = ( r T µ + c ) / ‖ Γr ‖ 2 .

Proof. We have X = ˆ X + µ where ˆ X ∼ N ( 0 , Σ ) . Thus, ˆ X = ΓU T Z ⇒ Z = UΓ -1 ˆ X , using U T = U -1 . Now, f ( X ) can be written as

which completes the proof.

Proof. (of Lemma 3.1) Using the normalization in Lemma E.13, we can write X = AZ + µ where A = Σ 1 / 2 U T such that X ∼ N ( µ , Σ ) ≡ Z ∼ N ( 0 , I ) . From the condition in (52) we can write the samples in Step 2 of Alg. 1 as x i = Az i + µ where z i are sampled from D asymvec ( k/q, -glyph[lscript] ) (see Sec.

E.1.3) for glyph[lscript] as given in Lemma E.13. Note that the maximum eigenvalue of A 2 is λ max which is the maximum eigenvalue of Σ . Thus, one can apply Lemma E.11 to ˆ µ B and ˆ Σ B .

Further, the difference vectors sampled in Step 6 can be written as x i = Az i where z i are sampled from D diffvec ( p, p ′ , -glyph[lscript] ) (see Sec. E.2) where p = ( k 2 ) / ( q 2 ) is the probability of sampling a pair of 1 -labeled feature-vectors from a bag, and p ′ = ( q -k 2 ) / ( q 2 ) is that of sampling a pair of 0 -labeled feature-vectors. Thus, one can apply Lemma E.12 to ˆ Σ D .

Using both the above applications with the error probability δ/ 2 and using union bound we complete the proof.

F Experimental Details and Results

F.1 Implementation Details

The implementations of the algorithms in this paper (Algs. 2, 3, 4) and of the random LTF algorithm are in python using numpy libraries. The code for the SDP algorithms of [25, 26] for bag sizes 2 and 3 is adapted from the publicly available codebase 1 . The experimental code was executed on a 16-core CPU and 128 GB RAM machine running linux in a standard python environment.

F.2 Experimental Results

In the following d denotes the dimension of the feature-vectors, q the size of the bags, with k/q ∈ (0 , 1) the bag label proportion, and m be the number of sampled bags in the training dataset. The instance-level test set is of size 1000 in all the experiments, and the reported metric is the accuracy over the test set.

glyph[negationslash]

Standard Gaussian without LTF offset. Here we primarily wish to evaluate the Algorithm 3 using unbalanced bag oracles such that k = q/ 2 . For d ∈ { 10 , 50 } , ( q, k ) ∈ { (3 , 1) , (10 , 8) , (50 , 35) } and m ∈ { 100 , 500 , 2000 } we create 25 datasets. In each LLP dataset, we (i) sample a random unit vector r ∗ and let f ( x ) := pos ( r ∗ T x ) , (ii) sample m training bags from Ex ( f, N ( 0 , I ) , q, k ) , (iii) sample 1000 test instances ( x , f ( x )) , x ← N ( 0 , I ) . We also evaluate the Algorithm 2 on these datasets. For comparison we have the random LTF algorithm (R) in which we sample 100 random LTFs and return the one that performs best on the training set. The results are reported in Table 5. We also evaluate the SDP algorithm (S) in [26] for ( q, k ) = (3 , 1) using m ∈ { 50 , 100 , 200 } (since the SDP algorithms do not scale to larger number of bags) whose results are reported in Table 6.

Table 2 reports a concise set of comparative scores of Algorithms 2, 3 and random LTF (R) using 2000 bags and of the SDP algorithms (S) with 200 bags.

Centered and general Gaussian. Here we evaluate Algorithms 2 and 4, and we have both balanced as well as unbalanced bag oracles. In particular, for d ∈ { 10 , 50 } , ( q, k ) ∈ { (2 , 1) , (3 , 1) , (10 , 5) , (10 , 8) , (50 , 25) , (50 , 35) } and m ∈ { 100 , 500 , 2000 } we create 25 datasets similar to the previous case, except that for each dataset we first sample µ and Σ and use N ( 0 , Σ ) for sampling feature-vectors in the centered Gaussian case and use N ( µ , Σ ) for sampling feature-vectors in the general Gaussian case. We perform the following set of experiments in each case. For the cases when bags are balanced, i.e. ( q, k ) ∈ { (2 , 1) , (10 , 5) , (50 , 25) } , for each our Algorithms 2 and 4 we evaluate their two possible solutions on the test data and report the better number.

With LTF offset. We sample ( r ∗ , c ∗ ) and create a dataset using pos ( r T ∗ x + c ∗ ) as the labeling function. Table 11 reports the test accuracy scores for Algorithm 4 and random LTF (R) with m = { 100 , 500 , 2000 } for centered and general Gaussians. Table 12 reports the corresponding scores for the SDP algorithm (S) [25, 26] with m = { 50 , 100 , 200 } and ( q, k ) ∈ { (2 , 1) , (3 , 1) } . Table 4 provides concise comparative scores with m = 2000 for Algorithm 4 and random LTF and m = 200 for the SDP algorithm (S).

Without LTF offset. We sample an r ∗ and create a dataset using pos ( r T ∗ x ) as the labeling function. Table 7 reports the test accuracy scores for Algorithm 2 and random LTF (R)

for centered and general Gaussians. Table 12 reports the scores for the SDP algorithm (S) [25, 26] on ( q, k ) ∈ { (2 , 1) , (3 , 1) } with m = { 50 , 100 , 200 } . Table 3 provides concise comparative scores with m = 2000 for Algorithm 4 and random LTF and m = 200 for the SDP algorithm (S).

Noisy Labels. We also experiment in a model with label noise. Here, the label of any instance can be independently flipped with some probability p , as a result the true bag label sum k ∗ has distribution over { 0 , . . . , q } . In this case the SDP algorithms are not applicable and we omit them. Tables 9 and 10 give the test accuracy scores for Algorithm 2 and rand. LTF (R) with label flip probability p = { 0 . 1 , 0 . 25 , 0 . 5 } for centered and general Gaussians. Like the balanced case, here also we evaluate both the solutions of Algorithm 2 on the test data and report the better number.

We observe that our algorithms perform significantly better in terms of accuracy than the comparative methods in all the bag distribution settings. Further, our algorithms have much lower error bounds on their accuracy scores. For the standard gaussian case, Algorithm 2 outperforms Algorithm 3 for larger m , possibly since with larger number of bags Algorithm 2 (which has higher sample complexity) can perform to its full potential. Conversely, we can observe that with larger bag sizes and dimensions, Algorithm 3 outperforms Algorithm 2 for smaller m .

For the noisy cases, from Tables 9 and 10 we observe that while the test accuracy degrades with large noise, it is fairly robust to small amounts of noise. This robustness is intuitive and we provide an explanation for the same in Appendix H (Lemma H.2).

(a) N ( 0 , Σ ) feature-vectors.

(b) N ( µ , Σ ) feature-vectors.

(a) N ( 0 , Σ ) feature-vectors.

(b) N ( µ , Σ ) feature-vectors.

G Class ratio estimation for LTFs

The work of [13] studies the problem of matching the classifier label proportion using a single sampled bag which they call class-ratio (CR) learning as distinct from LLP. Indeed, in LLP the goal is to learn an accurate instance-level classifier from multiple sampled bags, whereas CRlearning does not guarantee instance-level performance. Further, similar to Prop. 18 of [13], CR learning LTFs over Gaussians is easy: for a bag B = { x ( i ) } n i =1 of iid Gaussian points, a random unit vector r has distinct inner products { s i := r T x ( i ) } n i =1 with probability 1. The LTFs { pos ( r T x -s ) | s ∈ {-∞ , s 1 , . . . , s n }} achieve all possible target label proportions { j/n } n j =0 , and one can then apply the generalization error bound in Thm. 4 of [13].

H Analysis of a Mixture of Label Sums

Definition H.1 (Mixed Bag Oracle) . Given a set of bag oracles Ex ( f, D , q, k ) for k ∈ { 0 , . . . , q } and p = ( p 0 , . . . , p q ) ∈ ∆ q where ∆ q is a q -simplex, a mixed bag oracle Ex ( f, D , q, p ) samples a bag size k from Multinoulli ( p ) distribution 2 and then samples a bag from Ex ( f, D , q, k ) .

Let Σ D be the covariance matrix of difference of a pair of vectors sampled u.a.r without replacement from Ex ( f, D , q, p ) and Σ B be the covariance matrix of vectors sampled u.a.r from Ex ( f, D , q, p ) . If Σ Dk is the covariance matrix of difference of a pair of vectors sampled u.a.r without replacement from Ex ( f, D , q, k ) and Σ Bk be the covariance matrix of vectors sampled u.a.r from Ex ( f, D , q, k ) then we have the following

Using the above, we prove the following geometric error bound which is analogous to Lemma 4.1.

H.1 Proof of Lemma H.2

We define and bound the following useful quantities based on q , λ max, λ min and p .

Definition H.3. Define, (i) κ 1 ( k ) := ( 2 k q -1 ) 2 2 π so that 0 ≤ κ 1 ( k ) ≤ 2 /π , (ii) κ 2 ( k ) := 1 q -1 k q ( 1 -k q ) 16 π so that 16 πq 2 ≤ κ 2 ( k ) ≤ 4 π ( q -1) whenever 1 ≤ k ≤ q -1 , (iii) κ 3 ( p ) := ∑ q k =0 p 2 k κ 2 ( k ) ∑ q k =0 p 2 k -∑ q k =0 p 2 k κ 1 ( k ) so that 16 ∑ q -1 k =1 p 2 k πq 2 ∑ q k =0 p 2 k ≤ κ 3 ( p ) , and (iv) θ ( p ) := 2 λ max λ min ( 1 2 ∑ q k =0 p 2 k -max(0 , 2 ∑ q k =0 κ 1 ( k ) -∑ q k =0 p 2 k κ 2 ( k )) + 1 ∑ q k =0 p 2 k -∑ q k =0 p 2 k κ 1 ( k ) ) so that θ ( p ) ≤ 3 λ max (1 -2 /π ) λ min ( ∑ q k =0 p 2 k ) .

Lemma H.4. The ratio ρ ( r ) := r T Σ D r / r T Σ B r is maximized when r = ± r ∗ . Moreover,

Proof. The proof follows directly from Lemma 4.4 which gives us the expression for Σ Bk and Σ Dk and (54). Once the expression for ρ ( r ) is obtained, it is easy to see that since | γ ( r ) | ≤ 1 , ρ ( r ) maximizes when γ ( r ) = ± 1 and thus when r = ± r ∗ .

Proof. (of Lemma H.2) By Lemma 3.1, taking m ≥ O ( ( d/ε 2 1 ) log( d/δ ) ) ensures that ‖ E B ‖ 2 ≤ ε 1 λ max and ‖ E D ‖ 2 ≤ ε 1 λ max w.p. at least 1 -δ where E B = ˆ Σ B -Σ B and E D = ˆ Σ D -Σ D . We start by defining ˆ ρ ( r ) := r T ˆ Σ D r r T ˆ Σ B r which is the equivalent of ρ using the estimated matrices.

Observe that it can be written as ˆ ρ ( r ) = r T Σ B r + r T E B r r T Σ D r + r T E D r . Using these we can obtain the following bound on ˆ ρ : for any r ∈ R d , | ˆ ρ ( r ) -ρ ( r ) | ≤ θ ( p ) ε 1 | ρ ( r ) | w.p. at least 1 -δ (*) as long as ε 1 ≤ ( ∑ q k =0 p 2 k -∑ q k =0 p 2 k κ 1 ( k )) 2 λ min λ max , which we shall ensure. This is obtained as follows.

Define that

e

|

(

λ

min

e

(

|

r

)

ˆ

ρ

(

r

(

2

a

(

r

)

r

)

q

k

∑

/a

)

(

r

))

-

ρ

=0

/

(

:=

ε

1

r

λ

p

2

k

T

Σ

-

(1 +

r

)

D

,

f

|

f

ε

1

r

(

)

r

, b

(

r

(

)

r

)

(

/b

θ

(

p

)

0

(

,

r

ρ

(

:=

ε

2

1

λ

q

Σ

B

,

|

r

T

∑

))

r

)

.

.

For convenience we denote the normalized projection of any vector r as ˜ r := Σ r ‖ Σ 1 / 2 r ‖ 2 . Now let ˜ r ∈ S d -1 be a vector such that min {‖ ˜ r -˜ r ∗ ‖ 2 , ‖ ˜ r +˜ r ∗ ‖ 2 } ≥ ε 2 . Hence, using the definitions from Lemma H.4, | γ ( r ) | ≤ 1 -ε 2 2 / 2 while γ ( r ∗ ) = 1 which implies ρ ( r ∗ ) -ρ ( r ) ≥ κ 3 ( p ) ε 2 2 / 2 . Note that ρ ( r ) ≤ ρ ( r ∗ ) = 2 + κ 3 ( p ) . Choosing ε 1 < κ 3 ( p ) 4 θ (2+ κ 3 ( p )) ε 2 2 , we obtain that ρ ( r ∗ )(1 -θ ( p ) ε 1 ) > ρ ( r )(1+ θ ( p ) ε 1 ) . Using this along with the bound (*) we obtain that w.p. at least 1 -δ , ˆ ρ ( r ∗ ) > ˆ ρ ( r ) when ε 2 > 0 . Since our algorithm returns ˆ r as the maximizer of ˆ ρ , w.p. at least 1 -δ we get min {‖ ˜ r -˜ r ∗ ‖ 2 , ‖ ˜ r +˜ r ∗ ‖ 2 } ≤ ε 2 . Using Lemma 2.1, min {‖ ˆ r -r ∗ ‖ 2 , ‖ ˆ r + r ∗ ‖ 2 } ≤ 4 √ λ max λ min ε 2 . Substituting ε 2 = ε 4 √ λ min λ max , ‖ r -r ∗ ‖ 2 ≤ ε w.p. at least 1 -δ . The conditions on ε 1 are satisfied by taking it to be ≤ O ( κ 3 ( p ) ε 2 λ min θ ( p )(2+ κ 3 ( p )) λ max ) , and thus we can take m ≥ O ( ( d/ε 4 ) log( d/δ ) ( λ max λ min ) 2 θ ( p ) 2 ( 1 κ 3 ( p ) ) 2 ) . Taking m ≥ O ( ( d/ε 4 ) log( d/δ ) ( λ max λ min ) 4 q 4 ( 1 / ∑ q -1 k =1 p 2 k ) 2 ) satisfies this using bounds in Defn. H.3. This completes the proof.

r

, f

p

E

q

D

k

r

T

(∑

p

2

k

κ

2

-

∑

=0

(

q

k

2

k

)

p

2

k

))

2

k

(

r

)

-

:=

q

T

r

E

B

∑

2

k

=0

k

p

κ

.

κ

1

r

1

(

Notice that

(

.

k

)

Thus,

)

and we get

|

r

b

(

)

ˆ

r

r

ρ

(

)

/ρ

(

) = (1 +

k

))

λ

min

λ

max allows us to claim that

1 / 2

We observe that the sample complexity bound is worse for { p 1 , . . . , p q -1 } which are not concentrated i.e., the probability of the label sum is supported over many different values. This occurs for e.g. in the noisy setting when the label flip noise is large (see Appendix F).

2

k

k

p

=0

Taking

a

κ

r

(

1

ε

, e

r

(

)

r

)

(

1

k

)

:=

λ

-

∑

(

∑

min

q

q

k

k

=0

2

=0

k

p

=0

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLearning from label proportions allow training data to aggregate into sets of feature vectors with sum or average of their labels for each set as a label. As supervised learning, the goal is to classify test set of instances and minimize error of the classifier. This work focuses on learnability of LLP over Gaussian distributions of linear threshold functions. Although PAC learnability of LTFs has been studies in the LLP setting, PAC-LLP is intractable to learn LTFs using LTFs. This work formalize the problem into Gaussian distribution, which makes LTFs PAC-learnable. The approach is directly to maximize the instance-level accuracy on the distribution.\n\nSTRENGTHS:\n1. PAC-learnability of LTFs in the LLP setting (which is a very interesting topic) is well-defined as Definition 1.2 and well-addressed by Theorem 1.3, 1.4 and 1.5, especially considering the case for k = q/2.\n2. The proof of Theorem 1.4 is sound and solid coming with an algorithm and lemmas (even proofs for lemmas).\n3. As a theoretical work, it is nice to have a section for experimental results, although the datasets look relatively simple.\n\nWEAKNESSES:\n1. For Theorem 1.3 and Theorem 1.5, it is better to provide one or two sentences for each about proofs instead of putting everything in the appendix. You dont have to provide a separate section like Theorem 1.4. Just few sentences please.\n2. The writing looks less readable for some sentences. Please refer to limitations section.\n\nQUESTIONS:\n1. In Theorem 1.5, what is \\lambda? What are \\lambda_max and \\lambda_min? Please clarify notations like what you did on Line 126.\n2. What are relationships for r* and r head in Theorem 1.3, 1.4 and 1.5? I know you explained this on Line 145. Please put it ahead.\n\nLIMITATIONS:\n1. If possible, please don’t include citations in the abstract. \n2. On Line 59, I guess you typed one more “level”.\n3. Please make “Theorem” (Like on Line 247 or 257) and “Thms.” (Like on Line 128) consistent (Use either one).\n4. Need a space between “independence” and “i.e.,” on Line 132.\n5. Similarly, for Line 148, need a space between “directions” and “We”.\n6. You should read over your manuscript carefully and try to rephrase some sentences to make them more readable (I wont correct them here verbatim).\n- Some sentences have no subject.\n- Some sentences have no periods or spaces like limitations 4 and 5.\n- Some pronouns refer ambiguously.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis theoretical paper investigates the learnability of linear threshold functions (LTFs) in the learning from label proportions (LLP) setting. When the feature-vectors are distributed according to a Gaussian distribution and conditioned on their underlying labels, LTFs can be efficiently properly learnt. For the general Gaussian distribution, this paper develops a principal component algorithm which estimates the means and covariance matrices using subgaussian concentration bounds. Generalization bounds from bag classification error to instance classification error are also provided to resolve the ambiguity between the obtained solutions. The experimental results validate the effectiveness of the proposed techniques.\n\nSTRENGTHS:\nThe learnability of LTFs in the LLP setting is an important issue, this work provides some novel and significant theoretical results.\n\nWEAKNESSES:\n$\\bullet$ From Theorem 1.3 to 1.5, as the distribution becomes more general, the sample complexity required to efficiently properly learn LTFs also increases, and the difference between these sample complexities seems to be obvious. Numerical results on the difference between these sample complexities and associated discussion and analysis are necessary to more intuitively elucidate these theoretical results.\n\n$\\bullet$ Intuitively, the theoretical results in this paper will depend on the label proportion $k/q$, but in the end, it is mainly implemented on the bag size $q$, i.e., the order of the sample complexity does not explicitly reflect the label proportion $k/q$. While this is not mathematically problematic, it would be helpful to add some remarks to illustrate this point.\n\n$\\bullet$ The readability of the paper needs to be improved. More introductions to the problem setting and related definitions of LLP, as well as further illustrations and explanations about the existing theoretical work will help readers better understand the theoretical results in this paper. In addition, some typos also harm the readability.\n\nTypos:\n\n$\\bullet$ Line 27: they're labels The goal -> their labels. The goal\n\n$\\bullet$ Line 132: independencei.e., -> independence, i.e.,\n\n$\\bullet$ Line 146: $r = q/2$ -> $k = q/2$\n\n$\\bullet$ Line 148: directionsWe -> directions. We\n\n$\\bullet$ Line 350: bag satisfaction -> bag classification\n\n$\\bullet$ Some typos in the Supplementary Material also need to be checked and corrected.\n\nQUESTIONS:\nPlease refer to the Weaknesses for details.\n\nLIMITATIONS:\nThis work does not seem to have any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the problem of learning linear threshold functions (LTFs, aka linear classifiers) under the setting of learning from label proportions (LLP), where the training data are \"bags\" (aka sets) of instances, and the training labels are classifier proportions of the instances in the bag. Under the assumption that the instances are gaussian distributed, bags are instances sampled iid, and the size of the bags are upper bounded, the authors show that it is possible to efficiently (properly) learning LTFs.\n\nSTRENGTHS:\nThe problem setting of PAC-LLP deserves more attention.\nThe authors gives good justification for why LLP matters, e.g., privacy and legal.\nIn light of the previous theory works on LLP that shows NP-hardness of PAC-LLP learning LTFs,\nthis current work is surprising, relvant, and interesting to the ML community.\nTheorem 1.4 gives an algorithm that highlights the interesting geometry and the clever exploitation of the subtle difference between sampling with and without replacement.\n\nWEAKNESSES:\nAfter Section 1.1, it will be useful for the reader if there is an overview of the paper. Something along the lines of\n\"Sec 1.3 are the main results. Sec 1.4 gives proof sketch of the main results and high level description of the algorithms. Section 3 state these algorithms precisely...\"\n\nFor Theorem 1.3, it will be useful to interpret the sample complexity. Is the sample complexity essentially the difficulty of estimating the mean of the bag vectors? If so, the authors should make this explicit.\n\nFor Theorem 1.4 and 1.5, are the lambdas eigenvalues of the covariance matrices? Please explicitly say so.\n\nQUESTIONS:\nWhy is Definition 1.2 called the bag oracle? Isn't it just the distribution over the training data (of bags)?\n\nDoes the authors believe that the results are tight? In other words, is LLP under the considered data assumption essentially equivalent to mean/covariance estimation?\n\nIt will be good if there is a discussion regarding the sample-complexities in LLP versus ordinary classification.\nFrom this point of view, what is the \"cost\" of only having access to bagged data?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies PAC learning when the training data is aggregated into sets or bags of feature vectors. For each bag, we observe the feature vectors of these bags and only the average of the labels in the bag. They focus on the case when the feature vectors are distributed according to a Gaussian distribution, and the hypothesis class is the set of Linear Threshold Functions (LTFs). Their main results include polynomial time algorithms for learning the correct halfspace when the Gausssian is either standard or skewed and the correct LTF is either homogeneous or not. Finally, they compare their Algorithm experimentally with various procedures from prior work.\n\nSTRENGTHS:\nThis paper proposes a very interesting direction for learning under label aggregation, which bypasses the lower bounds that existed for this setting in prior work. I think it opens the way for more interesting problems to be considered in this setting, by possibly changing the assumptions on the distribution or the hypothesis class. All the claims are sufficiently explained and the presentation is generally clear. \n\nThe authors introduce some interesting technical novelties to extract information about the true vector of the LTF using the aggregated labels. In particular, the idea of comparing the variance of a random pair with and without replacement and finding that it is maximized in the direction of the true vector $\\mathbf{r^\\star}$ is very elegant and leads to some interesting algorithms, which can be implemented in polynomial time using PCA. Moreover, the techniques for handling non-centered and skewed Gaussians with unknown parameters are non-trivial and interesting.\n\nWEAKNESSES:\nThis is not a significant weakness, but the technical presentation in some parts of the paper seems quite dense and not necessarily adding to the understanding of this work. For example, in the proof of Theorem 1.4 or Lemma 4.2, too much emphasis is given in the calculation of the optimal setting of parameters and sample complexity. I would prefer a more intuitive explanation at times, although I'm not sure it that's possible.\n\nQUESTIONS:\nHow tight are the sample complexities provided in Theorem 1.4 and 1.5 in terms of their dependence on the minimum and maximum eigenvalues of the covariance matrix and the size of the bag? In Theorem 1.5, we also see a term involving $l$, which depends on the parameters of the true LTF. Is such a dependence necessary?\n\nLIMITATIONS:\nNo limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}