Learnability of Linear Thresholds from Label Proportions

1 Introduction . A linear threshold function ( LTF ) over the d-dimensional feature-vectors x is given by pos ( g ( x ) ) for some linear function g ( x1 , . . . , xd ) = ∑d i=1 cixi + cd+1 , where pos ( z ) : = 1 { z > 0 } . Also known as linear classifiers or halfspaces , LTFs are one of the most fundamental classes studied in computational learning and lie at the core of several machine learning algorithms such as Perceptron [ 25 , 21 ] and SVM [ 7 ] . It is known [ 4 ] that LTFs are properly learnable in the supervised learning setup : given a training set of labeled feature-vectors which are consistent with some LTF , using polynomial-time linear programming one can efficiently find one such LTF . In this work we initiate an investigation into the proper learnability of LTFs in the framework of learning from label proportions ( LLP ) . In this , instead of labels for each feature-vector , the training data consists of bags ( subsets ) of feature-vectors along with the proportion of labels in each bag . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Given a collection of bags with their respective label proportions which are consistent with ( i.e , are satisfied by ) some unknown feature-vector level classifier , the goal is to find a classifier ( from the same concept class for proper learning ) satisfying the most number of bags . Such a model was formalized by Yu et . al [ 30 ] who showed bounds on the generalization error for predicting the label proportion on unseen bags . When all bags are unit-sized this becomes the usual supervised learning in which as we noted above LTFs are efficiently properly learnable . A natural question is whether this remains true even when larger bag sizes are allowed . In particular , we consider the simplest setting with bags of size at most two and study the following question : Given a collection of bags , all of size at most two and guaranteed to be satisfied by some unknown LTF , what is the maximum fraction of bags that can be satisfied using an efficiently computable LTF ? There naturally are two types of bags : monochromatic bags whose feature vectors have the same label , and the non-monochromatic ones whose feature-vectors have different labels and are necessarily of size two . Clearly , the monochromatic bags determine the labels of their constituent feature-vectors . Therefore , for a collection of bags consistent with some LTF , one can efficiently find an LTF using linear programming to satisfy all the monochromatic bags . However , such an algorithm is not guaranteed to satisfy any non-monochromatic bag . In traditional supervised learning of LTFs , the trivial algorithms of taking the best of any LTF or its complement , or a random homogeneous LTF , both achieve an accuracy of half . Unfortunately , these algorithms provide no such guarantees in the LLP setting , even for bags of size at most two . At a high level , in LLP the objective is given by comparing the predicted labels of a bag ’ s feature-vectors with each other , unlike supervised learning where they are only compared with observed ones . This seems to qualitatively change the nature of the problem in terms of the applicable techniques . We first provide an algorithm that satisfies ( in expectation ) at least half of the non-monochromatic bags and one-fourths of the monochromatic ones . This can be combined with the linear programming algorithm which satisfies all monochromatic bags ( and possibly none of the non-monochromatic ones ) to obtain an algorithm which outputs an LTF satisfying ( 2/5 ) -fraction of the bags . We also show that for the special case of the feature vectors being d-dimensional boolean and the bags being consistent with an OR ( boolean disjunction ) the algorithmic guarantees improve by an Ω ( 1/d ) additional factor . 1.1 Previous Work on LLP . The LLP problem naturally arises in many real world scenarios where the labels are not available individually but only as proportions for bags of feature-vectors due to privacy [ 28 ] and legal [ 26 ] reasons , high label supervision cost [ 6 ] or technical limitations of labeling mechanisms [ 9 ] . Some of the earlier works [ 8 , 16 , 22 , 26 ] applied supervised learning methods such as MCMC , clustering , and linear classifiers to the LLP problem . The work of [ 24 ] assumed an exponential generative model and proved performance bounds under assumptions on the distribution of the bags , and was further generalized by [ 23 ] while the work of [ 29 ] proposed novel proportional SVM based algorithms . Subsequently , approaches based on deep neural nets for large-scale and multi-class data [ 18 , 10 , 19 ] , as well as bag pre-processing techniques [ 27 ] have been developed for LLP . In contrast to the long line above of application focused research in LLP , the theoretical study of algorithmic and complexity issues in LLP has been rather limited . On the complexity side , [ 13 ] studied a one-bag version of this problem where the observed global label proportion is given over all the feature-vectors and the goal is to find a classifier to minimize the deviation from this . They showed , among other results that common concept classes such as monotone-ORs can not be learnt in their model to arbitrary accuracy . The LLP model which we study – of predicting the bag label proportions – was first formalized in the work of [ 30 ] who also bounded the generalization error when taking the ( bag , label-proportion ) -pairs as instances sampled iid from some distribution . Their bound has the same dependence on the featurevector classifier ’ s VC-dimension as in supervised learning , with an additional logarithmic dependence on the bag size . While the loss function they consider is slightly different , their bound can be applied to minimizing the number of unsatisfied bags of size two ( see Appendix E of the supplemental ) . In the rest of this section we define the problem of learning LTFs from label proportions , describe our results and the previous related work along with an informal description of our methods and proof techniques . 1.2 Problem Definition . Let X = { x1 , . . . , xn } ⊆ Rd be the set of feature-vectors , and bags B = { B1 , . . . , Bm } ⊆ ( X 1 ) ∪ ( X 2 ) , i.e . each bag is of size at most 2 . The label set is binary i.e. , { 0 , 1 } , and for each bag Bk there is a proportion σk which is the average of the labels of the vectors in the bag , satisfying σk ∈ { 0 , 1 } if |Bk| = 1 and σk ∈ { 0 , 0.5 , 1 } if |Bk| = 2 . If σk 6= 0.5 then Bk is said to be monochromatic i.e. , bags which have same labels for their feature-vectors . The remaining bags Bk necessarily of size 2 having σk = 0.5 are called non-monochromatic . A bag Bk ∈ B is satisfied by some f : X→ { 0 , 1 } if ( ∑ x∈Bk f ( x ) ) / |Bk| = σk . An instance of LLP-LTF is given ( X , B = { Bk } mk=1 , { σk } mk=1 ) . It is said to be satisfiable if there exists an LTF that satisfies all the bags . The goal is to find an LTF that satisfies the most bags . We denote by LLP-OR the special case when X has boolean vectors and the guaranteed LTF for satisfiable instances is an OR . A further special case is when the guaranteed OR is monotone i.e. , it has no negated variables . 1.3 Our Results . We provide the following algorithmic guarantees for LLP-LTF on bags of size at most 2 . Theorem 1.1 . There is a polynomial time algorithm which given a satisfiable instance of LLP-LTF on bags of size at most 2 , computes an LTF that satisfies at least ( 2/5 ) -fraction of the bags . If all bags are non-monochromatic this satisfies at least half of the bags . For satisfiable LLP-OR on bags of size at most 2 with d-dimensional boolean feature-vectors , there is a polynomial-time algorithm computing an LTF satisfying at least ( 2/5 + γ0/d ) -fraction of the bags in general , and ( 1/2 + γ0/d ) -fraction if all bags are non-monochromatic , for some absolute constant γ0 > 0 . The above theorem for LLP-LTF is proved in Sec . 2 with the proof for LLP-OR provided in Sec . 2.1 . The algorithms use random hyperplane rounding of vectors , so the above guarantees are in expectation . They can however be derandomized by the sophisticated procedure given in [ 20 ] and we provide an explanation in Appendix A of the supplemental . Our result on the intractability of LLP-OR ( an thereby of LLP-LTF ) is stated below . Theorem 1.2 . Given a satisfiable instance of LLP-OR consisting only of non-monochromatic bags each of size 2 s.t . there is a monotone OR that satisfies all bags , it is NP-hard to find any function of ` LTFs that satisfies ( 1/2 + δ ) -fraction of the bags , for any constants ` ∈ Z+ , δ > 0 . The proof follows via a reduction from a variant of the Label Cover problem . We include the guarantees of the hardness reduction as Theorem 3.3 which directly implies Theorem 1.2 . However , due to lack of space we defer the formal proof of Theorem 3.3 to Appendices C and D of the supplemental , while including an informal description in Sec . 1.5 . We note that Theorem 1.2 shows the optimality of the approximation factor obtained in Theorem 1.1 for non-monochromatic bags . Previous Related Work . As mentioned earlier , the proper supervised-learnability of LTFs [ 4 ] is well known without any distributional assumptions . In the presence of adverserial label noise however , the problem was shown to be NP-hard [ 17 ] and the works of [ 1 , 2 , 5 ] proved stronger inapproximability bounds with the optimal ( 1/2 + ε ) -factor hardness obtained independently by [ 11 ] and [ 15 ] , further generalized by [ 3 ] to apply for constant degree polynomial thresholds as hypotheses . Subsequently , [ 12 ] showed that this also holds for the special case of learning OR with an LTF which was recently strengthened by [ 14 ] to also rule out functions of constantly many LTFs as hypotheses . 1.4 Overview of Algorithmic Results . Algorithm for LLP-LTF . Given a satisfiable instance of LLP-LTF we can first assume by slightly translating the constant part of the unknown satisfying LTF that it classifies all the feature-vectors of the instance with a non-zero margin . Next , we observe that there are strict quadratic constraints ( over the coefficients of the LTF ) that distinguish the monochromatic from the non-monochromatic bags . Indeed , consider the unknown LTF pos ( 〈c∗ , x〉 ) ( WLOG by appending a last coordinate with value 1 to all feature-vectors x of the instance ) that satisfies all bags . Then , 〈c∗ , xi〉〈c∗ , xj〉 is positive for monochromatic bags and negative for the non-monochromatic ones whose constituent vectors are xi and xj . Taking C = c∗cT∗ we obtain the same conditions for x T i Cxj , which are constraints linear in the entries of positive semi-definite ( psd ) C , and can be efficiently solved for C using a semi-definite program ( SDP ) . Using the psd decomposition C = LTL we map xi to the vector zi : = Lxi so that 〈zi , zj〉 = xTi Cxj . From here , the random hyperplane rounding of zi to 〈g , zi〉 ( g ← N ( 0 , 1 ) d ) yields an LTF pos ( h ( xi ) ) = pos ( 〈g , zi〉 ) that separates ( in expectation ) the vectors of at least half of the non-monochromatic bags and at most half of the monochromatic ones . Taking the better of pos ( h ( x ) ) and pos ( −h ( x ) ) satisfies in expectation at least half of the non-monochromatic bags and at least one-fourths of the monochromatic ones . Combining this with the known linear programming algorithm that satisfies all the monochromatic bags we obtain the desired ( 2/5 ) -approximation . Algorithm LLP-OR . Given a satisfying OR we can write an equivalent LTF with a margin property – the thresholded linear form has value either ( −1/2 ) if the OR evaluates to 0 , and otherwise takes values in the range [ 1/2 , d ] . This allows us to add margin terms in the SDP constraints corresponding to the bags . The rest of the algorithm mapping xi to zi remains the same , while the strengthened constraints imply an extra Ω ( 1/d ) in the probabilities that the random hyperplane rounding separates the vectors of non-monochromatic bags and does not separate those of monochromatic bags , yielding a corresponding improvement in the fraction of satisfied bags .

Summary: This work studies the problem of learning from label proportions (LLP), that is given bags of vectors and given the average of the positive labels in each bag, we need to find an LTF that satisfies most of the bags. The authors study the case where the bags contain at most 2 vectors. They study two cases, one where the vectors are taken from $\mathcal X\subseteq \mathbb{R}^d$ and they are searching for an LTF; in the other case the vectors are taken from the $d$-dimensional hypercube $\{0,1\}^d$, and they are looking for an OR-LTF. In the first case, they provide an algorithm that satisfies $2/5$ of the bags and if all the bags are non-monochromatic (contain vectors with different labels) then the algorithm satisfies $1/2$ of the bags, for the other case they provide an algorithm that satisfies $(2/5 + \gamma_0/d)$ bags and if all the bags are non-monochromatic it satisfies $(1/2+\gamma_0/d)$ of the bags. Furthermore, for the last case, the authors provide an NP-hardness reduction to the Label Covering problem that shows that their result is qualitatively tight and that in general, we cannot hope for an algorithm that satisfies all the bags.