Learnability of Linear Thresholds from Label Proportions

1 Introduction . A linear threshold function ( LTF ) over the d-dimensional feature-vectors x is given by pos ( g ( x ) ) for some linear function g ( x1 , . . . , xd ) = ∑d i=1 cixi + cd+1 , where pos ( z ) : = 1 { z > 0 } . Also known as linear classifiers or halfspaces , LTFs are one of the most fundamental classes studied in computational learning and lie at the core of several machine learning algorithms such as Perceptron [ 25 , 21 ] and SVM [ 7 ] . It is known [ 4 ] that LTFs are properly learnable in the supervised learning setup : given a training set of labeled feature-vectors which are consistent with some LTF , using polynomial-time linear programming one can efficiently find one such LTF . In this work we initiate an investigation into the proper learnability of LTFs in the framework of learning from label proportions ( LLP ) . In this , instead of labels for each feature-vector , the training data consists of bags ( subsets ) of feature-vectors along with the proportion of labels in each bag . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Given a collection of bags with their respective label proportions which are consistent with ( i.e , are satisfied by ) some unknown feature-vector level classifier , the goal is to find a classifier ( from the same concept class for proper learning ) satisfying the most number of bags . Such a model was formalized by Yu et . al [ 30 ] who showed bounds on the generalization error for predicting the label proportion on unseen bags . When all bags are unit-sized this becomes the usual supervised learning in which as we noted above LTFs are efficiently properly learnable . A natural question is whether this remains true even when larger bag sizes are allowed . In particular , we consider the simplest setting with bags of size at most two and study the following question : Given a collection of bags , all of size at most two and guaranteed to be satisfied by some unknown LTF , what is the maximum fraction of bags that can be satisfied using an efficiently computable LTF ? There naturally are two types of bags : monochromatic bags whose feature vectors have the same label , and the non-monochromatic ones whose feature-vectors have different labels and are necessarily of size two . Clearly , the monochromatic bags determine the labels of their constituent feature-vectors . Therefore , for a collection of bags consistent with some LTF , one can efficiently find an LTF using linear programming to satisfy all the monochromatic bags . However , such an algorithm is not guaranteed to satisfy any non-monochromatic bag . In traditional supervised learning of LTFs , the trivial algorithms of taking the best of any LTF or its complement , or a random homogeneous LTF , both achieve an accuracy of half . Unfortunately , these algorithms provide no such guarantees in the LLP setting , even for bags of size at most two . At a high level , in LLP the objective is given by comparing the predicted labels of a bag ’ s feature-vectors with each other , unlike supervised learning where they are only compared with observed ones . This seems to qualitatively change the nature of the problem in terms of the applicable techniques . We first provide an algorithm that satisfies ( in expectation ) at least half of the non-monochromatic bags and one-fourths of the monochromatic ones . This can be combined with the linear programming algorithm which satisfies all monochromatic bags ( and possibly none of the non-monochromatic ones ) to obtain an algorithm which outputs an LTF satisfying ( 2/5 ) -fraction of the bags . We also show that for the special case of the feature vectors being d-dimensional boolean and the bags being consistent with an OR ( boolean disjunction ) the algorithmic guarantees improve by an Ω ( 1/d ) additional factor . 1.1 Previous Work on LLP . The LLP problem naturally arises in many real world scenarios where the labels are not available individually but only as proportions for bags of feature-vectors due to privacy [ 28 ] and legal [ 26 ] reasons , high label supervision cost [ 6 ] or technical limitations of labeling mechanisms [ 9 ] . Some of the earlier works [ 8 , 16 , 22 , 26 ] applied supervised learning methods such as MCMC , clustering , and linear classifiers to the LLP problem . The work of [ 24 ] assumed an exponential generative model and proved performance bounds under assumptions on the distribution of the bags , and was further generalized by [ 23 ] while the work of [ 29 ] proposed novel proportional SVM based algorithms . Subsequently , approaches based on deep neural nets for large-scale and multi-class data [ 18 , 10 , 19 ] , as well as bag pre-processing techniques [ 27 ] have been developed for LLP . In contrast to the long line above of application focused research in LLP , the theoretical study of algorithmic and complexity issues in LLP has been rather limited . On the complexity side , [ 13 ] studied a one-bag version of this problem where the observed global label proportion is given over all the feature-vectors and the goal is to find a classifier to minimize the deviation from this . They showed , among other results that common concept classes such as monotone-ORs can not be learnt in their model to arbitrary accuracy . The LLP model which we study – of predicting the bag label proportions – was first formalized in the work of [ 30 ] who also bounded the generalization error when taking the ( bag , label-proportion ) -pairs as instances sampled iid from some distribution . Their bound has the same dependence on the featurevector classifier ’ s VC-dimension as in supervised learning , with an additional logarithmic dependence on the bag size . While the loss function they consider is slightly different , their bound can be applied to minimizing the number of unsatisfied bags of size two ( see Appendix E of the supplemental ) . In the rest of this section we define the problem of learning LTFs from label proportions , describe our results and the previous related work along with an informal description of our methods and proof techniques . 1.2 Problem Definition . Let X = { x1 , . . . , xn } ⊆ Rd be the set of feature-vectors , and bags B = { B1 , . . . , Bm } ⊆ ( X 1 ) ∪ ( X 2 ) , i.e . each bag is of size at most 2 . The label set is binary i.e. , { 0 , 1 } , and for each bag Bk there is a proportion σk which is the average of the labels of the vectors in the bag , satisfying σk ∈ { 0 , 1 } if |Bk| = 1 and σk ∈ { 0 , 0.5 , 1 } if |Bk| = 2 . If σk 6= 0.5 then Bk is said to be monochromatic i.e. , bags which have same labels for their feature-vectors . The remaining bags Bk necessarily of size 2 having σk = 0.5 are called non-monochromatic . A bag Bk ∈ B is satisfied by some f : X→ { 0 , 1 } if ( ∑ x∈Bk f ( x ) ) / |Bk| = σk . An instance of LLP-LTF is given ( X , B = { Bk } mk=1 , { σk } mk=1 ) . It is said to be satisfiable if there exists an LTF that satisfies all the bags . The goal is to find an LTF that satisfies the most bags . We denote by LLP-OR the special case when X has boolean vectors and the guaranteed LTF for satisfiable instances is an OR . A further special case is when the guaranteed OR is monotone i.e. , it has no negated variables . 1.3 Our Results . We provide the following algorithmic guarantees for LLP-LTF on bags of size at most 2 . Theorem 1.1 . There is a polynomial time algorithm which given a satisfiable instance of LLP-LTF on bags of size at most 2 , computes an LTF that satisfies at least ( 2/5 ) -fraction of the bags . If all bags are non-monochromatic this satisfies at least half of the bags . For satisfiable LLP-OR on bags of size at most 2 with d-dimensional boolean feature-vectors , there is a polynomial-time algorithm computing an LTF satisfying at least ( 2/5 + γ0/d ) -fraction of the bags in general , and ( 1/2 + γ0/d ) -fraction if all bags are non-monochromatic , for some absolute constant γ0 > 0 . The above theorem for LLP-LTF is proved in Sec . 2 with the proof for LLP-OR provided in Sec . 2.1 . The algorithms use random hyperplane rounding of vectors , so the above guarantees are in expectation . They can however be derandomized by the sophisticated procedure given in [ 20 ] and we provide an explanation in Appendix A of the supplemental . Our result on the intractability of LLP-OR ( an thereby of LLP-LTF ) is stated below . Theorem 1.2 . Given a satisfiable instance of LLP-OR consisting only of non-monochromatic bags each of size 2 s.t . there is a monotone OR that satisfies all bags , it is NP-hard to find any function of ` LTFs that satisfies ( 1/2 + δ ) -fraction of the bags , for any constants ` ∈ Z+ , δ > 0 . The proof follows via a reduction from a variant of the Label Cover problem . We include the guarantees of the hardness reduction as Theorem 3.3 which directly implies Theorem 1.2 . However , due to lack of space we defer the formal proof of Theorem 3.3 to Appendices C and D of the supplemental , while including an informal description in Sec . 1.5 . We note that Theorem 1.2 shows the optimality of the approximation factor obtained in Theorem 1.1 for non-monochromatic bags . Previous Related Work . As mentioned earlier , the proper supervised-learnability of LTFs [ 4 ] is well known without any distributional assumptions . In the presence of adverserial label noise however , the problem was shown to be NP-hard [ 17 ] and the works of [ 1 , 2 , 5 ] proved stronger inapproximability bounds with the optimal ( 1/2 + ε ) -factor hardness obtained independently by [ 11 ] and [ 15 ] , further generalized by [ 3 ] to apply for constant degree polynomial thresholds as hypotheses . Subsequently , [ 12 ] showed that this also holds for the special case of learning OR with an LTF which was recently strengthened by [ 14 ] to also rule out functions of constantly many LTFs as hypotheses . 1.4 Overview of Algorithmic Results . Algorithm for LLP-LTF . Given a satisfiable instance of LLP-LTF we can first assume by slightly translating the constant part of the unknown satisfying LTF that it classifies all the feature-vectors of the instance with a non-zero margin . Next , we observe that there are strict quadratic constraints ( over the coefficients of the LTF ) that distinguish the monochromatic from the non-monochromatic bags . Indeed , consider the unknown LTF pos ( 〈c∗ , x〉 ) ( WLOG by appending a last coordinate with value 1 to all feature-vectors x of the instance ) that satisfies all bags . Then , 〈c∗ , xi〉〈c∗ , xj〉 is positive for monochromatic bags and negative for the non-monochromatic ones whose constituent vectors are xi and xj . Taking C = c∗cT∗ we obtain the same conditions for x T i Cxj , which are constraints linear in the entries of positive semi-definite ( psd ) C , and can be efficiently solved for C using a semi-definite program ( SDP ) . Using the psd decomposition C = LTL we map xi to the vector zi : = Lxi so that 〈zi , zj〉 = xTi Cxj . From here , the random hyperplane rounding of zi to 〈g , zi〉 ( g ← N ( 0 , 1 ) d ) yields an LTF pos ( h ( xi ) ) = pos ( 〈g , zi〉 ) that separates ( in expectation ) the vectors of at least half of the non-monochromatic bags and at most half of the monochromatic ones . Taking the better of pos ( h ( x ) ) and pos ( −h ( x ) ) satisfies in expectation at least half of the non-monochromatic bags and at least one-fourths of the monochromatic ones . Combining this with the known linear programming algorithm that satisfies all the monochromatic bags we obtain the desired ( 2/5 ) -approximation . Algorithm LLP-OR . Given a satisfying OR we can write an equivalent LTF with a margin property – the thresholded linear form has value either ( −1/2 ) if the OR evaluates to 0 , and otherwise takes values in the range [ 1/2 , d ] . This allows us to add margin terms in the SDP constraints corresponding to the bags . The rest of the algorithm mapping xi to zi remains the same , while the strengthened constraints imply an extra Ω ( 1/d ) in the probabilities that the random hyperplane rounding separates the vectors of non-monochromatic bags and does not separate those of monochromatic bags , yielding a corresponding improvement in the fraction of satisfied bags .

Summary: This work considers the problem of learning a linear threshold function (LTF) from label proportions. In particular they consider the special case where both single (as in the standard PAC learning setting) labeled (as 0 or 1) examples are observed or pairs of points $(x_1, x_2)$ together with their average label (which can be 0, 1, or 1/2) are observed.  They give two main results: a polynomial time algorithm that finds an LTF that satisfies at least 2/5 of the labeled examples (pairs and singletons together) and a lower bound that shows that finding an LTF that satisfies more than 1/2 of the labeled examples is NP-Hard.  For the special case where only mixed pairs, i.e., with label 1/2, are observed, the provided algorithm satisfies 1/2 of the labeled examples matching the lower bound (which also assumes only mixed pairs).