Learning from Label Proportions with Consistency Regularization

1 INTRODUCTION . In traditional supervised learning , a classifier is trained on a dataset where each instance is associated with a class label . However , label annotation can be expensive or difficult to obtain for some applications . Take the embryo selection as an example ( Hernández-González et al. , 2018 ) . To increase the pregnancy rate , clinicians would transfer multiple embryos to a mother at the same time . However , clinicians are unable to know the outcome of a particular embryo due to limitations of current medical techniques . The only thing we know is the proportion of embryos that implant successfully . To increase the success rate of embryo implantation , clinicians aim to select high-quality embryos through the aggregated results . In this case , only label proportions about groups of instances are provided to train the classifier , a problem setting known as learning from label proportions ( LLP ) . In LLP , each group of instances is called a bag , which is associated with a proportion label of different classes . A classifier is then trained on several bags and their associated proportion labels in order to predict the class of each unseen instance . Recently , LLP has attracted much attention among researchers because its problem setting occurs in many real-life scenarios . For example , the census data and medical databases are all provided in the form of label proportion data due to privacy issues ( Patrini et al. , 2014 ; Hernández-González et al. , 2018 ) . Other LLP applications include fraud detection ( Rueping , 2010 ) , object recognition ( Kuck & de Freitas , 2012 ) , video event detection ( Lai et al. , 2014 ) , and ice-water classification ( Li & Taylor , 2015 ) . The challenge in LLP is to train models without direct instance-level label supervision . To overcome this issue , prior work seeks to estimate either the individual label ( Yu et al. , 2013 ; Dulac-Arnold et al. , 2019 ) or the mean of each class by the label proportions ( Quadrianto et al. , 2009 ; Patrini et al. , 2014 ) . However , the methodology behind developing these models do not portray LLP situations that occur in real life . First , these models can be improved by considering methods that can better leverage unlabeled data . Second , these models assume that bags of data are randomly generated , which is not the case for many applications . For example , the data of population census are collected on region , age , or occupation with varying group sizes . Third , training these models requires a validation set with labeled data . It would be more practical if the process of model selection relies only on the label proportions . This paper aims to resolve the previous problems . Our main contributions are listed as follows : • We first apply a semi-supervised learning technique , consistency regularization , to the multi-class LLP problem . Consistency regularization considers an auxiliary loss term to enforce network predictions to be consistent when its input is perturbed . By exploiting the unlabeled instances , our method captures the latent structure of data and obtains the SOTA performance on three benchmark datasets . • We develop a new bag generation algorithm – the K-means bag generation , where training data are grouped by attribute similarity . Using this setup can help train models that are more applicable to actual LLP scenarios . • We show that it is possible to select models with a validation set consisting of only bags and associated label proportions . The experiments demonstrate correlation between baglevel validation error and instance-level test error . This potentially reduces the need of a validation set with instance-level labels . 2 PRELIMINARY . 2.1 LEARNING FROM LABEL PROPORTIONS . We consider the multi-class classification problem in the LLP setting in this paper . Let xi ∈ RD be a feature vector of i-th example and yi ∈ { 1 , . . . , L } be a class label of i-th example , where L is the number of different classes . We define e ( j ) to be a standard basis vector [ 0 , . . . , 1 , . . . , 0 ] with 1 at j-th position and ∆L = { p ∈ RL+ : ∑L i pi = 1 } to be a probability simplex . In the setting of LLP , each individual label yi is hidden from the training data . On the other hand , the training data are aggregated by a bag generation procedure . We are given M bags B1 , . . . , BM , where each bag Bm contains a set Xm of instances and a proportion label pm , defined by pm = 1 |Xm| ∑ i : xi∈Xm e ( yi ) , M⋃ m=1 Xm = { x1 , . . . , xN } . We do not require each subset to be disjoint . Also , each bag may have different size . The task of LLP is to learn an individual-level classifier fθ : RD → ∆L to predict the correct label y = arg maxi fθ ( x ) i for a new instance x . Figure 1 illustrates the setting of learning from label proportions in the multi-class classification ( Dulac-Arnold et al. , 2019 ) . 2.2 PROPORTION LOSS . The feasibility of the binary LLP setting has been theoretically justified by Yu et al . ( 2014 ) . Specifically , Yu et al . ( 2014 ) propose the framework of Empirical Proportion Risk Minimization ( EPRM ) , proving that the LLP problem is PAC-learnable under the assumption that bags are i.i.d sampled from an unknown probability distribution . The EPRM framework provides a generalization bound on the expected proportion error and guarantees to learn a probably approximately correct proportion predictor when the number of bags is large enough . Furthermore , the authors prove that the instance label error can be bounded by the bag proportion error . That is , a decent bag proportion predictor guarantees a decent instance label predictor . Based on the profound theoretical analysis , a vast number of LLP approaches learn an instance-level classifier by directly minimizing the proportion loss without acquiring the individual labels . To be more precise , given a bag B = ( X , p ) , an instance-level classifier fθ and a divergence function dprop : RL × RL → R , the proportion loss penalizes the difference between the real proportion label pm and the estimated proportion label p̂ = 1 |X| ∑ x∈X fθ ( x ) , which is an average of the instance predictions within a bag . Thus , the proportion loss Lprop can be defined as follows : Lprop ( θ ) = dprop ( p , p̂ ) . The commonly used divergence functions are L1 and L2 function in prior work ( Musicant et al. , 2007 ; Yu et al. , 2013 ) . Ardehaly & Culotta ( 2017 ) and Dulac-Arnold et al . ( 2019 ) , on the other hand , consider the cross-entropy function for the multi-class LLP problem . 2.3 CONSISTENCY REGULARIZATION . Since collecting labeled data is expensive and time-consuming , the semi-supervised learning approaches aim to leverage a large amount of unlabeled data to mitigate the need for labeled data . There are many semi-supervised learning methods , such as pseudo-labeling ( Lee , 2013 ) , generative approaches ( Kingma et al. , 2014 ) , and consistency-based methods ( Laine & Aila , 2016 ; Miyato et al. , 2018 ; Tarvainen & Valpola , 2017 ) . Consistency-based approaches encourage the network to produce consistent output probabilities between unlabeled data and the perturbed examples . These methods rely on the smoothness assumption ( Chapelle et al. , 2009 ) : if two data points xi and xj are close , then so should be the corresponding output distributions yi and yj . Then , the consistency-based approaches can enforce the decision boundary to traverse through the low-density region . More precisely , given a perturbed input x̂ taken from the input x , consistency regularization penalizes the distinction of model predictions between fθ ( x ) and fθ ( x̂ ) by a distance function dcons : RL × RL → R. The consistency loss can be written as follows : Lcons ( θ ) = dcons ( fθ ( x ) , fθ ( x̂ ) ) . Modern consistency-based methods ( Laine & Aila , 2016 ; Tarvainen & Valpola , 2017 ; Miyato et al. , 2018 ; Verma et al. , 2019 ; Berthelot et al. , 2019 ) differ in how perturbed examples are generated for the unlabeled data . Laine & Aila ( 2016 ) introduce the Π-Model approach , which uses the additive Gaussian noise for perturbed examples and chooses the L2 error as the distance function . However , a drawback to Π-Model is that the consistency target fθ ( x̂ ) obtained from the stochastic network is unstable since the network changes rapidly during training . To address this problem , Temporal Ensembling ( Laine & Aila , 2016 ) takes the exponential moving average of the network predictions as the consistency target . Mean Teacher ( Tarvainen & Valpola , 2017 ) , on the other hand , proposes averaging the model parametes instead of network predictions . Overall , the Mean Teacher approach significantly improves the quality of consistency targets and the empirical results on semi-supervised benchmarks . Instead of applying stochastic perturbations to the inputs , Virtual Adversarial Training or VAT ( Miyato et al. , 2018 ) computes the perturbed examples x̂ = x + radv , where radv = arg max r : ||r||2≤ DKL ( fθ ( x ) ‖fθ ( x + r ) ) . ( 1 ) That is , the VAT approach attempts to generate a perturbation which most likely causes the model to misclassify the input in an adversarial direction . Finally , the VAT approach adopts Kullback-Leibler ( KL ) divergence to compute the consistency loss . In comparison to the stochastic perturbation , the VAT approach demonstrates the greater effectiveness in the semi-supervised learning problem . 3 LLP WITH CONSISTENCY REGULARIZATION . With regards to weak supervision , the LLP scenario is similar to the semi-supervised learning problem . In the semi-supervised learning setting , only a small portion of training examples is labeled . On the other hand , in the LLP scenario , we are given the weak supervision of label proportions instead of the strong label on individual instances . Both settings are challenging since most training examples do not have individual labels . To address this challenge , semi-supervised approaches seek to exploit the unlabeled examples to further capture the latent structure of data . Motivated by these semi-supervised approaches , we combine the idea of leveraging the unlabeled data into the LLP problem . We make the same smoothness assumption and introduce a new concept incorporating consistency regularization with LLP . In particular , we consider the typical cross-entropy function between real label proportions and estimated label proportions . Given a bag B = ( X , p ) , we define the proportion loss Lprop as follows : Lprop ( θ ) = − L∑ i=1 pi log 1 |X| ∑ x∈X fθ ( x ) i. Interestingly , the proportion loss Lprop boils down to standard cross-entropy loss for fullysupervised learning when the bag size is one . To learn a decision boundary that better reflects the data manifold , we add an auxiliary consistency loss that leverages the unlabeled data . More formally , we compute the average consistency loss across all instances within the bag . Given a bag B = ( X , p ) , the consistency loss Lcons can be written as follows : Lcons ( θ ) = 1 |X| ∑ x∈X dcons ( fθ ( x ) , fθ ( x̂ ) ) , where dcons is a distance function , and x̂ is a perturbed input of x . We can use any consistencybased approach to generate the perturbed examples and compute the consistency loss . Finally , we mix the two loss functions Lprop and Lcons with a hyperparameter α > 0 , yielding the combined loss L for LLP : L ( θ ) = Lprop ( θ ) + αLcons ( θ ) , where α controls the balance between the bag-level estimation of proportion labels and instancelevel consistency regularization . To understand the intuition behind combining consistency regularization into LLP , we follow the Π-Model approach ( Laine & Aila , 2016 ) to adopt the stochastic Gaussian noise as the perturbation and to use L2 as the distance function dcons in a toy example . Figure 2 illustrates how our method is able to produce a decision boundary that passes through the low-density region and captures the data manifold . On the other hand , the vanilla approach , which simply optimizes the proportion loss , gets easily stuck at a poor solution due to the lack of label information . This toy example shows the advantage of applying consistency regularization into LLP . According to Miyato et al . ( 2018 ) , VAT is more effective and stable than Π-Model due to the way it generates the perturbed examples . For each data example , the Π-Model approach stochastically Algorithm 1 LLP-VAT algorithm Require : D = { ( Xm , pm ) } Mm=1 : collection of bags Require : fθ ( x ) : instance-level classifier with trainable parameters θ Require : g ( x ; θ ) = x + radv : VAT augmentation function according to Equation 1 Require : w ( t ) : ramp-up function for increasing the weight of consistency regularization Require : T : total number of iterations for t = 1 , . . . , T do for each bag ( X , p ) ∈ D do p̂← 1|X| ∑ x∈X fθ ( x ) . Estimated proportion label Lprop = − ∑L j=1 pi log p̂i . Proportion loss Lcons = 1|X| ∑ x∈XDKL ( fθ ( x ) ‖fθ ( g ( x ; θ ) ) ) . Consistency loss L = Lprop + w ( t ) · Lcons . Total loss update θ by gradient∇θL . e.g . SGD , Adam end for end for return θ perturbs inputs and trains the model to assign the same class distributions to all neighbors . In contrast , the VAT approach focuses on neighbors that are sensitive to the model . That is , VAT aims to generate a perturbed input whose prediction is the most different from the model prediction of its original input . The learning of VAT approach tends to be more effective in improving model generalization . Therefore , we adopt the VAT approach to compute the consistency loss for each instance in the bag . Additionally , to prevent the model from getting stuck at a local optimum in the early stage , we use the exponential ramp-up scheduling function ( Laine & Aila , 2016 ) to increase the consistency weight gradually to the maximum value α . The full algorithm of LLP with VAT ( LLP-VAT ) is described in Algorithm 1 .

Summary: of the paper:  Learning from label proportions (LLP) is an area in machine learning that tries to learn a classifier that predicts labels of instances, with only bag-level aggregated labels given at the training stage.  Instead of proposing a loss specialized for this problem, this paper proposes a regularization term for the LLP problem. The core contribution of this paper is to use the idea of consistency regularization, which has become very popular in semi-supervised learning in the recent years.  The regularization term takes a perturbation of an input sample, and then force the output of the original and perturbed sample to be similar by minimizing  a KL divergence of the two output distributions.  Experiments show the performance of the proposed method under two bag generation settings.  The paper also finds empirically that the hard L_1 has high correlation with the test error rate, which makes it an ideal candidate when the user splits the validation data from training data (meaning there are no ground truth labels for each instances).