Learning from Label Proportions with Consistency Regularization

1 INTRODUCTION . In traditional supervised learning , a classifier is trained on a dataset where each instance is associated with a class label . However , label annotation can be expensive or difficult to obtain for some applications . Take the embryo selection as an example ( Hernández-González et al. , 2018 ) . To increase the pregnancy rate , clinicians would transfer multiple embryos to a mother at the same time . However , clinicians are unable to know the outcome of a particular embryo due to limitations of current medical techniques . The only thing we know is the proportion of embryos that implant successfully . To increase the success rate of embryo implantation , clinicians aim to select high-quality embryos through the aggregated results . In this case , only label proportions about groups of instances are provided to train the classifier , a problem setting known as learning from label proportions ( LLP ) . In LLP , each group of instances is called a bag , which is associated with a proportion label of different classes . A classifier is then trained on several bags and their associated proportion labels in order to predict the class of each unseen instance . Recently , LLP has attracted much attention among researchers because its problem setting occurs in many real-life scenarios . For example , the census data and medical databases are all provided in the form of label proportion data due to privacy issues ( Patrini et al. , 2014 ; Hernández-González et al. , 2018 ) . Other LLP applications include fraud detection ( Rueping , 2010 ) , object recognition ( Kuck & de Freitas , 2012 ) , video event detection ( Lai et al. , 2014 ) , and ice-water classification ( Li & Taylor , 2015 ) . The challenge in LLP is to train models without direct instance-level label supervision . To overcome this issue , prior work seeks to estimate either the individual label ( Yu et al. , 2013 ; Dulac-Arnold et al. , 2019 ) or the mean of each class by the label proportions ( Quadrianto et al. , 2009 ; Patrini et al. , 2014 ) . However , the methodology behind developing these models do not portray LLP situations that occur in real life . First , these models can be improved by considering methods that can better leverage unlabeled data . Second , these models assume that bags of data are randomly generated , which is not the case for many applications . For example , the data of population census are collected on region , age , or occupation with varying group sizes . Third , training these models requires a validation set with labeled data . It would be more practical if the process of model selection relies only on the label proportions . This paper aims to resolve the previous problems . Our main contributions are listed as follows : • We first apply a semi-supervised learning technique , consistency regularization , to the multi-class LLP problem . Consistency regularization considers an auxiliary loss term to enforce network predictions to be consistent when its input is perturbed . By exploiting the unlabeled instances , our method captures the latent structure of data and obtains the SOTA performance on three benchmark datasets . • We develop a new bag generation algorithm – the K-means bag generation , where training data are grouped by attribute similarity . Using this setup can help train models that are more applicable to actual LLP scenarios . • We show that it is possible to select models with a validation set consisting of only bags and associated label proportions . The experiments demonstrate correlation between baglevel validation error and instance-level test error . This potentially reduces the need of a validation set with instance-level labels . 2 PRELIMINARY . 2.1 LEARNING FROM LABEL PROPORTIONS . We consider the multi-class classification problem in the LLP setting in this paper . Let xi ∈ RD be a feature vector of i-th example and yi ∈ { 1 , . . . , L } be a class label of i-th example , where L is the number of different classes . We define e ( j ) to be a standard basis vector [ 0 , . . . , 1 , . . . , 0 ] with 1 at j-th position and ∆L = { p ∈ RL+ : ∑L i pi = 1 } to be a probability simplex . In the setting of LLP , each individual label yi is hidden from the training data . On the other hand , the training data are aggregated by a bag generation procedure . We are given M bags B1 , . . . , BM , where each bag Bm contains a set Xm of instances and a proportion label pm , defined by pm = 1 |Xm| ∑ i : xi∈Xm e ( yi ) , M⋃ m=1 Xm = { x1 , . . . , xN } . We do not require each subset to be disjoint . Also , each bag may have different size . The task of LLP is to learn an individual-level classifier fθ : RD → ∆L to predict the correct label y = arg maxi fθ ( x ) i for a new instance x . Figure 1 illustrates the setting of learning from label proportions in the multi-class classification ( Dulac-Arnold et al. , 2019 ) . 2.2 PROPORTION LOSS . The feasibility of the binary LLP setting has been theoretically justified by Yu et al . ( 2014 ) . Specifically , Yu et al . ( 2014 ) propose the framework of Empirical Proportion Risk Minimization ( EPRM ) , proving that the LLP problem is PAC-learnable under the assumption that bags are i.i.d sampled from an unknown probability distribution . The EPRM framework provides a generalization bound on the expected proportion error and guarantees to learn a probably approximately correct proportion predictor when the number of bags is large enough . Furthermore , the authors prove that the instance label error can be bounded by the bag proportion error . That is , a decent bag proportion predictor guarantees a decent instance label predictor . Based on the profound theoretical analysis , a vast number of LLP approaches learn an instance-level classifier by directly minimizing the proportion loss without acquiring the individual labels . To be more precise , given a bag B = ( X , p ) , an instance-level classifier fθ and a divergence function dprop : RL × RL → R , the proportion loss penalizes the difference between the real proportion label pm and the estimated proportion label p̂ = 1 |X| ∑ x∈X fθ ( x ) , which is an average of the instance predictions within a bag . Thus , the proportion loss Lprop can be defined as follows : Lprop ( θ ) = dprop ( p , p̂ ) . The commonly used divergence functions are L1 and L2 function in prior work ( Musicant et al. , 2007 ; Yu et al. , 2013 ) . Ardehaly & Culotta ( 2017 ) and Dulac-Arnold et al . ( 2019 ) , on the other hand , consider the cross-entropy function for the multi-class LLP problem . 2.3 CONSISTENCY REGULARIZATION . Since collecting labeled data is expensive and time-consuming , the semi-supervised learning approaches aim to leverage a large amount of unlabeled data to mitigate the need for labeled data . There are many semi-supervised learning methods , such as pseudo-labeling ( Lee , 2013 ) , generative approaches ( Kingma et al. , 2014 ) , and consistency-based methods ( Laine & Aila , 2016 ; Miyato et al. , 2018 ; Tarvainen & Valpola , 2017 ) . Consistency-based approaches encourage the network to produce consistent output probabilities between unlabeled data and the perturbed examples . These methods rely on the smoothness assumption ( Chapelle et al. , 2009 ) : if two data points xi and xj are close , then so should be the corresponding output distributions yi and yj . Then , the consistency-based approaches can enforce the decision boundary to traverse through the low-density region . More precisely , given a perturbed input x̂ taken from the input x , consistency regularization penalizes the distinction of model predictions between fθ ( x ) and fθ ( x̂ ) by a distance function dcons : RL × RL → R. The consistency loss can be written as follows : Lcons ( θ ) = dcons ( fθ ( x ) , fθ ( x̂ ) ) . Modern consistency-based methods ( Laine & Aila , 2016 ; Tarvainen & Valpola , 2017 ; Miyato et al. , 2018 ; Verma et al. , 2019 ; Berthelot et al. , 2019 ) differ in how perturbed examples are generated for the unlabeled data . Laine & Aila ( 2016 ) introduce the Π-Model approach , which uses the additive Gaussian noise for perturbed examples and chooses the L2 error as the distance function . However , a drawback to Π-Model is that the consistency target fθ ( x̂ ) obtained from the stochastic network is unstable since the network changes rapidly during training . To address this problem , Temporal Ensembling ( Laine & Aila , 2016 ) takes the exponential moving average of the network predictions as the consistency target . Mean Teacher ( Tarvainen & Valpola , 2017 ) , on the other hand , proposes averaging the model parametes instead of network predictions . Overall , the Mean Teacher approach significantly improves the quality of consistency targets and the empirical results on semi-supervised benchmarks . Instead of applying stochastic perturbations to the inputs , Virtual Adversarial Training or VAT ( Miyato et al. , 2018 ) computes the perturbed examples x̂ = x + radv , where radv = arg max r : ||r||2≤ DKL ( fθ ( x ) ‖fθ ( x + r ) ) . ( 1 ) That is , the VAT approach attempts to generate a perturbation which most likely causes the model to misclassify the input in an adversarial direction . Finally , the VAT approach adopts Kullback-Leibler ( KL ) divergence to compute the consistency loss . In comparison to the stochastic perturbation , the VAT approach demonstrates the greater effectiveness in the semi-supervised learning problem . 3 LLP WITH CONSISTENCY REGULARIZATION . With regards to weak supervision , the LLP scenario is similar to the semi-supervised learning problem . In the semi-supervised learning setting , only a small portion of training examples is labeled . On the other hand , in the LLP scenario , we are given the weak supervision of label proportions instead of the strong label on individual instances . Both settings are challenging since most training examples do not have individual labels . To address this challenge , semi-supervised approaches seek to exploit the unlabeled examples to further capture the latent structure of data . Motivated by these semi-supervised approaches , we combine the idea of leveraging the unlabeled data into the LLP problem . We make the same smoothness assumption and introduce a new concept incorporating consistency regularization with LLP . In particular , we consider the typical cross-entropy function between real label proportions and estimated label proportions . Given a bag B = ( X , p ) , we define the proportion loss Lprop as follows : Lprop ( θ ) = − L∑ i=1 pi log 1 |X| ∑ x∈X fθ ( x ) i. Interestingly , the proportion loss Lprop boils down to standard cross-entropy loss for fullysupervised learning when the bag size is one . To learn a decision boundary that better reflects the data manifold , we add an auxiliary consistency loss that leverages the unlabeled data . More formally , we compute the average consistency loss across all instances within the bag . Given a bag B = ( X , p ) , the consistency loss Lcons can be written as follows : Lcons ( θ ) = 1 |X| ∑ x∈X dcons ( fθ ( x ) , fθ ( x̂ ) ) , where dcons is a distance function , and x̂ is a perturbed input of x . We can use any consistencybased approach to generate the perturbed examples and compute the consistency loss . Finally , we mix the two loss functions Lprop and Lcons with a hyperparameter α > 0 , yielding the combined loss L for LLP : L ( θ ) = Lprop ( θ ) + αLcons ( θ ) , where α controls the balance between the bag-level estimation of proportion labels and instancelevel consistency regularization . To understand the intuition behind combining consistency regularization into LLP , we follow the Π-Model approach ( Laine & Aila , 2016 ) to adopt the stochastic Gaussian noise as the perturbation and to use L2 as the distance function dcons in a toy example . Figure 2 illustrates how our method is able to produce a decision boundary that passes through the low-density region and captures the data manifold . On the other hand , the vanilla approach , which simply optimizes the proportion loss , gets easily stuck at a poor solution due to the lack of label information . This toy example shows the advantage of applying consistency regularization into LLP . According to Miyato et al . ( 2018 ) , VAT is more effective and stable than Π-Model due to the way it generates the perturbed examples . For each data example , the Π-Model approach stochastically Algorithm 1 LLP-VAT algorithm Require : D = { ( Xm , pm ) } Mm=1 : collection of bags Require : fθ ( x ) : instance-level classifier with trainable parameters θ Require : g ( x ; θ ) = x + radv : VAT augmentation function according to Equation 1 Require : w ( t ) : ramp-up function for increasing the weight of consistency regularization Require : T : total number of iterations for t = 1 , . . . , T do for each bag ( X , p ) ∈ D do p̂← 1|X| ∑ x∈X fθ ( x ) . Estimated proportion label Lprop = − ∑L j=1 pi log p̂i . Proportion loss Lcons = 1|X| ∑ x∈XDKL ( fθ ( x ) ‖fθ ( g ( x ; θ ) ) ) . Consistency loss L = Lprop + w ( t ) · Lcons . Total loss update θ by gradient∇θL . e.g . SGD , Adam end for end for return θ perturbs inputs and trains the model to assign the same class distributions to all neighbors . In contrast , the VAT approach focuses on neighbors that are sensitive to the model . That is , VAT aims to generate a perturbed input whose prediction is the most different from the model prediction of its original input . The learning of VAT approach tends to be more effective in improving model generalization . Therefore , we adopt the VAT approach to compute the consistency loss for each instance in the bag . Additionally , to prevent the model from getting stuck at a local optimum in the early stage , we use the exponential ramp-up scheduling function ( Laine & Aila , 2016 ) to increase the consistency weight gradually to the maximum value α . The full algorithm of LLP with VAT ( LLP-VAT ) is described in Algorithm 1 .

Summary: This paper proposes using Consistency Regularization and a new bag generation technique to better learn classification decision boundaries in a Label Proportion setting.  The consistency regularization works to make sure that examples in the local neighbourhood have similar outputs. The authors further use K-means clustering to create a new bagging scenario they use to mimic real-world LLP settings.