Learning from Label Proportions with Consistency Regularization
Authors: Kuen-Han Tsai
Hsuan-Tien Lin
Venue:  (2020)
Keywords: learning from label proportions
consistency regularization
semi-supervised learning

Abstract: The problem of learning from label proportions (LLP) involves training classifiers with weak labels on bags of instances, rather than strong labels on individual instances. The weak labels only contain the label proportions of each bag. The LLP problem is important for many practical applications that only allow label proportions to be collected because of data privacy or annotation costs, and has recently received lots of research attention. Most existing works focus on extending supervised learning models to solve the LLP problem, but the weak learning nature makes it hard to further improve LLP performance with a supervised angle. In this paper, we take a different angle from semi-supervised learning.
In particular, we propose a novel model inspired by consistency regularization, a popular concept in semi-supervised learning that encourages the model to produce a decision boundary that better describes the data manifold. With the introduction of consistency regularization, we further extend our study to non-uniform bag-generation and validation-based parameter-selection procedures that better match practical needs. Experiments not only justify that LLP with consistency regularization achieves superior performance, but also demonstrate the practical usability of the proposed procedures.

Under review as a conference paper at ICLR 2020
LEARNING FROM LABEL PROPORTIONS WITH CONSISTENCY REGULARIZATION
Anonymous authors Paper under double-bling review
ABSTRACT
The problem of learning from label proportions (LLP) involves training classifiers with weak labels on bags of instances, rather than strong labels on individual in- stances. The weak labels only contain the label proportion of each bag. The LLP problem important for many practical applications that only allow label propor- tions to be collected because of data privacy or annotation cost, and has recently received lots of research attention. Most existing works focus on extending su- pervised learning models solve the LLP problem, but the weak learning nature makes it hard to further improve LLP performance with a supervised angle. this paper, take different angle from semi-supervised learning. In particu- lar, we propose a novel model inspired by consistency regularization, a popular concept in semi-supervised learning that encourages the model produce a de- cision boundary that better describes the data manifold. With the introduction of consistency regularization, we further extend our study to non-uniform bag- generation and validation-baan based parameter-selection procedures that better match practical needs. Experiments not only justify that LLP with consistency regular- ization achieves superior performance, but also demonstrate the practical usability of the proposed procedures.
INTRODUCTION
In traditional supervised learning, a classifier trained on a dataset where each instance associated with class label. However, label annotation can be expensive or difficult to obtain for some appli- cations. Take the embryo selection as an example [Hernández-Gionzllee et al. 2018). To increase the pregnancy rate, clinicians would transfer multiple embryos to mother at the same time. How- ever, clinicians are unable to know the outcome of a particular embryo due to limitations of current medical techniques. The only thing we know is the proportion of embryos that implant successfully. To increase the success rate of embryo implantation, clinicians aim to select high-quality embryos through the aggregated results. In this case, only label proportions about groups of instances are provided to train the classifier, a problem setting known as learning from label proportions (LLP).
In LLP, each group of instances is called bag, which is associated with a proportion label of different classes. A classifier is then trained on several bags and their associated proportion labels in order to predict the class of each unseen instance. Recently, LLP has attracted much attention among researchers because its problem setting occurs in many real-life scenarios. For example, the census data and medical databases are all provided in the form of label proportion data due to privacy issues (Patrini et al. 2014 Henzndezz-Gozzález et al. 2018). Other LLP applications include fraud
detection (Rueping 2010) object recognition (Kuck& & de Freitas 2012), video event detection (Lai
et al. 2014). and ice- water classification (Li & Taylor] 2015).
The challenge in L.P is to train models without direct instance-level label supervision. overcome this issue, prior work seeks to estimate either the individual label (Yu et al. (0033) Dulac-Aroo
ct al. 2019) or the mean each class by the label proportions (Quadrianto et al. 2009) Patrini et al.) 2014). However, the methodology behind developing these models do not portray LLP situations that occur in real life. First, these models can be improved by considering methods that can better leverage unlabeled data. Second, these models assume that bags of data randomly generated, which is not the case for many applications. For example, the data of population census are collected region, age, or occupation with varying group sizes. Third, training these models requires

---

Under review as a conference paper at ICLR 2020
Training Data
Class2
Bag1
Bag2
Class2
Class1
Bag Generation
Learning
Class
Label Proportions
Classs
50%
Class3
40%
20%
25%
25%
Class
Figure 1: An illustration of multi-class learning from label proportions. Before training, the data grouped according to a bag generation procedure. During the training stage, we are given bags of unlabeled data and their corresponding proportion labels. The goal of L.PP is to learn an individual- level classifier.
validation set with labeled data. It would be more practical if the process of model selection relies only on the label proportions.
This paper aims to resolve the previous problems. Our main contributions are listed as follows:
We first apply a semi-supervised learning technique, consistency regularization, to the multi-class LLP problem. Consistency regularization considers an auxiliary loss term to enforce network predictions to be consistent when its input is perturbed. By exploiting the unlabeled instances, our method captures the latent structure of data and obtains the SOTA performance on three benchmark datasets.
We develop new bag generation algorithm the K-means bag generation, where training data are grouped by attribute similarity. Using this setup can help train models that are more applicable actual LLP scenarios.
We show that it is possible select models with a validation set consisting of only bags and associated label proportions. The experiments demonstrate correlation between bag- level validation eиrоr and instance-level test error. This potentially reduces the need of validation set with instance-level labels.
PRELIMINARY
2.1 LEARNING FROM LABEL PROPORTIONS
We consider the multi-class classification problem in the LLP setting in this paper. Let 2; E RD be a feature vector of i-th example and yi E {l..... .......,) L} be a class label i-th example, where L is the number different classes. We define e(i) to be a standard basis vector [o,..., 1,.., with at j-th position and A = (p E RL E LLP, each individual label yi is hidden from the training data. On the other hand, the training data are aggregated by a bag generation procedure. We are given M bags BI...., Ba, where each bag B,, contains set Xmm of instances and a proportion label Pm. defined by
1} to be a probability simplex. In the setting of
M
elvc)
X xm = (x....... ZN m=1
Pm
We not require each subset to be disjoint. Also, each bag may have different size. The task of LLP is to learn an individual-level classifier So RDD A.. to predict the correct label y = arg max; fe(x), for a new instance æ. Figure I illustrates the setting of learning from label proportions in the multi-class classification (Dulac-Amold al. 2019).
2.2 PROPORTION LOSS
The feasibility of the binary LLP setting has been theoretically justified by Yu et al. (0114) Specif- ically, Yu al. (2014) propose the framework of Empirical Proportion Risk Minimization (EPRM),
2

---

Under review as a conference paper at ICLR 2020
proving that the LLP problem is PAC-leamable under the assumption that bags are i.id sampled from an unknown probability distribution. The EPRM framework provides generalization bound on the expected proportion error and guarantees to learn probably approximately correct propor- tion predictor when the number of bags is large enough. Furthermore, the authors prove that the instance label error be bounded by the bag proportion error. That is, decent bag proportion predictor guarantees a decent instance label predictor.
Based on the profound theoretical analysis, a vast number of LLP approaches learn an instance-level classifier by directly minimizing the proportion loss without acquiring the individual labels. To be more precise, given bag B (X. p), an instance-level classifier fo and a divergence function dprop R x R² + R, the proportion loss penalizes the difference between the real proportion
label pm and the estimated proportion label P
fo())), which is an average of the instance ΦEX
Σ
X
predictions within bag. Thus, the proportion loss prop can be defined follows:
Cprop (0) = dprap (p.)) p).
The commonly used divergence functions are L and L function in prior work (Musicant al., 2007 Yu et al. 2013). AArdhaaly & Culotta (2017) and Dulac- et al. (2019), on the other hand, consider the cross-entropy function for the multi-class LLP problem.
2.3 CONSISTENCY REGULARIZATION
Since collecting labeled data is expensive and time-consuming. the semi-supervised learning ap- proaches aim to leverage a large amount of unlabeled data to mitigate the need for labeled data. There are many semi-supervised learning methods, such pseudo-labeling (Lee) 2013). genera- tive approaches (Kingma et al.) 2014), and consistency-based methods (Laine & Aila 2016 Miy- ato et al.] 2018 Tarvainen & Valpola 2017 Nonsistency-basee approaches encourage the net- work to produce consistent output probabilities between unlabeled data and the perturbed examples. These methods rely on the smoothness assumption (Chapelle al. 2009): if data points I; and Tú are close, then so should be the corresponding output distributions consistency-badt based approaches can enforce the decision boundary to traverse through the low-density region. More precisely, given a perturbed input & taken from the input æ, consistency regulariza- tion penalizes the distinction of model predictions between fo(æ) and fa() by a distance function deons: IR_ x IR - IR. The consistency loss can be written as follows:
and yj- Then, the
Lcous (0) deons (fo fo(t)).
Modern consistency- based methods (Laine & Aila 2016 Tarvainen & Valpola 2017 Miyato et al., 2018 Verma et al. 2019 Berthelot et al.) 2019) difter in how perturbed examples generated for the unlabeled data. Laine & Aila (2016) introduce the II-Model approach, which uses the additive Gaussian noise for perturbed examples and chooses the L error as the distance function. However, a drawback to I1-Model is that the consistency target fa( obtained from the stochastic network unstable since the network changes rapidly during training. To address this problem, Temporal Ensembling (Laine & Aila 2016) takes the exponential moving average of the network predictions as the consistency target. Mean Teacher (Tarvainen Valpolal 2017), on the other hand, proposes averaging the model parametes instead of network predictions. Overall, the Mean Teacher approach significantly improves the quality of consistency targets and the empirical results on semi cemi-supervised benchmarks.
Instead of applying stochastic perturbations to the inputs, Virtual Adversarial Training or VAT Miy- ato et al. 2018) computes the perturbed examples & + Tadv. where
T Tadv = arg max DKL (f +r)
r:lln|lase
That is, the VAT approach attempts to generate a perturbation which most likely causes the model to misclassify the input in adversarial direction. Finally, the VAT approach adopts Kullback-Leible (KL) divergence to compute the consistency loss. In comparison to the stochastic perturbation, the VAT approach demonstrates the greater effectiveness in the semi-supervised learning problem.
3

---

Under review as a conference paper at ICLR 2020
Foxh 193
Faoch Sce
Feedh ICEO
In this toy example, we generate 5 bags, each of which contains 20 data points uniformly sampled from the "two moons" dataset without replacement. The vanilla approach, which simply optimizes the proportion loss, suffers from poor performance as the label information is insufficient. In contrast, the "two moons" can be effectively separated into two clusters by LLP with consis- tency regularization. Our method enforces the network to produce consistent outputs for perturbed examples, and thus help capture the underlying structure of the data.
Figure 2:
LLP WITH CONSISTENCY REGULARIZATION
With regards to weak supervision, the L.P scenario is similar the semi-supervised learning prob- lem. In the semi-supervised learning setting, only small portion of training examples is labeled. On the other hand, in the LLP scenario, we are given the weak supervision of label proportions instead of the strong label on individual instances. Both settings are challenging since most training examples do have individual labels. To address this challenge, coni-supervised approaches seek to exploit the unlabeled examples to further capture the latent structure of data.
Motivated by these semi-supervised approaches, we combine the idea of leveraging the unlabeled data into the LLP problem. We make the same smoothness assumption and introduce a new concept incorporating consistency regularization with L.P. In particular, we consider the typical cross-entropy function between real label proportions and estimated label proportions. Given a bag B = (X, p). we define the proportion loss Lprop as follows:
ppop (0)
Σ Pe log Eo(x)).
Interestingly, the proportion loss Lprop boils down to standard cross-entropy loss for fully- supervised learning when the bag size is one. To learn a decision boundary that better reflects the data manifold, we add an auxiliary consistency loss that leverages the unlabeled data. More formally, we compute the average consistency loss across all instances within the bag. Given a bag B = (X, p), the consistency loss Lcons can be written as follows:
Loons (0) =
Σ TEX
deons (fe(
where deans is a distance function, and is perturbed input of æ. We can use any consistency- based approach to generate the perturbed examples and compute the consistency loss. Finally, we mix the two loss functions Eprop and Loons with a hyperparameter a 0, yielding the combined loss L for LLP:
(()) = prop (0) CoLosns (0), where controls the balance between the bag-level estimation of proportion labels and instance- level consistency regularization.
To understand the intuition behind combining consistency regularization into LLP, we follow the TI-Model approach (Laine Aila 2016) to adopt the stochastic Gaussian noise as the perturbation and to use 2 the distance function deons in toy example. Figure illustrates how method able to produce a decision boundary that passes through the low-density region and captures the data manifold. On the other hand, the vanilla approach, which simply optimizes the proportion loss, gets easily stuck at a poor solution due to the lack label information. This toy example shows the advantage of applying consistency regularization into LLP.
According to Miyato et al. (2018), VAT is more effective and stable than II-Model due to the way it generates the perturbed examples. For each data example, the II-Model approach stochastically
4

---

Under review as a conference paper at ICLR 2020
Algorithm LLP-VAT algorithm
Require: = {(xm: Pmm) Pmm) collection of bags
Require: [ar): instance-level classifier with trainable parameters 0
| Require: w(t): ramp-up function for increasing the weight of consistency regularization
Require: g( 0) x + Tadv: VAT augmentation function according to Equation
Require: T: total number of iterations do
for t 1,..., T...,,To
for each bag (x, p) E D do p X 2 fo(x)
Estimated proportion label
prop Σ P. lo
> Proportion loss
Lcons
[ EeEx DRL (ea((f())](((((((())
D Consistency loss
L= = w(() fecoss
Total loss
update by gradient VoL
> e.g. SGD, Adam
end for end for return 0
perturbs inputs and trains the model to assign the same class distributions to all neighbors. In contrast, the VAT approach focuses neighbors that are sensitive to the model. That is. VAT aims to generate a perturbed input whose prediction is the most different from the model prediction of its original input. The learning of VAT approach tends to be more effective in improving model generalization. Therefore, we adopt the VAT approach to compute the consistency loss for each instance in the bag. Additionally, prevent the model from getting stuck a local optimum in the early stage, we use the exponential ramp-up scheduling function (Laine & Aila, (016) to increase the consistency weight gradually to the maximum value The full algorithm of L.P with VAT (LLPPVAT) is described in Algorithm T
4 EXPERIMENTS
We evaluate our LLP- VAT on three benchmarks, including SVHN CIFAR1O, and CIFAR100. For model selection, we choose hyperparameters using a validation set without individual labels. Lastly, we report the test instance accuracy averaged over the last 10 epochs. The full experiment details provided in the supplementary material.
4.1 UNIFORM BAG GENERATION
For convenience, most LLP works validate their proposed methods with the uniform bag generation where the training data are randomly partitioned into bags of the same size. We evaluate our method using this bag generation procedure with the bag size incomplete bag if the number of training data is indivisible by the bag size. Table shows the experimental results for the LLP scenario with uniform bag generation.
{16, 32, 64, 128, 256). We drop the last
In comparison to the vanilla approach, our LLP-VAT significantly improves the performance on CIFAR10 and CIFAR100. This indicates that applying consistency regularization into LLP does help learn a better classifier. As for SVHN, since the test accuracy close to the fully-supervised performance when the bag size is small, there is no clear difference among three methods. In addition, the results also show that the performance of ROT is unstable and lead us to conclude that the unhelpful pseudo-labels would easily result in worse classifier. Conversely, our LLP-VAT is more stable and obtains better test accuracy in most cases.
4.2 K-MEANS BAG GENERATION
In this section, we further investigate our LLP- VAT in a more practical scenario. We observe that the uniform bag generation barely fits the real-world LLP situation because of following two reasons. First, the real-life data are usually grouped by attribute similarity instead of uniformly sampled. Sec- ond, each bag may have different bag sizes, i.e., the distribution of bag sizes is diverse. Consider the
5

---

Under review as a conference paper at ICLR 2020
Table 1:
Test accuracy with the uniform bag generation. The performance of the vanilla approach with bag size of one corresponds to the fully-supervised setting.
Bag Size
Dataset
Method
16
32
64
128
256
SVHN
vanilla
95.52
95.28
95,20
94.41
88.93
12.64
ROT
95.35
94.84
93.74
92.29
13.14
LAP-VAT
95.66
95.73
94.60
91.24
11.18
CIFARIO
vanilla ROT 90.64
88.77
85,02
70.68
47,48
38.69
86.97
77.01
62.93
48,95
40.16
LLP-VAT
89.30
85,41
72.49
50.78
41.62
CIFARIO0
vanilla
59.79
58.58
48.09
20.66
5.82
2.82
ROT
54.16
47.75
29.38
7.95
2.63
ILP-VAT
59.47
48.98
22.84
9.40
3.29
CIARRO
CEPAR100
500
200
3029
of ciliteters
Figure 3:
The distribution of bag sizes from the K-means procedure on three benchmarks. When the number of clusters increases, the distribution of bag sizes becomes various.
US presidential election results (Sun al. 2017). where the statistics of voting results are collected by geological regions (e.g., states). Also, each state have varying number of voters. Therefore, we introduce a new bag generation procedure the K-means bag generation, where we cluster examples into bags by the K-means algorithm. Although those bags generated from the K-means bag genera- tion are dependent on each other, violating the i.i.d. assumption, this setting both challenging and worth-studying.
Since we perform experiments on image datasets, it is meaningless to cluster data examples based on RGB pixels. We first adopt the principle component analysis algorithm, which is an unsupervised dimension reduction technique, to project the data into a now-dimensional representation space. This space may capture more important patterns in images. Then we group the now-dimensional repre- sentations the images following the K-means bag generation procedure. We conduct experiments with the number of clusters K E (31 20, 1560, 780, 390, 195) CIFAR10 and CIFARI 100, and K {4576, 2288, 1 1144, 572, 286} on SVHN. These numbers are selected to match the number of proportion labels the uniform bag generation procedure. The distribution of bag sizes generated from the K-means procedure are shown in Figure
For experiments, we do compare our proposed method the ROT loss, which needs estimate individual labels iteratively for each bag. The procedure of the ROT algorithm is time-consuming and cannot be accelerated bags are of varying sizes. Besides, for the K-means bag generation, there may be some large bags when the value of K is small. Because of the limited computational resource, we take subsample each bag if the bag size larger than the threshold of 256. Partic- ularly, when large bag is sampled, we randomly sample 256 instances and assign the original label proportions to the reduced bag.
The experimental results of the K-means bag generation are shown in Table and Table 3 Although this scenario violates the i..d. assumption, the results demonstrate that it is feasible to learn an instance-leve classifier by simply minimizing the proportion loss. Also, our LLP-VAT significantly brings benefits for the k-means bag generation scenario on SVHN and CIFAR1O, while showing comparable performance CIFAR100 Interestingly, the performance of model is not well-
and

---

Under review as a conference paper at ICLR 2020
Table 2: Test accuracy with the K-means bag generation on SVHN.
K
Dataset
Method
4576
2288
1144
572
286
SVHN
vanilla LLP-VAT
92.07
91.16
92.00
78.70
47.16
93.11
91.69
93.21
82.05
46.38
Table 3: Test accuracy with the K-means bag generation CIFAR10 and CIFAR100.
Dataset
Method
3120
1560
780
390
195
CIFARIO
vanilla
73.93
66.54
44.12
49.85
39.86
LLP-VAT
77.43
68.01
51.04
50.22
38.27
CIFAR100
vanilla
38.65
22.16
16.07
15,47
7.82
LLP-VAT
37.98
21.90
15.61
15.31
8.13
correlated with the value of K. One possible reason that we might drop informative bags as we randomly split bags into validation and training.
4.3 VALIDATION METRICS
Many modern machine learning models require a wide range of hyperparameter selections about the architecture, optimizer and regularization. However, for the realistic LLP scenario, have no access to labeled instances during training. It is crucial to choose appropriate hyperparameters based on the bag-level validation error that is computed with only proportion labels. To evaluate the performance at the bag level, we consider four validation metrics: soft L error, hard L error, soft KL divergence, and hard KL divergence. Their definitions are given as follows. First, we define the output probabilities of instance the soft prediction and its one-hot encoding the hard prediction. For each bag. we then compute the estimated label proportions by averaging these soft or hard predictions. Finally, use the L error KL divergence to measure the bag-level prediction error.
To investigate the relationship between the instance-level test error and the bag-level validation ror, we compute the Pearson correlation coefficient between them on models trained for 400 epochs. The results are shown in Table 4 Surprisingly, we find that the hard L error has strong positive correlation to test error rate on all benchmarks. This implies that it is feasible to select hyperparam- eters with only label proportions realistic LLP scenarios. Interestingly, our finding is coherent to Yu et al. (2013). Although their and our works both adopt the hard L eтror for model selection, we focus on the multi-class LLP scenario instead of the binary classification problem they considered. Therefore, we suggest future multi-class LLP works could adopt the hard I validation metric for model selection
RELATED WORK
Kuck Freitas (0122) first introduce the LLP scenario and formulate the probabilistic model with the MCMC algorithm to generate consistent label proportions. Several following works (Chen ct al. 2006 Musicant et al. 2007) extend the LLP setting to a variety of standard supervised learning algorithms. Without directly inferring instance labels, Quaddianto al. (2009) propose Mean Map algorithm with exponential-family parametric models. The algorithm uses empirical mean operators of each bag to solve convex optimization problem. However, the success of the Mean Map algorithm is based on a strong assumption that the class-conditional distribution of data is
Nevertheless, we do not suggest using our validation metric for early stopping since the correlation is computed after the model converges.

---

Under review as a conference paper at ICLR 2020
Table The Pearson correlation coefficient between the test error rate and the following validation metrics on benchmarks.
Uniform
K-means
SVHN
CIFARIO
CIFAR100
SVHN
CIFARIO
CIFAR100
Hard L,²
0.97
0.81
0.81
0.99
0.75
0.67
Soft I,'
0.83
0.33
-0.50
0.90
0.61
0.26
Hard KL
0.69
-0.18
0.64
0.81
0.10
0.40
Soft KL
0.69
0.89
-0.16
0.71
0.62
0.57
independent of bags. To loosen the restriction, Patrini al. (2014) propose Laplacian Mean Map algorithm imposing an additional Laplacian regularization. Nevertheless, these Mean Map algorithms suffer from a fundamental drawback: they require the classifier to be linear model.
Several works tackle the LLP problem from Bayesian perspectives. For example, Fan et al. (2014) propose RBM-based generative model to estimate the group-conditional likelihood of data. Hernández-Goozáález et al. (2013). on the other hand, develop a Bayesian classifier with an EM algorithm. Recently, Sun al. (2017) propose graphical model using counting potential to pre- dict instance labels for the US presidential election. Furthermore, other works (Chen et al.) 2009 Stolpe & Morik 2011 I adopt k-means approach cluster training data by label proportions. While some works (Fan et al. 20144) Sun et al. (017)) claim that they are suitable for large-scale settings, algorithms are rather inefficient and computationally expensive when applied to large image datasets.
both Bayesian methods and clustering-be
Another line of work adopts a large-margin framework for the problem of LLP. Stolpe & Morik (2011) propose variant of support vector regression using the inverse calibration method to es- timate the class-conditional probability for bags. On the other hand, Yu et al.] (2013) propose a procedure that alternates between assigning a label to each instance, also known pseudo-labeling in the literature, and fitting an SVM classifier. Motivated by this idea, a number of works (Wang et al. 2015; Qi al. 2016 Chen et al., 2017) infer individual labels and updated model parame- ters alternately. One major drawback of SVM-based approaches is that they are tailored for binary classification; they cannot extend to the multi-class classification setting efficiently.
As deep learning has garnered huge success in a number of areas, such as natural language process- ing, speech recognition, and computer vision, many works leverage the power of neural networks for the LLP problem. [Ardehaly & Culotta 2017) are the first to apply deep models to the multi- class LLP setting. Also, [Bortsova et al. (2018) propose deep LLP method learning the extent of emphysema from the proportions of disease tissues. Concurrent to our work, Dulac-Arold et al. (2019) also considers the multi-class LLP setting with bag-level cross-entropy loss. They introduce a ROT loss that combines two goals: jointly maximizing the probability of instance predictions and minimizing the bag proportion loss.
6 CONCLUSION
In this paper, we first apply a novel semi-supervised learning technique, consistency regularization, to the multi-class LLP problem. Our proposed approach leverages the unlabeled data learn deci- sion boundary that better depicts the data manifold. The empirical results validate that our approach obtains better performance than that achieved by existing LLP works. Furthermore, we introduce non-uniform bag scenario the K-means bag generation, where training instances are clustered by attribute relationships. This setting simulates more practical LLP situations than the uniform bag generation setting, which is often used in previous works. Lastly, we introduce a bag-level valida- tion metrics, hard L error, for model selection with only label proportions. We empirically show that the legelevel hard L error has strong correlation to the test classification error. For real-world applicability, we suggest that multi-class LLP methods relying on paramete tuning could evaluate their methodology based on the bag-level hard L' error. One interesting future direction combining the Mixup. In a nutshell, we hope that future LLP work can further explore the ideas presented in this paper.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Ehsan Mohammady Ardehaly and Aron Culotta. Co-training for demographic classification using deep learning from label proportions. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 1017-1024. IEEE, 2017.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv preprint
arXiv: 19055:9905.0000 /90.00299 02249,229999 2019.
Gerda Bortsova. Florian Dubost, Silas Ørting, Ioamnis Katramados, Laurens Hogeweg, Laura Thom- sen, Mathilde Wille, and Marleen de Bruijne. Deep learning from label proportions for emphy- sema quantification. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 768-776. Springer, 2018.
Olivier Chapelle. Bernhard Scholkopf. and Alexander Zien. Semi-supervised learning (chapelle, et al., eds.; [book reviews]. IEEE Transactions on Neural Networks, 20(3):542-542, 2009.
Bee-Chung Chen, Lei Chen, Raghu Ramakrishnan, and David R Musicant. Learning from aggregate views. In 22nd International Conference on Data Engineering (ICDE'06) pp. 3-3. IEEE, 2006.
Shuo Chen, Bin Liu, Mingjie Qian, and Changshui Zhang. Kernel k-means based framework for aggregate outputs classification. In 2009 IEEE International Conference Data Mining Work- shops, pp.
IEEE, 2009.
Zhensong Chen, Zhiquan Qi, Bo Wang, Limeng Cui, Fan Meng, and Yong Shi. Learning with label proportions based on nonparallel support vector machines. Knowledge-Bore Systems, 119: 141-141, 2017.
Gabriel Dulac-Amold, Neil Zeghidour, Marco Cuturi, Lucas Beyer, and Jean-Philippe Vert. Deep multi-class learning from label proportions. arXiv preprint arXiv: 1905, 12909, 2019.
Kai Fan, Hongyi Zhang, Songbai Yan, Liwei Wang, Wensheng Zhang, and Jufu Feng. Learning a generative classifier from label proportions. Neurocomputing, 139:47-55, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016.
Jerónimo Fermández-Gonzille Iñaki Inza, and Jose A Lozano. Learning bayesian network classi- fiers from label proportions. Pattern Recognition, 46(12):3425-344 3440, 2013.
Jerónimo Hernández-Gonzáäee Inaki Inza, Lorena Crisol-Ortíz, María A Guembe, María Iñarra, and Jose A Lozano. Fitting the data from embryo implantation prediction: Learning from label proportions. Statistical methods in medical research. 27(4):0056-166 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988, 2014.
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling, Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589, 2014.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Tech- nical report, Citeseer, 2009.
Hendrik Kuck and Nando de Freitas. Learning about individuals from group statistics. arXiv preprint arXiv:1207.1395 2012.
Kuan-Ting Lai, Felix Yu, Ming-Syan Chen, and Shih-Fu Chang. Video event detection by ferring temporal instance labels. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 2253-2250, 2014.
Samuli Laine and Timo Aila. Temporal ensembling for supiviupervised learning. arXiv preprint arXiv:1610.0224 02242, 2016.

---

Under review as a conference paper at ICLR 2020
Dong- Hygu Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning. ICML, volume 3, pp- 2, 2013.
Fan Li and Graham Taylor. Alter-enn: An approach to learning from label proportions with appli- cation ice-water classification. In Neural Information Processing Systems Workshops (NIPSW) on Learning and privacy with incomplete data and weak supervision, 2015.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv: 2013.
Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 2018.
David R Musicant, Janara M Christensen, and Jamie F Olson. Supervised learning by training on aggregate outputs. In Seventh IEEE International Conference on Data Mining (ICDM 2007), pp- 252-261. IEEE, 2007.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. Advances in neural information pro- cessing systems, 01 2011.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realis- tic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems, pp. 3235-3334444 2018.
Giorgio Patrini, Richard Nock, Paul Rivera, and Tiberio Caetano. (almost) no label no cry. In Advances in Neural Information Processing Systems, pp. 190-198, 2014.
Zhiquan Qi, Bo Wang, Fan Meng, and Lingfeng Niu. Learning with label proportions via npsvmn
IEEE transactions on cybernetics, 47110):3233-3355 2016.
Novi Quadrianto, Alex J Smola, Tiberio S Caetano, and Quoe V Le. Estimating labels from label proportions. Journal of Machine Learning Research, 10(Oct):2349 227744 2009.
Stefan Rueping. Svm classifier estimation from group probabilities. In Proceedings of the 27th international conference on machine learning (ICMI-10), pp. 911-918, 2010.
Marco Stolpe and Katharina Morik. Learning from label proportions by optimizing cluster model selection. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 349-364. Springer, 2011.
Tao Sun, Dan Sheldon, and Brendan OConnor. A probabilistic approach for learning with label proportions applied to the us presidential election. In 2017 IEEE International Conference on Data Mining (ICDM), pp. 445 454. IEEE, 2017.
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weiggt-averaged consis- tency targets improve semi-supervised deep learning results. In Advances in neural information processing systems, pp. 1195-1204, 2017.
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. Interpolation con- sistency training for semi-supervised learning. arXiv preprint 1903:0090333822 2019.
Bo Wang. Zhensong Chen, and Zhiquan Qi. Linear twin svm for learning from lahel proportions. In 2015 JEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Tech- nology (WI-IAT), volume 3, pp. 56-59. IEEE, 2015.
Felix X Yu, Dong Liu, Sanjiv Kumar, Tony Jebara, and Shih-Fu Chang. ossmm for learning with label proportions. arXiv preprint arXiv:1306.0880 0886, 2013.
Felix X Yu, Krzysztof Choromanski, Sanjiv Kumar, Tony Jebara, and Shih-Fu Chang. On learning from label proportions. arXiv preprint arXiv: /002.599, 2014.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. Procedings of the British Ma-
chine Vision Conference 2016, 2016.
10

---

Under review as a conference paper at ICLR 2020
A EXPERIMENT DETAILS
A.1 DATASETS
To evaluate the effectiveness of our proposed method, we conduct experiments on three benchmark datasets, including SVHN (Netzer et al. 2011), CIFARIO, and CIFAR100 (Krizhevsky & Hinton 2009). The SVHN dataset consists of 32x32 RGB digit images with 73,257 examples for training, 26,032 examples for testing, and 531,131 extra training examples that are not used in our experi- ments. The CIFARIO and CIFAR 100 datasets both consist 50,000 training examples and 10,000 test examples. Each example is 32x32 colored natural image, drawn from 10 classes and 100 classes respectively.
A.2 EXPERIMENT SETUP
Implementation details. For all experiments in this section, we adopt the Wide Residual Network with depth 28 and width 2 (WRN-28-2) following the standard specification in the paper (Zagoruyko Komodakis 2016). We use the Adam optimizer (Kingma & Bal 2014) with a learning rate of 0.0003. Additionall,, we train models for a maximum of 400 epochs with a scheduler that scales the learning rate by 0.2 once the model finishes 320 epochs. To simulate the LLP setting, we split the training data by two bag generation algorithms described in Section 4.1 and 4.2 Once completing the bag generation, we then compute the proportion labels by averaging the class labels over each bag. To avoid over-fitting, we follow the common practice of data augmentation (He al.
2016 Lin
(et al. (201)) padding an image by pixels on each side, taking a random 32x32 crop and randomly flipping the image horizontally with the probability of 0.5 for all benchmarks.
Hyperparameters. We compare our method, LLP-VAT, to ROT (Dulac-Arnold et al. 2019) and the vanilla approach, which simply minimizes the proportion loss. For ROT, conduct experiments with a hyperparameter of a E {0.1, 0.4, 0.7, 0.9} to compute the ROT loss. Following Oliver et al. (0188. we adopt the VAT approach generate perturbed examples with a perturbation weight of and 6 for SVHN and CIFAR10 (or CIFAR 100) respectively. We measure the consistency loss with the KL divergence and consistency weight of {0.5, 0.1, 0.05, 0.01}.
Model selection. For a fair comparison, we randomly sample 90% of bags for training and reserve the rest for validation. In the LLP setting, since there are no individual labels available in the validation set, we select hyperparameters based on the hurd L error which is computed with only proportion labels. To be more specific, the hard L error for a bag = (X, p) is defined by
Err =11p- Pl1,,
P Σ (1))
where 8" = arg max; fo(æ); and eli)) is the one-hot encoding of the prediction. Lastly, we report the test instance accuracy averaged over the last 10 epochs.

---

Under review as a conference paper at ICLR 2020
B CONVERGENCE ANALYSIS OF LLP-VAT
To analyze the convergence performance of LLP-VAT, we plot the instance accuracy on the test over training epochs. Figure 4 and show the accuracy curve on the test set with the uniform bag generation and the K-means bag generation respectively. As shown in Figure and [S] the experimental results demonstrate the stability of our LLP-VAN When the training epoch gradually increases, the test instance accuracy goes up quickly and converges in the end.
B.1 UNIFORM BAG GENERATION
250
(a) SVHN
(b) CIFARIO
(c) CIFARI00
Figure 4: Evolution of the test accuracy on benchmarks with the uniform bag generation of varying bag sizes.
B.2 K-MEANS BAG GENERATION
Mhr /// / Man
200
(a) SVHN
(b) CIFARIO
(c) CIFAR100
Figure 5: Evolution of the test accuracy benchmarks with the K-means bag generation of varying number clusters.
12


Reviews:
[{"review_id": "rJx0ZWZNqr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\nThe paper presents a solution to the problem of learning from label proportion (LLP) by incorporating regularization by data generation motivated by semi-supervised learning.   \nIt is argued that, by consistency assumption, the classification function defined on the data manifold should be a locally consistent mapping (in the sense of bag label proportion) such that the discrepancy between two bags of data points within a small neighborhood should be constraint. This is further combined with the conventional LLP loss (cross entropy between bag label proportion and classification results) to produce the final loss function for learning. For bag data generation, image attributes are leveraged to group similar data points in the feature space. Evaluation is performed on three benchmarks against vanilla solution and ROT loss. \n\n[Comments]\nI’m not sure if I fully follow the contribution and several technical details.\n\n It looks to me that the major contribution claimed is the novel loss function that combines the proportion loss and the consistency loss, but both seem to be from off-the-shelf solutions from literature with slight variation. E.g., J_prop is the standard cross entropy loss (Ardehaly & Culotta (2017) and Dulac-Arnold et al. (2019)), and J_cons is from the vanilla consistency definition. I had a hard time getting the novelty here. \n\nThe notion and definition are somehow confusing to me too. What is K in the second line of page 3? Shouldn’t J_prop in page 4 defined as sum of all per-bag losses? In J_cons of pge 4, shouldn’t x under the summation be x_\\hat as x_\\hat is sampled? The equations should be properly numbered for easy reference.\n\nThe use of image attributes in 4.4 for K-means bag generation seems a strong requirement? What kinds of image attributes are used and how are they generated? Looks like this strategy only applies to image classification?\n\nThe results reported could be more clear. The gap between the proposed method and vanilla or ROT does not seem quite big in many cases (less than 1% in the best cases in table 1). I’m not sure if these results are convincing or not as statistical significance is unclear.   \n\nWith all of the above uncertainty, I do not have confidence to have the paper accepted in the current format based on my preliminary assessment.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJgxABc9tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper:  Learning from label proportions (LLP) is an area in machine learning that tries to learn a classifier that predicts labels of instances, with only bag-level aggregated labels given at the training stage.  Instead of proposing a loss specialized for this problem, this paper proposes a regularization term for the LLP problem. The core contribution of this paper is to use the idea of consistency regularization, which has become very popular in semi-supervised learning in the recent years.  The regularization term takes a perturbation of an input sample, and then force the output of the original and perturbed sample to be similar by minimizing  a KL divergence of the two output distributions.  Experiments show the performance of the proposed method under two bag generation settings.  The paper also finds empirically that the hard L_1 has high correlation with the test error rate, which makes it an ideal candidate when the user splits the validation data from training data (meaning there are no ground truth labels for each instances).\n\nReasons for the decision of the paper:  The proposed consistency regularization term seems to be borrowed directly from semi-supervised learning (SSL) research, and is not really specialized for the LLP problem.  Although the discovery that adding this regularization term for the LLP problem increases generalization performance is novel, using this term for data without ground truth labels in image datasets such as CIFAR10/100 and SVHN itself has low novelty, since these datasets satisfy the smoothness assumption used by consistency regularization methods.  Another issue is that there are only one trial for every experiment, making it hard for the reader to see if the results are statistically significant or not.  For these two reasons, it was hard to give a high score decision for this paper.  However, the paper’s motivation is interesting, and the direction to explore regularization techniques for the label proportion learning problem is important and seems novel for this area.  If the regularization method can be extended in a way that relates to the LLP problem, it would make the paper much stronger.\n\nOther minor comments:\n\nIn the experiments, the average test accuracy of the last 10 epochs are reported.  I was curious if the last epochs have better models than earlier epochs.  In many weakly-supervised areas such as noisy labels, it is common that the accuracy goes up very quickly but then gradually decreases.  Does this also happen for the LLP problem?\n\nIn J_cons, does the expectation need to be over p?\n\n\n***After author response:\nThank you for answering my questions!  Although I have a better understanding of the paper now, I still have the same concerns and would like to keep my score.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJlG4z0RtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper proposes using Consistency Regularization and a new bag generation technique to better learn classification decision boundaries in a Label Proportion setting.  The consistency regularization works to make sure that examples in the local neighbourhood have similar outputs. The authors further use K-means clustering to create a new bagging scenario they use to mimic real-world LLP settings. \n\nOverall, this paper sheds light on how we can use the underlying structure to better be able to make predictions on labels even when we don't have original labels but proportional bags. This is a very interesting problem area and will be interesting to the field.\n\n# For the experiments, a few notes and comments?\n1. It's a bit harder to appreciate some of the performance gains here without understanding the error rates around the accuracy. This would be especially good to know for the larger bag sizes where one expects more uncertainty.\n2. For each of the datasets, it would have been also enlightening to provide what the normal labelled performance would be as it would give an indication of the limit when the bag sizes get smaller.\n3. I am a bit confused by the k-means scenario. With bag sizes similar to the uniform bagging scenario, would the performance not be more as we are already capturing latent structure with the k-means algorithm? \n4. Following up on 3, Does this not have implications for actually preserving some privacy?\n\n\n# Other Comments\n5: It would be interesting to see an approach like mixup can be combined with the consistency concept in this case. So what do you expect when you now combine examples and train the underlying mixed bag objective.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: After reading the author's rebuttal, the reviewer still hold that the main contribution is just the simple combination of already known losses. And the paper need to pay more attention on the clarity of the paper.
Decision: Reject