Learning under Label Proportions for Text Classification
Authors: Jatin Chauhan
Xiaoxuan Wang
Wei Wang
Venue: EMNLP 2023 Findings (2023)
Keywords: Label Proportions
Privacy
Weak Supervision
Theory

Abstract: We present one of the preliminary NLP works under the challenging setup of Learning from Label Proportions (LLP), where the data is provided in an aggregate form called bags and only the proportion of samples in each class as the ground truth. This setup is inline with the desired characteristics of training models under Privacy settings and Weakly supervision. By characterizing some irregularities of the most widely used baseline technique DLLP, we propose a novel formulation that is also robust. This is accompanied with a learnability result that provides a generalization bound under LLP. Combining this formulation with a self-supervised objective, our method achieves better results as compared to the baselines in almost 87% of the experimental configurations which include large scale models for both long and short range texts across multiple metrics.

Learning under Label Proportions for Text Classification
Jatin Chauhan
Xiaoxuan Wang UCLA
Wei Wang UCLA weiwangêcs www. edu
UCLA
chauhanjat wwwanntennnns....mo ucla. edl
xw27@cs. edla.eu
Abstract
mand of user privacy in almost all applications of digital mediums, the individual record (here the label) for each sample (say a user's document) can't be exposed and thus learning in a fully super- vised manner deems infeasible. The most notable applications include medical data (Yadav et al., 2016) and e-commerce data Brien et al., 2022), amongst various others. Both of these contain abun- dant language data with multiple use cases of learn- ing classification model perform disease pre- diction from patient's medical data and analyzing user's behavior for example, respectively. This makes LLP a highly relevant learning paradigm to train models, both large and small, for data that needs k-anonymity. The second relevant property is Weak Supervision. Since it is not always feasible to obtain clean data at a large scale, the training of the NLP models, both large and small, relies heavily on weakly supervised or self-supervised learning scrapped from the open web. LLP can play a key role as obtaining data at an aggregated level (formal definition in Section 2) can be a rela- tively casier and more feasible process.
We present one of the preliminary NLP works under the challenging setup of Learning from Label Proportions (LLP), where the data is pro- vided an aggregate form called bags and only the proportion of samples in each class as the ground truth. This setup is inline with the desired characteristics of training models un- der Privacy settings and Weakly supervision. By characterizing some irregularities of the most widely used baseline technique DLLP, we propose novel formulation that is also ro- bust. This is accompanied with a learnability result that provides generalization bound un- der LLP. Combining this formulation with self-supervised objective, our method achieves better results compared to the baselines in almost 87% the experimental configurations which include large scale models for both long and short range texts across multiple metrics (Code Link).
1 Introduction
The supervised classification setting in machine learning usually requires access to data samples with ground truth labels and our goal is to then learn a classifier that infers the label for new data points. However, in many scenarios obtaining such dataset with labels for each individual sample can be infeasible and thus calls attention alternative training paradigms. In this work, we study the setup where we are provided with bags of data points and the proportion of samples belonging to each class (instead of the one-one correspondence berween a sample and its label). The inference however, still needs to be performed at the instance level in order to extract meaningful predictions from the model. This setup, which following (Yu Learning from Label
While there exist various prior works that have proposed new formulations to learn under this set- ting (Musicant et al., 2007; Kuck and de Freitas, 2012; Tsai and Lin, 2019), many of them ei- ther readily applicable to learning deep models or very difficult to align with language data. Fur- thermore, to our knowledge, there exists only one prior work of (Ardehaly and Culotta, 2016) that discusses LLP for language data but is out of scope of this work they focus on domain adaptation. We provide an elaborate discussion of the LLP lit- erature in Section 5.
In this work, of one of the first loss formulations proposed for learning deep neural networks under LLP setup by (Ardehaly and Culotta, 2017), termed DLLLP method. For given bag, DLLP optimizes the KL divergence between the ground truth bag level pro-
address some shortcomings
al., 2013) we refer Proportions (LLP), attractive especially for at least two broad reasons (Nandy et al., 2022).
The first of these is Privacy centric learning (O' Briee et al., 2022). With ever increasing de-

---

portion and the aggregate of the predictions for the instances in the entire bag. In Section 3, we high- light certain properties of DLLP objective that can be highly undesirable for training deep networks. Motivated by this, we propose a novel objective function that is parametrization of the Total Varia- tion Distance (TVD), which itself is lower bound to the KL via the Pinsker's inequality. Our formu- lation enjoys more functional flexibility because of the introduced parameter while retaining the outlier robustness property of the TVD. We also discuss some theoretical results for the proposed novel formulation. Lastly, we combine our formu- lation with an auxiliary self-supervised objective that greatly aids in representation learning during the fine-tuning stage of the large scale NLP models experimented with.
as:
[x E B,y; = =
p²
(1)
B
It is important to note that the ground truth labels Yi are not accessible and the entire information is contained in p. This p essentially provides the weak supervision for training the model.
Given the training bags along with the label pro- portions as the tuple (B,, pt) the objective is still to learn an instance level predictor that maps an instance x to the class distribution A {z E _ Σ of = 1). The predicted class attained as: arg max A. We denote the classifier by fo X 7 A with learnable parameters O E θ which typically is neural network such BERT with appropriate classification head.
Experimentally, we first demonstrate that the proposed formulation is indeed better and align with the theoretical motivation provided. In the main results, we demonstrate that our formulation achieves better results compared to the baselines in almost 87% of the 20 extensive configurations across 4 widely used models and 5 datasets. We observe up to 40% improvement in weighted pre- cision metric on the BERT model. In many cases, the improvements range between 2% to 9% which are highly substantial. Further analysis including the bag sizes and hyperparameters also provide interesting insights into our formulation.
3 Method
Since the supervision under the LLP setup is coarse grained, given by p. the works in the literature (Ardehaly and Culotta, 2017), (Dulac-Arnold et al., 2019) utilize training criteria based on the true proportions Pi and predicted proportions LG for the input bag B,. The predicted proportions Ps are typically some function of predicted class distributions fo(x,) )) for Xj E B, as 6: (x.). for g is the mean function and we retain the same in this work. A descriptive pipeline of the LLP setup is provided in Figure 1.
B.))). The popular choice
To summarize, we have the following contribu- tions: (i) A novel loss formulation that addresses the shortcomings of the previous work with sup- porting theoretical and empirical results. (ii) One of the preliminary works discussing the applica- tion of LLP to natural language tasks. (iii) Strong empirical results demonstrating that our method outperforms the baselines in most of the configura- tions.
3.1 Motivation
The work of (Ardehaly and Culotta, 2017) pro- posed one of the first loss formulations to learn deep neural networks under the LLP setup. Their loss objective over the proportions and p. is given by:
2 Preliminaries
LOLLP Σ poog p
We consider the standard multi-class classification setup with C' classes (C [C] = (1,.. C)). The input instances are sampled from an unknown distribution over the feature space X. Contrary to the availability of the data samples of the form (x, y), where supervision, we are provided bag of input in- stances, = {xq E [[]]] (B) is the cardi- nality), with associated proportions of the labels p = ,p,,.., The elements are defined
p
Definition 1 (Lipschitz Continuity). A function is called Lipschitz continuous if there exists a positive constant K such that [[h(x) M(y)l < K K|| - y|l, Væ, E dom(h).
C, to train the model in full
We note some irregularities of DDLP which can lead to sub-optimal parameter values post training:

---

Training Phase
Test Phase
Stacked Enssder Layera
Model Stacked Encoder Legers
Bog of Instances
Some Neresten
Sun Domatere
Single Instance
Instance Pred cton
Proportion Loss
Figure I: Demonstration of the Training (left) and Test (right) phase for LLP setup respectively. During training. bag of instances is received and the output probabilities 4 class example setup, outputs colored based on the arg max values) generated by the model are aggregated (computation of p). The proportion loss against the true proportion is then computed. the test phase, the predictions are required to be performed the instance level.
1. It is not upper bounded: Since the denomi- nator of the log is the quantity P. which can be arbitrarily close to O, the loss can go bounded on the positive real axis.
where, DKL. is the KL. Divergence and Drv is the Total Variation Distance, pe as defined previously. pe is used shorthand notation over the domain here.
2. It is not robust: follows from the previous point as small changes in o and of can lead to huge variations in the output.
3.2 Proposed Formulation
Drv has been utilized as a loss function in mul- tiple works in the deep learning literature (Zhang et al., 2021; Knoblauch and Vomfell, 2020). While admits a lower bound to the LDLLP term, more importantly it is robust (Knoblauch and Vomfell, 2020). To further obtain higher functional flexi- bility while retaining this robustness property, we propose the following loss formulation, termed Liv, with hyperparameter a:
3. It is not symmetric: due to the asymmetric nature of the definition of the objective.
4. It is not Lipschitz continuous in either of the arguments: follows from the first point.
5. It is not Lipschitz smooth (associating to gra- dients) in either of the arguments: stating that the gradients of the loss not Lipschitz continuous. This can be associated with the largest eigenvalue of the Hessian, which exists for most losses in NN training and is critical to the training dynamics (Gilmer et al., 2022), thus dictating the optima, if achieved.
2/a
1 CtV -
Σ
lp p01a LEE
(3)
We discuss and prove various properties of L in Appendix A.2 and note that it does not admit any of the aforementioned irregularities stated in Section 3.1 for a broad range of values for a thus providing a better alternative. Further, we also empirically demonstrate the superiority of optimizing the model with the Liv. against L DLLP in Section 4.
The aforementioned irregularities are undesir- able especially for deep neural network optimiza- tion. This motivates us to propose a new loss func- tion that partly remedies the potential failure modes of CPLLP. To begin, we state the following result:
Lemma 1. [Pinsker's 's Inequality] For the probabil- ity mass functions and p.
3.3 Learnability under Li
We provide the following learnability result for class of binary classifiers under the proposed formu- lation LT The extension to multi-class setting is feasible via the use of Natarajan dimension and corresponding tools, however it is slightly more sophisticated and thus we leave it for future work.
CDLLP = DKI(Pllp) 2
Σ 100h 17)
EEC
= Drv(ellp)
(2)

---

Definition 2 (VC Dimension). The maximum car- dinality of S (x1, .m) E xm of points that can be shattered by the hypothesis class H under consideration.
from the penultimate layer of the model. The high- light of their proposed approach is that it does not require explicit data augmentation or random sam- pling to construct negative pairs. This directly im- proves the runtime of the algorithm by a factor of 2, assuming linearity, as compared to many other pop- ular choices of SSC losses (we refer the reader to the survey of (Jaiswal et al., 2020)). We therefore utilize their SSC objective improve the overall performance of our formulation. Rewriting the model as fo = f (x)) where f and f² are the parametrized sub-networks performing classi- fication (only the final layer, ie, the classification head) and the representation learning respectively, the loss is given as
Theorem 2. Consider hypothesis class H con- taining functions X + {0,1} and target function fo II. Assume a sample S with m instances sampled i.i.d from X. We represent the VC dimension of the class H by V. Fur- ther, denoting Ps = (fx=))((()=]]]] where μ rep- resents the probability measure over X and also P. Exes [f(x]=1]- Then with probability at least 1 - Vf E H and'va > 0, we have
C Liv.lps-Po PS)) S VVooglem/V)
(4/8(4/))
(4)
Lssc = |Bi
1
Σ X,EB,
log ExEB, (x)()))))
e*(((()))1(1x))))
m
where K. 22/0-11
(5)
The proof of the theorem is delegated to the Appendix. This result characterizes that for any function f the hypothesis class, the loss Ltv. between the population output aggregates obtained from f and the target function can be bounded via the corresponding sample output aggregates. We use the term aggregates to denote the mean over the indicator values [f(x)=1]- (Fish and Reyzin, 2017) provided one of the early results under the framework of Learnable from Label Proportions, where they provided in-depth analysis for the specific case which corresponds to in our formulation.
here is similarity function between the embeddings, given by cosine, s(a, b) Toll (b12
a
Thus the overall training objective for our method is the following objective:
== L= + ALssc
(6)
where is the hyperparameter to control the auxil- iary loss. The other hyperparameter in this objec- tive is a under LTV.
Experiments
To investigate the effectiveness of the novel loss formulation in Eq. 6, we conduct extensive experiments across various models and datasets. These cover both short and long range texts as well as corresponding models developed to handle these texts. We begin by providing a brief overview of the setup.
3.4 Auxiliary Loss
While the theorem above provides guarantees for the performance over aggregated output, in order to further improve the quantitative performance of the model, we follow the success of various works in the deep learning literature that utilize auxiliary losses along with the primary training objective for improved generalization of the model (Trinh et al., 2018; Zhu et al., 2022; Rei and Yannakoudakis, 2017). This is directly tied to better representation learning for the model.
4.1 Setup
Models: We consider the following models: (i) BERT proposed by (Devlin et al., 2019) where we use a fully connected layer on the [CLS] token for classification. The documents are truncated at length of 256. (ii) RoBERTa proposed by (Liu et al., 2019b) where we use the similar setting BERT to perform classification. (iii) Longformer proposed by (Beltagy et al., 2020) is explicitly de- signed to handle long range documents via an atten- tion mechanism linear in the sequence length. The
The category of Self Supervised Contrastive (SSC) losses has emerged one of the most promising representation learning paradigms in recent years (Fang and Xie, 2022; Meng al., 2022) and thus a popular choice for auxiliary losses as well. The recent work of (Nandy et al., 2022) proposed an SSC loss based on the embeddings of the instances

---

input documents are truncated at the length of 2048. (iv) DistilBert proposed by (Sanh al., 2019) is a distilled version of the primary BERT model with only around 40% of the parameters but retaining up to 95% performance. We truncate the sequences at the length of 512. All models are loaded using the Huggingface Transformers library (Wolf et al.. 2019).
is much harder and potentially requires extremely large sized datasets (Nandy al., 2022). Adher- ing to the memory constraints, we utilize one bag per iteration typically containing 16- 32 instances (Tables 6 and 7) which is the standard batch size range for fine-tuning, Although, there is no strict constraint for the number of bags per iteration and more bags can be utilized given sufficient memory. Most of the fine-tuning details and parts of code are retained from the recent work of (Park et al., 2022) which provides an extensive comparison of large scale models for standard supervised classifi- cation. Since we don't assume access to true labels even for the validation set, common techniques for parameter selection (based on performance on the validation set) are difficult to utilize. Following (Nandy et al., 2022), we thus use aggregation of both training and validation losses across the last few epochs to tune the hyperparameters. Tables 6 and 7 detail the important hyperparameters. All the experiments were conducted on a single NVIDIA A 100 GPU.
Datasets: We experiment on the following datasets: (i) Hyperpartisan News Detection (Kiesel al., 2019), is news dataset where the task is to classify an article having an extreme left or right wing standpoint. (ii) IMDB (Maas et al., 201 1), is a movie review dataset for binary sentiment clas- sification. (iii) Medical Questions Pairs (Medi- cal QP) (McCreery et al., 2020), contains pairs of questions where the task is to identify if the pair is similar or distinct. Following the common practice, we concatenate the questions in given (iv) Rotten Tomatoes (Pang and Lee, 2005), is a movie reviews dataset where each review is classified as positive or negative. Along with these, we also include a multi-class (non-binary) dataset Financial Phrasebank (Malo et al., 2014) with 3 classes to assess the formula- tions under a more difficult setting. It is sentiment dataset of sentences written by various financial journalists from financial news. The statisties of the datasets are provided in Table 5.
pair before tokenizing.
While there are no prior works di- rectly studying the setup of LLP for natural lan- guage classification tasks, we consider two existing approaches that can be readily applied here. The first is the formulation of (Ardehaly and Culotta, 2017) which we refer as DLLP and the correspond- ing loss as LDLLP, presented in Section 3. The second is the work by (Nandy et al., 2022) which uses the LDLLP loss in conjunction with the con- trastive auxiliary objective in Eq. 5, which we will refer as LLPSelfCLR baseline. They also propose an additional diversity penalty to further diversify the learned representations. Section 5 provides an elaborate overview of other works in the literature.
Baselines:
We fine tune the pretrained versions of the 4 models under the LLP setup over the bags. The input to the models follows the structure of (input, label), where "input" is a bag B; and "label" is the ground truth proportion Ps- To construct the bags B, during training, we gather the instances in groups based on the size mentioned in Tables 6 and 7, from the list of all training instances. To obtain the corresponding ground truth label proportions per bag Pi. we directly average the onc-hot vector representations of the labels for the instances in the bag. Note that the ground truth labels are not used at any other phase of the setup except testing which is performed at the instance level. There, the use of the ground truth labels is restricted to com- pare the performance of our formulation against the basclines. We construct new bags at each epoch by shuffling the data. This is inline with practical purposes where an instance can be shared across different bags across different training epochs, for example- advertising (CTR prediction) and medi- cal records. The setup with fixed bags beforehand
Training:
We use the weighted versions of precision, re- call and Fl score to evaluate the performance of the models as these metrics quantify the performance in a holistic and balanced manner. The acronyms W-P, W-R and W-FI for these metrics are used while describing the results. We perform the analy- sis experiments on diverse configurations in order to ensure coverage and completeness across the possible combinations.
4.2 Main Results
Table highlights the main results of our complete loss formulation in Eq. and comparison against the baselines on the binary classification datasets. We also quantify the difficulty of the LLP setup

---

Base Model
Formulation
Dataset
Hyperaarison
Tomatnes
Medical QP
w-F
W-R
W-T1
W-F
WRR
W
W-F
W-R
W-F1
W-F
W-K
W-FI
34.17
43.13
55.25
51.45
50.41
Longformer
LLPSeRCLLR
79.94
23.38
72.99
90.39
90.38
90.38
82.57
83.42
83.37
53.18
53.03
52.55
Ours
79.92
79.94
92.49
92.47
85.07
85.01
53.19
52.52
Omale
95.72
95.38
95.33
95.49
95.48
95.48
85.63
88.39
88.58
86.91
86.37
86.32
DLLP
34.17
58.46
45.13
89:03
89.01
89.01
28.83
09.29
66.39
25.08
50.08
33.42
RoBERTa
L.PSARCLR
54.34
53.84
54.04
00.08
90.08
77.42
72.24
27.19
50.47
49.31
Ours
61.06
61.53
56.21
90.14
90.13
90.13
83.41
85.30
85.29
53.70
53.53
53.03
Oracle
69.6
64.61
64.39
93.58
93.58
93.57
88.47
88.45
88.44
83.62
82.92
82.83
DLLP
55.10
58,46
45.69
84.33
84.35
84.33
75.04
74.13
73.89
56.36
56.32
56.22
50.71
56.92
46.95
86.85
86.82
86.83
75.04
74.13
73,89
66.08
66.00
65.96
BERT
Ours
77.36
63.07
52.73
87.58
87.50
87.49
74 94
73,80
73.49
64.54
64.53
64.52
87.70
91.37
85.86
$1.44
81.25
34.17
86.85
A5.85
79.6.
27.41
76.98
25.08
33.42
LLPSeRCLR
67.92
67.69
65.60
86.89
85,86
85.85
79.53
77.41
76.95
52.73
$2,70
52.63
DistiBERT
Ours
70.21
87.64
87.62
77.46
55.17
55.17
Oracle
92.56
92.30
92.22
92.81
92.82
92.81
83.76
83.75
83.75
77.85
76.35
76.04
Table 1: Comparison of our formulation against the baseline methods DLLP and LLPSeIICLR. Oracle denotes the performance in the supervised setting, the bag size of 1, and is demonstrated solely to quantify the difficulty of the problem. The best numbers are highlighted in bold and second best underlined. Our formulation achieves better results in almost 83% of the configurations.
Formulation
Longtormrr
RoBERTa
precision. On the other hand, in the settings where we outperform the basclines, the relative gains are as large as up to 10% (BERT on Hyperpartisan dataset) and up to 10% (RoBERTa on Rotten Toma- toes and Hyperpartisan datasets) in weighted preci- sion. Weighted Recall and FI -scorr mostly show similar gains in the corresponding configurations. In various configurations including DistilBERT and RoBERTa Medical QP, Longformer IMDB and Rotten Tomatoes), also observe notable improvements ranging from 2% to 9% uniformly across the metrics, These results also align with the architectural designs and findings of previous works as Longformer achieves comparatively better performance especially on long-range datasets: Hy- perpartisan and IMDB while Dis exhibits more robust results compared to BERT. It is also important note that the margins observed for our formulation are significant for the relatively small sized dataset Hyperpartisan which also generalizes to Medical QP to an extent. To summarize, the gains exhibited by our formulation are consistent across all models and datasets including small and large sized datasets with both long and short texts. The results for the Financial Phrasebank dataset are provided in Table 2. Similar to the results in the binary dataset configurations, again observe that our formulation achieves consistently better results across the models. Although the gains are not as high as 40% which were observed in some binary settings, we still note significant improve- ments of around 23% w.r.t to DLLP and 5.5% w.r.t
W-R
W.P
DLLP
09.97
10.18
65.85
76.97
73,91
72,44
LLPSeLICLR
78.57
78.30
78.62
Ours
81.38
79.50
79.06
80.64
77.43
76.82
Oracle
80.53
82.64
$1.18
BERT
DistiBERT
W-P
W-R
WiFI
W.P
W-R
W-FI
DLLP
61.90
74.14
P.PS
23.92
70.80
20.96
72.45
28.08
71.80
Ours
78.02
75.67
73.88
76.58
75.46
72.75
Oracle
79.67
77.01
76.31
80.90
77.63
78.12
Table 2: Comparison of our formulation against the baselines on Financial Phrasebank dataset. The nota- tions follow same meaning as described previously. Our formulation achieves hetter results across all the models.
by providing the results in the corresponding fully supervised setup, with bag size I, named as Ora- cle. Note that Oracle is not a true feasible baseline here. The same result for DILP and LLPSeIfCLR in some configurations in the table correspond to the cases where L.PSSIfCLR does not outperform DLLP baseline and the optimal choice for its hyper- parameter controlling the contrastive loss is almost 0. Same results for the DLLP baseline across the models such as RoBERTa, Longformer and Distil- BERT on Hyperpartisan dataset, demonstrate that is not able to optimize and learn properly. Similar observations are accounted for Medical QP dataset. We that our formulation achieves the best re- sults in almost 83% of the values in the table cu- mulating across all configurations in the precision, recall and Fl scores. In the settings where we don't 't outperform the baselines, the relative margins are extremely small, often in the order of the decimal

---

Dataset
W-P
W-R
W-FI
method against the baselines for varying bag sizes, Previous works (Liu al., 2019a; Dulac-Arnold et al., 2019) have exhaustively demonstrated that the performance of the models degrade as the bag size increases. This is due to reduction in the su- pervision available to train the models. Figure 2(a) plots the comparison for the different formulations for RoBERTa Rotten Tomatoes dataset. These results align with the findings in the literature counting for the amount of supervision provided. noteworthy that our method is more stable compared to the baselines. For smaller bag sizes (2 8), all the models demonstrate fairly competi- tive performance. For the size 16 DLLP performs closer to our formulation however LLPSelfCLR incurs non-trivial degradation. For larger bag size, the performance reduces further (significantly for DLLP) and the gap between our formulation to the baselines admit large difference. This result thus inline with the claims presented for our formulation in Section 3.
DLLP
34.17
58.46
43.13
Hyperpartisan
Ctv.
76.80
61.53
49.72
Overall
79.92
80.00
79.94
DLLP
64.66
55.25
46.71
Rotten
Civ.
84.37
84.32
84.31
Tomatoes
Overall
85.61
85.07
85.01
Table 3: Comparison of DLLP loss against our proposed Ctv. in Eq. 3 (note this is withou the auxiliary loss of Eq. 5) for Longformer. Performance the overall loss Eq. 6, is also provided.
LLPSeIfCLR on W-P for BERT model. Generally speaking, we observe around 2 3% gains on the metrics across the models. These relative margins can be attributed to the fact that the multi-class set- ting with more than 2 classes is more challenging in both theory and practice.
4.3 Analysis
4.3.1
Justification of v,v. and Lssc
As stated in Section 3, the DLLP loss admits var- ious undesirable properties which motivated us propose the first part of the loss function in Eq. 6. Here, we provide an empirical evidence that Liv, (Eq. 2) indeed generalizes better than LDLLP under various configurations. Table 3 pro- vides the results comparing the two formulations on Longformer model over Hyperpartisan and Rotten Tomatoes datasets. It is evident that our formula- tion achieves significantly better results of which particularly noteworthy are the metric values of W-P on Hyperpartisan, more than 2x, and W-F1 on Rotten tomatoes, almost 1.8x. Other values also highlight similar trends and noteworthy margins. More experiments are provided in appendix A.1.1. Correspondingly, we can also observe the practi- cal benefits of using SSC objective by comparing the results of LTv. against the overall loss formu- lation of Eq. in Table 3.We note that for Hy- perpartisan, the relative improvements brought via the SSC objective are more notable in the W-F1 score. On the other hand, for Rotten Tomatoes, we observe only marginal improvements using SSC. This demonstrates that while Liv, leads to strong results, using auxiliary objective such as an SSC loss can further improve the performance, notably marginally.
4.3.3 Variation of hyperparameters
To analyze the effect of the hyperparameters on the performance, we conduct experiments by varying the values of a and X, Eq. 6. While varying the value of one, the other is kept constant. The results are plotted in Figures 2(b) and 2(c). The analysis for is performed over BERT on Medical QP and for over BERT on IMDB ensure diverse cov- erage. For both the subfigures corresponding to a and X, it is non-trivial to find a direct trend. This aligns with many works in various machine learn- ing literatures since the hyperparameters play a key role in determining the output performance and a small variation can account for large deviations in the model performance. In both the configurations, first observe the optimal performance point we increase the value of the corresponding hyper- parameter followed by a sharp degradation and noteworthy further improvement. However, that for almost all the cases, all the 3 metrics pro- vide highly calibrated results (ie, highly balanced results) across the y-axis, thus further verifying the efficacy of our approach.
5 Related Work
We discuss the literature of LLP via the following taxonomy:
4.3.2 Variation of Bag size
The size of the input bags is a direct control factor for the performance in the LLP setup. We thus con- duct an experiment to note the performance of our
Shallow Learning based: This is the category of models that doesn't 't leverage deep learning models.
The notable works include (Musicant et al., 2007)

---

Metric Valuee
Weightes
Weghtal Precuaon Wegtoed Recal Weihtted
Waghiad Precsess Waggried Recal
Wiegheed
50
-22 -10 -as Value of a (Log Base scale)
10
Bag Size
Value of (lgg Base 10 scale)
Variation of Bag Size
(b) Variation of a
(c) Variation of À
Figure 2: Variation of Bag size (subfigure a): For increasing bag size, we observe substantial difference between our method and the baseline formulations. Analysis of Hyperparameters a (subfigure b) and À (subfigure c): We don't observe a direct trend however the performance is well calibrated across the metrics.
(using SVM, N-NN), (Kuck and de Freitas, 2012; Fernández-Gonzääle et al., 2013) (probabilistic models), (Scott and Zhang, 2020) (mutual contami- nation framework) etc. Note that these methods do not scale for higher dimensional and larger datasets, thus making these unfit for language tasks.
this overhead can be larger than the primary objec- tive task itself. For example, the generation new instances via GANs. (Nandy et al., 2022) proposed a contrastive self-supervised auxiliary loss to im- prove the representation learning of the underlying deep network in conjunction with the DLLP loss. They augment this with extra penalty term further diversify the representations learned.
Deep Learning based: Some of the notable works include (Ardehaly and Culotta, 2017) and (Dulac-Arnold et al., 2019) which aim to minimize divergence between the true proportions P and the predicted proportions of given bag ý. (Dulac- Arnold et al., 2019) also used the concept of pseudo labeling (Lee, 2013), where an estimate of the indi- vidual labels within each bag is jointly optimized with the model during training. (Liu et al., 2021) revisited pseudo labeling for LLP by proposing a two-stage scheme that alternately updates the net- work parameters and the pseudo labels. However, both these works leveraging pseudo labeling per- form experiments with computer vision datasets only. Furthermore, open source codes could be found thus making difficult to compare against these formulations.
Lastly, to our knowledge, the only work that emphasizes the setting of LLP for natural language classification tasks is the work by (Ardehaly and Culotta, 2016). However, it is out of the scope of this work as they aim to improve the performance of models under the LLP setup when the training and testing data admit change the underlying domain distributions, ie, the Domain adaptation setting.
Conclusion
We study the setup of learning under label propor- tions for natural language classification tasks. The LLP setup admits two properties: Privacy centric learning and Weak Supervision that render it as highly practical modern rescarch topic. We first discuss the shortcomings of one of the first LLP loss formulations proposed for learning deep neu- ral networks. These provide the motivation for our novel loss formulation, termed as 29 that does not admit such irregularities. The formulation is also supported with theoretical justification and hardness results. Empirically, justify that the new loss function achieves better performance on diverse configurations. The main results achieved by combining this loss with a self-supervised con- trastive formulation achieve significantly better per- formance as compared to the baselines.
Auxiliary Loss based: Recent trends in learning with auxiliary losses alongside primary objec- tive also gained attention the LLP community, (Liu et al., 2019a) proposed LLP--AN framework by incorporating generative adversarial networks (GAN). (Tsai and Lin, 2019) proposed LLP-VAT that incorporated an additional consistency regu- larizer to learn consistent representations in the setting where the input instances are perturbed ad- versarially. is noteworthy that both these methods leverage vision datasets for experiments. The tasks of generation and adversarial perturbation incur significant overhead for NLP and in various cases,

---

7 Limitations
Justin Gilmer, Behrooz Ghorbami, Ankush Garg, Sneha Kudugunta, Behnam Neyshabur, David Cardoze, George Edward Dahl, Zachary Nado, and Orhan Fi- rat. 2022. A loss curvature perspective on training instabilities of deep learning models. In International Conference on Learning Representations.
While the proposed formulation achieves better re- sults in the majority of the configurations, the gap between the best results and the corresponding su- pervised oracle is large some cases, This can be configuration dependent and thus harder to inter- pret. Another limitation which follows from the black-box nature of the large models is that the current theoretical understanding of the proposed formulation are not sufficient especially at the in- terplay between CTV. and Lsse in eq 6. More work in this direction will help unfold not only the underlying mechanisms but also aid in designing better loss formulations.
Jerónimo Hrrnández-Gonzälle Iñaki Inza, and Jose A. Lozano. 2013. Learning bayesian network clas- sifiers from label proportions. Pattern Recogn., 46(12):3425-344)
Ashish Jaiswal, Ashwin Ramesh Babu, Moham- mad Zaki Zadeh, Dehapriya Banerjee, and Fillia Makedon. 2020. A survey on contrastive self- supervised learning.
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Em- manuel Vincent, Payam Adineh, David Comey, Benno Stein, and Martin Potthast. 2019. SemEval- 2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 829-839, Minneapolis, Minnesota, USA. Association for Computational Lin- guistics.
References
Ehsan Mohammady Ardehaly and Aron Culotta. 2016.
Domain adaptation for learning from label propor- tions using self-training. In Proceedings of the Twenty-Fifth International Joint Conference on Artifi- cial Intelligence, IJCAI' 16, page 3670-3676. AAAT Press.
Jeremias Knoblauch and Lara Vomfell. 2020. Robust bayesian inference for discrete outcomes with the total variation distance.
Hendrík Kuck and Nando de Freitas. 2012. Learning about individuals from group statistics.
Ehsan Mohammady Ardehaly and Aron Culotta. 2017. Co-training for demographic classification using deep learning from label proportions. In 2017 JEEE International Conference Data Mining Workshops (ICDMW). IEEF
Dong- Lee. 2013. Pseudo-label The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop Challenges in Representation Learning (WREPI))
Iz Beltagy, Matthew E. Peters, and Arman Cohan, 2020. Longformer: The long-document transformer.
Jiabin Liu, Bo Wang. Zhiquan Qi, Yinglie Tian, and Yong Shi. 2019a. Learning from label proportions with generative adversarial networks. In Advances in Neurul Information P Processing Systems, volume 32. Curran Associates, Inc.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186. Minncapolis, Minnesota. Association for Computational Linguistics.
Jiabin Liu, Bo Wang, Xin Shen, Zhiquan Qi, and Yingjie Tian. 2021. Two-stage training for learning from label proportions. In Proceedings of the Thirtieth International Joint Conference on Artificial Intel- ligence, IJCAI-21, pages 2737-2743, International Joint Conferences on Artificial Intelligence Organi- zation. Main Track.
Gabriel Dulac-Arnold, Neil Zeghidour, Marco Cuturi, Lucas Beyer, and Jean-Philippe Vert. 2019. Deep multi-class learning from label proportions.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining ap- proach.
Hongchao Fang and Pengtao Xie. 2022. An end-to-ooe and-to-end contrastive sulf-supe learning framework for language understanding. Transactions of the Asso- ciation,for Computational Linguistics, 10(0):1324- 1340.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham. Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Benjamin Fish and Lev Reyzin. 2017. On the com-
plexity of learning from label proportions. In Pro- ceedings of the Twenty-Sixth International Joint Con- ference on Artificial Intelligence, IJCAI-17, pages 1675-1681.

---

P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. 2014. Good debt or bad debt: Detecting se- mantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65.
Systems, volume 33, pages 22256-22267 Curran As- sociates, Inc.
Shai Shalev-Shwartz and Shai Ben-David. 2014, Un- derstanding Machine Learning: From Theory to Al- gorithms. Cambridge University Press.
Clara H. McCreery, Namit Katariya, Anitha Kannan, Manish Chablani, and Xavier Amatriain. 2020. Effec- tive transfer learning for identifying similar questions: Matching user questions to covid-19 faqs.
Trieu Trinh, Andrew M. Dai, Minh- Minh-Thang Luong. and Quoe V. Le. 2018. Learning longer-term depen- dencies in rass with auxiliary losses.
Zhao Meng, Yihan Dong, Mrinmaya Sachan, and Roger Wattenhofer. 2022. Self- superviee contrastive learn- ing with adversarial perturbations for defending word substitution-bass bbased attacks. Findings of the Associ- ation for Computational Linguistics: NAACL 2022, pages 87-101, Seatle, United States. Association for Computational Linguistics.
Kuen- Meen--annPe Tsai and Hsuan-Tien Lin. 2019. Learning from label proportions with consistency regulariza- tion.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- Cistac, Tim Rault, Rémi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Hug- gingface's transformers: State-of ofate-of-the-e the--rt- natural lan- guage processing.
David R. Musicant, Janara M. Christensen, and Jamie F. Olson. 2007. Supervised learning by training on aggregate outputs. In Proceedings of the 2007 Sev- enth IEEE International Conference on Data Mining, ICDM '07, page 252-261, USA. IEEE Computer Society.
Shweta Yadav, Asif Ekbal, Sriparna Saha, and Push- pak Bhattacharyya. 2016. Deep learning architecture for patient data de-identification in clinical records. Proceedings of the Clinical Natural Language Processing Workshop ( (ClinicalNLP), pages 32-41, Osaka, Japan. The COLING 2016 Organizing Com- mittee
Jay Nandy. Rishi Saket, Prateek Jain, Jatin Chauhan, Balaraman Ravindran, and Aravindan Raghuveer. 2022. Domain-agnostic contrastive representations for learning from label proportions. In Proceedings of the 31st ACM International Conference on Infor- mation Knowledge Management, CIKM '22, page 1542-1551, New York, NY, USA. Association for Computing Machinery.
Felix Yu, Dong Liu, Sanjiv Kumar, Jebara Tony, and Shih-Fu Chang. 2013. osvmm for learning with la- bel proportions. In Proceedings of the 30th Interna- tional Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 504-512, Atlanta, Georgia, USA. PMLR.
Conor O'Brien, Arvind Thiagarajan, Sourav Das, Rafael Barreto, Chetan Verma, Tim Hsu, James Neufield, and Jonathan J Hunt. 2022. Challenges and ap- proaches to privacy preserving post-click conversion prediction.
Yivan Zhang, Gang Niu, and Masashi Sugiyama. 2021. Learning noise transition matrix from only noisy la- bels via total variation regularization.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACI.
Lvxing 7hu, Hao Chen, Chao Wei, and Weiru Zhang. 2022. Enhanced representation with contrastive loss for long-tail query classification in e-commerce. In Proceedings of the Fifth Workshop on e- e-Commerce and NLP (ECNLP 5), pages 141-150, Dublin, Ireland. Association for Computational Linguistics.
Hyunji Park, Yogarshi Vyas, and Kashif Shah. 2022. Efficient classification of long documents using trans- formers. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume Short Papers), pages 702-709, Dublin, Ireland. Association for Computational Linguistics.
Marek Rei and Helen Yannakoudakis. 2017. Auxiliary objectives for neural error detection models. In Pro- ceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 33-43, Copenhagen, Denmark. Association for Com- putational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.
Clayton Scott and Jianxin Zhang. 2020. Learning from label proportions: A mutual contamination frame- work. In Advances in Neural Information Processing

---

Dataset
W-P
W-R
W-FI
Proof. Given the arguments as discrete distri- butions Pt P2, we prove the result for pt and by symmetry follows in both arguments.
DLLP
86.06
85,89
85.87
IMDB
Liv.
88.40
88.36
88.36
Overall
92.49
94.48
92.47
Financial
DLLP
69.97
70.18
65.85
We have that Liv, (P1, P2) is differentiable almost everywhere in the interior the simplex. We com- pule the gradient of LTv, (PI: (2) w.r.t to pi, cor- responding to the 7th element in the support.
viv.
76.36
74.94
71.48
Phrasebank
Overall
81.38
79.50
79.06
Table 4: Comparison of DLLP loss against our proposed Ctv- in Eq 3 (note this is without the auxiliary loss of Eq 5) for Longformer. Performance of the overall loss
8L (PI.2(P1.P2) Op}
Eq 6, is also provided.
Σ pi
P2l
x p 221 1 x sign(p) -pb)
A
Appendix
A.1
Additional Experiments
Here, sign(-) is the sign function. To show hipschitz.cnntnnuity. it sufficient to show that the norm of the gradient, |VV (p1, p2)]l. is bounded. Now,
This section contains additional experiments to sup- port the proposed loss formulation as well as fur- ther analyse the hyperparameter variation. The di- selection of model and datasets in the follow- ing experiments provide sufficient coverage across the configurations.
V (VC.vv(r1 =
Σ lpt - -2/1
A.1.1 Analysis of Civ.
Table 4 reports the comparison of DLLP against Civ+ on more datasets over longformer model. We note that the findings from table extend other configurations as well.
lot 20-2 (8)
Given a fixed number of classes, is trivial that
A.1.2 Variation of hyperparameters
VC P2)1l (k((C)
(9)
The variation of a value has been plotted in figure 3(a) for BERT on Hyperpartisan dataset. The variation of À value has been plotted figure 3(b) for RoBERTa on Financial Phrasebank dataset.
Here, k(C)) is a constant function of the number of classes. Thus, Lipschitz continuity holds.
Lemma 6. L Lipschitz smooth in both the arguments for a 2 1.
A.2 Properties of Div.
Here, we discuss the theoretical properties that the proposed loss formulation V% admits. The fol- lowing lemmas demonstrate that it does not exhibit any of the irregularitics highlighted Scction 3.1 for a broad range of values of used in majority of experiments.
Proof. Following the previous lemma, given the two arguments as discrete distributions P1, P1,P2, we prove the result for p1 and by symmetry it follows in both arguments.
We begin by noting that Liv, is convex function in either of the arguments due to the convexity of norms.
Lemma 3. Liv= admits an absolute upper bound for all inputs pairs and ğ for a 1.
Furthermore, we also have
Proof. For 2 1, using that Drv S 1, we have that CVV. <2x D+v 2
a < Liv. S D2
(10)
for constants C1 and c2. This is obtained using equivalence of norms.
Lemma 4. v symmetric.
Proof. By definition of C we have Ctv, (p. P) (p.
To show the desired smoothness result, it is suf- ficient to prove that the function
O
Lemma 5. Ltv. is Lipschitz continuous in both the arguments for a 2 1.
B p (9 (11)
f(pi) -

---

Metric Value
Wegrtte Pratsion Rocal
-35 -La
Lo
Value of a (Log Base 2 scale!
Value of llag Base scale)
Variation of cx
(b) Variation of À
Figure 3: Variation of hyperparameters.
is convex Yp1 and fixed p2. B being the smooth- ness constant.
H, we have:
2log(2/0) m
1
(x
1((,2) S
The key element to the proof is the convexity of the following
m
ZES
2m
US x ml H]]
(14)
Be T (tv
(12)
2
Using the lemma we have that
Be, being the smoothness constant, which depends upon the fixed constant 61 stated in Eq 10.
((., H3|
(15)
Thus, upper bound Eq 11 using Eq 12 to obtain the desired Lipschitz smoothness Va 2 1.
and we also have
em
((,,),))).,,,f,,mm)))). H) <
v
A.3 Proof of Theorem 2
(16)
Definition (Empirical Rademacher Complexity). For a sample $ (x1,...x.n) E x of points and a function class A of real valued functions, A X + R. the empirical Rademacher Complexity of A given S is
Since x( S 1 the Massart's lemma implies that for sample S, have the following
Rs(((((f, (1), U(f, mm 1) H3) S
2Vlog(em/VV (17)
1 Eg
VIL
m
- Rs
sup AEA
Axx)
(13)
Thus combining Eq 14, 15, 16 and 17 above, for the loss we have with probability 1 5/2,, Vʃ H
m
where a; are independent random variables drawn from the Rademacher distribution, ie, Pr(a, +1) Pr(a; = 1))
Ex-p Ex=pl(( (f,
Σ ((f SCS
Vlog(emm/ (V)
21og (4/8)
2
(18)
Proof of Theorem 2. We begin using the equivalent bounds in PAC learning by (Shalev-Shwartz and Ben-David, 2014) and using the proof sketch of (Fish and Reyzin, 2017).
m
m
Repeating the argument, we also have with proba- bility 8/2, Vf
First, consider the loss function i(f,x) (f(x)
Σ (ff.z) Ex-uu((fxx) 1
f This is function with [[(f < 1, thus a generalization bound can be derived using the empirical Rademacher Complexity Rs of the class
FES
2log (4/5) 2Vlog +
2
(19)
H, stating that with probability at least V
m
m

---

Thus using the union bound we obtain with prob- ability 6, Vf II
1 Σ =
E Ex [[f
m
2Vlog(em/V)
(4/g)
2/
(20)
m
m
Using Jensen's s inequality and for any > 0 we also have the following.
Ex ((f, x)] 2 (ps - -Pfo)2
1 x (lps Pfol + 111 - py) - (1- P5o pf))
2/a
=
2
(21)
21122/a Ctv. (pf. P(o)
(22)
Similarly, we also obtain
Σ (f.xz) 2 (ps o
1 2 (103 15 +1(1 P5) (1
2/a
(5,))
(23)
2/12 Cīv.(0f Pio)
(24)
Under a mild monotonicity condition, we can com- bine Eq 20, 22 and 24 to obtain the desired bound of Eq 4
S Civa(ps, Pfa)+ VVloglem/V>
2log (4/8)
(25)
m
771
where 22/a-1

---

Average
Dataset
Train
# Val
Test
# Words
Hyperpartisan
516
64
65
565
IMDB
22500
2500
25000
231
Rotten Tomatoes
7464
1068
2130
21
Medical QP
2134
305
609
40
Fin. Phrasebank
3394
486
966
23
Table Statistics of the datasets. The first columns represent total number of instances. Bag sizes in training and validation are provided in Table 6
Model
Dataset
Hyperpartisan
IMDB
Rotten Tomatoes
E
LR
X
a
B
E
LR
B
LR
X
a
Longformer
16
20
5e
1.0
1.0
32
10
-3
0.5
32
10
2e
1.0
0.33
RoBERTa
16
10
3e
3
1.0
2.5
32
10
3e
0,0
1.0
32
10
2e
1.0
0.33
BERT
20
5e
5
1.0
1.0
1e
2c
1e
0.33
DistiBBERT
16
10
55
1.0
1.0
32
10
3e
0.0
4.0
32
10
2e-
16"
0.33
Table Hyperparameter Details for the different configurations. The notations represent following: size of bag. E number of epochs in training, LR learning rate, À regularization parameter in the final formulation Eq 6, a parameter in loss Civ, in Eq 3.
Model
Dataset
Medical QP
Financial Phraschank
B
E
LR
B
E
LR
X
Longformer
16
10
3e-5
1.0
1.0
16
10
2e
1e-2
2.5
RoBERTa
16
10
3e-a
5.0
10
2
1e
2.0
BERT
16
10
3e
1e-2
0.5
16
10
-3
2c
0.1
1.0
DistilBERT
3e-3
0.33
10
3e
-3
1c
Table 7: Hyperparameter Details for Medical Questions Pairs and Financial Phrasebank datasets.


Reviews:
[{"review_id": "ugrJhuJuFN", "reviewer": "Reviewer_qeAx", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and suggestions. We address the questions and concerns in order below:\n\n**Q1**: Regarding the evaluation metrics -  \n**A1**: We agree with your comment and will provide a detailed description of the metrics in the updated manuscript.\n\n**Q2**: Use of decoder models -  \n**A2**: Since the problem setting is mostly concerned with classification, we used the encoder family as most of the classification tasks are performed using the final layer embeddings of the encoder component. These are then typically passed through a linear layer and subsequently into the loss function.  \nHowever, we have taken your comment into note and are looking into the literature for any relevant decoder model that can be leveraged (we will update the manuscript accordingly).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I've found the LLP setting quite interesting, especially for data privacy applications. The paper has also adequately supported the newly proposed loss function based on the total variation distance, from both theoretical and empirical perspectives. Overall, it is a good quality paper. ", "reasons_to_reject": "I don't have any major reasons for rejection. However, I have a few comments:\n\n- While the evaluation metrics (weighted precision, recall and F1) follow prior works, to be self-contained, it would be beneficial to precisely define these metrics, especially considering (a) how important evaluation is and (b) different versions of these weighted metrics exist.\n\n- The base models used in the experiments predominantly belong to the BERT family. Apart from the encoder models, have you considered using decoder models as the base models?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a robust objective function for the learning from label proportions (LLP) setting. Unlike the traditional KL divergence utilized in the DLLP work by Ardehaly and Culotta (2017), the new objective function applies the total variation distance, which is a tight lower bound of the KL. The paper has provided theoretical results, based on structural risk minimization framework, and empirical results,  on a number of text classification datasets, to support the proposed new formulation.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "oxXgVzKDaY", "reviewer": "Reviewer_g6p6", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and suggestions. We address the questions and concerns in order below:\n\n**Q1**: Unclear motivation -  \n**A1**: We understand the concern with the writeup and will update in the subsequent version of the manuscript. We would like to clarify both the points in order:  \n    **Privacy**: Since there are no existing datasets that follow the LLP setup in the NLP domain, we have to manually simulate this by aggregating ground truth labels during training. We also highlight that the background literature works - DLLP and LLPSelfCLR both operate similarly. \nIn real world scenarios, we typically have access to the aggregated data beforehand. A real world example of this is - when firms like twitter and google gather data from e-commerce platforms like amazon and shopify (oracle in this case), the latter already provide aggregated statistics to the former. Thus the former has to begin training their models using the aggregated data from the get go.\n\n**Weak Supervision**: Here, we have used the broad definition of weak supervision as discussed by Nandy et al. 2022. This helps us remain coherent with the literature. Under this broad definition, we refer to a task which does not provide ground truths at the instance level but rather noisy (as you have pointed out) or aggregated (in this paper) labels, as weak supervision.\nWe have discussed this in the introduction, line 56, of the paper.\n\n**Q2**: Nothing NLP-specific in the approach -  \n**A2**: As you have already described, we provide one of the preliminary works discussing the application of LLP to NLP tasks. The aim here is to bring the attention of the community to this highly important problem. Furthermore, due to this reason, the baselines used in our comparison are more general purpose. Going forward, we hope to incorporate more NLP specific paradigms into LLP.\n\n**Q3**: Not all theoretical groundings seem to be relevant -   \n**A3**: We note that the section describing irregularities of $L_{DLLP}$ is what primarily motivated our novel loss function formulation. Thus, we believe that adding some theoretical characterization of these properties along with the main result in theorem 2 leads to a more complete work. Furthermore, properties such as #2 (robustness) and #5 (lipschitz smoothness) have high significance in machine learning research, which our formulation exhibits.\n\n**Q4**: Discussion on Section 3.3 -  \n**A4**: We wish to clarify that Section 3.3 solely focuses on the theoretical result. For ease of discussion and succinctness in the theory part, we provide theorem 2 in the light of binary classifiers, but is well extendable to multi-class settings.  \nThe main experiments do contain both binary and multi-class (with more than 2 labels) settings for the completeness of practical research purposes.\n\n**Q5**: Unclear experimental setting -  \n**A5**: We note that the main results in Tables 1 and 2 focus on the comparison between our loss formulation and the baselines. This is to exhibit that the proposed loss function is better suited for the LLP setting across a broad range of configurations. To demonstrate this, we selected a diverse suite of models and datasets. Thus each model-dataset combination can be observed on a standalone basis and it is noted that our formulation performs better in most of the cases. The discussion regarding dataset selection follows from the response of **Q1** and **Q2** above.\n\n**Q6**: No significance testing is done -   \n**A6**: Since we found that the results across all the configurations are fairly stable and reproducible, we provide the mean value of the numbers. This is inline with the literature (corresponding to fine-tuning of large scale pretrained models) where the variance in results are typically of no significance, as observed  across various types of datasets.\n\nWe will adjust the presentation style and correct the typos as per your suggestion in the updated manuscript. We appreciate your efforts in pointing out things in such detail.\n\nIf the response satisfies your concerns, we hope that you will improve the score to reflect the same. We are happy to answer any more questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The application of LLM to natural language tasks is new and of some interest.\n- The proposed method is theoretically justified; properties of the method are discussed.\n- Interesting ablation study and result analysis.\n", "reasons_to_reject": "- The paper is hard to read and somewhat difficult to follow.  \n- The motivation is unclear. The authors argue that the LLP setup is relevant for (1) privacy and (2) weak supervision. (1) Privacy: the authors claim that the LLP paradigm is relevant for training on sensitive data as the labels for such datasets are not publicly available. However, the setting proposed in this paper does require gold (and publicly available) labels to formulate the ground truth proportion. If this proportion can be formulated without gold labels, it should be discussed. (2) Weak Supervision: in lines 136-137, the authors mention that the associated label proportions \"...provides the weak supervision for training the model\". However, weak supervision is a paradigm in which data is automatically labeled with noisy labels using some heuristics and labeling functions. It remains unclear to me in what way this setting is related to the proportion parameter authors use in their work. \n- The authors claim it to be one of the preliminary works discussing the application of LLP to NLP tasks. However, I don't see anything NLP-specific in their approach. \n- Not all theoretical groundings seem to be relevant to the main topic (e.g., some of the L_dppl irregularities). Additional clarification of their relevance is needed. \n- Section 3.3 says the results are provided for binary classifiers only, and the multi-class setting remains for future work. However, one of the datasets used for experiments is multi-label. \n- The experimental setting is unclear: does Table 1 contain the test results of the best model? If so, how was the best model selected (given that there is no validation set)? Also, if the proposed method is of special relevance to the sensitive data, why not select a sensitive dataset to demonstrate the method's performance on it? Or LLP data?\n- The authors claim the results to be significant. However, no results of significance testing are provided.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper investigates the application of the Learning form Label Proportion (LLP) to natural language. In particular, the authors present a new training objective which is a weighted combination of a modified version of total variation distance and self-supervised contrastive loss. The paper presents some theoretical justification and an extensive parameter analysis. The effectiveness of the method is demonstrated by experiments on five well-known datasets and four models. \n", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The font of Figure 1 and Table 1 is too small and hard to read. \n- Tables 5 - 7 are listed in Appendix, but there is no reference to the Appendix when referring to them.\n- Superfluous brackets in some citations (lines 35-36, 71, 79, 161, 556, 558, 559, ...).\n- lines 213, 493: the word \"Appendix\" is missing a capital letter.\n- lines 481, 483, 497: missing full stop after \"Eq\"."}}, {"review_id": "AcMp4I3AKo", "reviewer": "Reviewer_i5AH", "summary": "", "questions": "Please give your comments on \"Reasons to Reject\"\n\nA. Motivation for the approach - how to justify your loss function if the mentioned properties are not needed for a good performance?\n\nB Do you think that the results on bag size of 16/32 give a good estimation of how your method would work in practice?\n\nC Please give the lacking details of experimental setup mentioned above.\n ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and suggestions. We address the questions and concerns in order below:\n\n\n**Q1**: Motivation for the approach -  \n**A1**: We first note that these claims are implicitly supported by the results across the tables and figures in the paper. Second and more importantly, the boundedness property helps bridge the gap between the theory and the experiments. A substantial chunk of work in the learnability literature requires bounded losses (and focus on losses such as “0-1 loss”) and similar literature for “cross-entropy” type losses in not deep. This claim is highlighted in line 932 of the appendix in the proof as well. Similarly, the robustness property holds key importance in the light of Lipschitzness. The popular book of [1] provides a thorough discussion of these topics in more detail.\n\n**Q2**: Discussion of the bag size and shuffling of data at each epoch -  \n**A2**: We wish to highlight that the primary batch sizes used during tuning of large NLP models typically follow the range between 16 and 32. Since we have followed the protocol to fine-tune the pretrained NLP models, we adhere to these conditions. Furthermore, the experiments in the related LLP literature are focused on CTR and computer vision datasets, wherein it is common to train with a batch size of around 1000. Thus, we think our experimental evaluation is very practical and inline with the large NLP models being deployed in production.\n\nWe discuss the practicality of shuffling at each epoch at line 364 under the “Training” part of Section 4.1. More succinctly, since it is highly likely that each instance could be shared across multiple bags, randomly constructing new bags at each epoch does simulate the real-world setup to an extent. \n\n**Q3**: More details of the experimental setup and noise induction -  \n**A3**: The details on how we perform cross-validation and tune the hyperparameters are described on line 382 onwards, under the “Training” part of Section 4.1. The range of values considered for $\\lambda$ are between $1e^{-7}$ and $1$, with an increment factor of $10$ ($0$ is also considered in a couple of cases as seen from Table 6). For $\\alpha$, the corresponding range is between $0.25$ to $5$.  \n\nWhile your suggestion of inducing some noise in the true proportions is indeed a very practical use case, we follow the current setup as an introduction of the problem statement to the NLP community. Furthermore, noise induction also requires much more careful analysis on the amount of noise induced and the corresponding theoretical properties can be harder to analyse in some cases. We will follow up on this suggestion in the future work.\n\n**Q4**: Motivation for LLP -   \n**A4**: We appreciate your effort in providing these thoughtfully crafted points. We will first discuss the privacy aspect, followed by the weak supervision:\n\n**Privacy**: First we note that while the data is provided in an aggregated manner, the final aim is to learn an instance level classifier. Thus, it is important to learn a “good enough” classifier (the quantification of “good enough” is a slightly non-trivial matter to resolve here). The recent work of [2] provides very interesting theoretical characterizations of LLP setup. Furthermore, by storing the data in the aggregated manner, one cannot directly infer the truth about a given instance. It is only when we have a well-trained model, we can seek to approximately infer which class a particular instance belongs to. This further makes it safer to transport (transfer) data via a public medium in various scenarios.\n\n**Weak Supervision**: While it may be harder to find such datasets for academic research, we note that such data is available in abundance with tech corporations. In fact, as noted in Nandy et al., 2022, an important use case of LLP arises in the “click through rate” prediction in the advertisement industry, where directly sharing the user level data might be a violation of privacy laws, but sharing on an aggregate level doesn’t directly violate any such constraint. Similar examples where the data belongs to medical and other domains follow.\n\n**Q5**: Other typos and presentation adjustments -  \n**A5**: We will correct all the typos and clarify the missing details in the updated manuscript. We appreciate your efforts in pointing out things in such detail.\n\nRegarding the SSC objective in Nandy et al., 2022, we note that their equation 3 is inline and similar to equation 5 in our paper.\nRest of your suggestions will be accounted for adequately. \n\n\n[1] Shalev-Shwartz and Ben-David : Understanding Machine Learning, From theory to Algorithms (2014)\n\n[2] Learning from Aggregated Data: Curated Bags versus Random Bags - Lin Chen, Thomas Fu, Amin Karbasi, Vahab Mirrokni; (2023)\n\nIf the response satisfies your concerns, we hope that you will improve the score to reflect the same. We are happy to answer any more questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- new proposal of a loss function for LLP with a theoretical background\n- good experimental results, simple ablation study\n- relatively clearly written", "reasons_to_reject": "1. Motivation for the approach\n\nThe authors claim in Section 3 that previously used LLP loss functions lack some properties that can lead to \"sub-optimal parameter values post training\". The first lacking property is that the loss functions are \"not upper bounded\". However, the cross-entropy loss function, which is commonly used in deep learning, is also not upper bounded and it does not seem to be a problem in practice. Cross-entropy loss is also not symmetric and not robust (in the sense of the authors). Therefore, the claim that these properties are really needed for more optimal training deserves some citation and discussion.\n\nThe authors also claim that their formulation retains \"outlier robustness\", but this claim is not verified in the experiments.\n\n2. Experiments\n\nMy main concern is the practicality of the experimental setups with small bag sizes such as 16/32. The related work tests much larger bag sizes, e.g. (Nandy et al., 2022) has bags with an average of >500 examples, (Liu et al., 2021) uses 128. In addition, the authors evaluate their algorithms on a simpler scenario where the examples change their bag assignment in each evaluation (shuffling the data). I assume that in most practical NLP scenarios the bags would be static.\n\nSome details of the experimental design are missing. For example, are the reported results calculated using cross-validation? Are they averaged over several repetitions? How many? The authors have tuned the lambda and alpha parameters of their method, which seem to be important for the final result (Fig. b,c) - how exactly were they chosen, e.g. what is the range of possible values considered?\n\nThe authors evaluate LLP on an ideal case where the proportions from the training data sets are the true proportions and do not consider a more practical variant where there is some noise in the true proportions.\n\n", "questions_to_authors": "Please give your comments on \"Reasons to Reject\"\n\nA. Motivation for the approach - how to justify your loss function if the mentioned properties are not needed for a good performance?\n\nB Do you think that the results on bag size of 16/32 give a good estimation of how your method would work in practice?\n\nC Please give the lacking details of experimental setup mentioned above.\n ", "paper_topic_and_main_contributions": "The paper proposes a new loss function for training deep learning classifiers in the Learning under Label Proportions (LLP) setup. The proposal is accompanied by a PAC learnability theorem. The paper presents experimental evaluation and comparison against two related approaches, demonstrating significant improvement on 5 text classification dataset.\n\nI would like to note that except for the evaluation on text classification datasets, the paper presents a general deep learning result that is not specific to NLP.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Motivation for LLP\n\nThe authors motivated their work on LLP by providing privacy and working with weak supervision. I found both of these motivations a little unconvincing.\n\na) Privacy. While storing user data without explicit labels but rather in groups with assigned probabilities seems to be privacy preserving, the authors actually show that it is possible to infer from such data a classifier that is able to assign labels with high accuracy. Therefore, the ability to infer labels using LLP violates privacy. In fact, any progress in improving the performance of LLP reduces the privacy of such data.\n\nb) Weak supervision. The NLP models rely on self-supervised learning because unsupervised corporas can be easily scraped from the web. Is large-scale data easily available for LLP? I don't think so, especially since the authors don't use LLP datasets in their experiments and rely on standard supervised datasets. \n\n2. Writing\n - The definition of L_SSC (Eq 5.) is not clear. The authors suggest in the text that this is the SSC objective of (Nandy et al., 2022) but in the original work a different similarity function is used than cosine similarity from the paper. Therefore, it is not clear if this modification is intended. If intended, the choice of different similarity function should be discussed. \n- \"This result is thus inline with the claims presented for our formulation in Section 3.\" - I don't see claims in Sec 3 with respect to bigger bag sizes.\n- The authors mention \"k-anonymity\" in the Introduction but it is not defined nor cited.\n- Definition 1 (Lipschitz Continuity) suddenly appears in the text without any introduction\n- In Tab 3 \"Overall\" is confusing = it is better to use the name of the approach\n- The authors state \"Following prior works, we use the weighted versions of [measures]\" but do not mention which prior works they are following. \n- \"Figure 2(a) plots the comparison (..) These results align with the findings in the literature\" = I assume that the results of your method wasn't already discussed in the literature."}}]

Author rebuttal:

Meta-review: In words of one of the reviews:
"The paper investigates the application of the Learning form Label Proportion (LLP) to natural language. In particular, the authors present a new training objective which is a weighted combination of a modified version of total variation distance and self-supervised contrastive loss. The paper presents some theoretical justification and an extensive parameter analysis. The effectiveness of the method is demonstrated by experiments on five well-known datasets and four models."

A clear strength of this paper is the out-of-the-trodden path setting which opens up interesting new research questions, as well as - arguably (see below) practical applications. The contributions of this paper are both theoretical as well as empirical.

The main criticisms are about the practicality of the setting - having proportion of levels but not fine-grained ones. While this seems like a reasonable situation, it would be arguably more compelling if a real use-case could be mentioned (if not used). There are although concerns about the clarity of the writing - the authors are encouraged to take those comments to heart.
Decision: Accept-Findings