ULF: UNSUPERVISED LABELING FUNCTION CORRECTION USING CROSS-VALIDATION FOR WEAK SUPERVISION
Authors: Anastasiia Sedova
Benjamin Roth
Venue: Submitted to ICLR 2023 (2023)
Keywords: nlp
weak supervision
text classification
sentiment analysis

Abstract: A way to overcome expensive and time-consuming manual data labeling is weak supervision - automatic annotation of data samples via a predefined set of labeling functions (LFs), rule-based mechanisms that generate artificial labels for the classes associated with the LFs. In this work, we investigate noise reduction techniques for weak supervision based on the principle of k-fold cross-validation. We introduce a new algorithm ULF for denoising weakly annotated data which uses models trained on all but some LFs to detect and correct biases specific to the held-out LFs. Specifically, ULF refines the allocation of LFs to classes by re-estimating this assignment on highly reliable cross-validated samples. We realize two variants of this algorithm: feature-based ULF (relying on count-based feature vectors), and DeepULF (fine-tuning pre-trained language models). We compare ULF to methods originally developed for detecting erroneous samples in manually annotated data, as well as to our extensions of such methods to the weakly supervised setting. Our new weak supervision-specific methods (ULF and extensions) leverage the information about matching LFs, making detecting noisy samples more accurate. Evaluation on several datasets shows that ULF can successfully improve weakly supervised learning without utilizing any manually labeled data.

Under review as a conference paper at ICLR 2023
ULF: UNSUPERVISED LABELING FUNCTION CORRECTION USING CROSS- VALIDATION FOR WEAK SUPERVISION
Anonymous authors Paper under double-blind review
ABSTRACT
A way to overcome expensive and time-consuming manual data labeling is weak supervision automatic annotation of data samples via a predefined set of label- ing functions (LFs), rule-based mechanisms that generate artificial labels for the classes associated with the LFs. In this work, we investigate noise reduction tech- niques for weak supervision based on the principle of k-fold cross-validation. We introduce a new algorithm ULF for denoising weakly annotated data which uses models trained on all but some LFs to detect and correct biases specific to the held-out LFs. Specifically, ULF refines the allocation of LFs to classes by re- estimating this assignment two variants of this algorithm: naature-bsse ULF (relying on count-based feature vectors), and DeepULF (fine-tuning pre-trained language models). We compare ULF to methods originally developed for detecting erroneous samples in manually annotated data, as well to our extensions of such methods to the weakly super- vised setting. Our new weak supervision-specte specific methods (ULF and extensions) leverage the information about matching LFs, making detecting noisy samples more accurate. Evaluation several datasets shows that ULF can successfully improve weakly supervised learning without utilizing any manually labeled data.
highly reliable cross-validated samples. We realize
INTRODUCTION
large part of today's machine learning success rests upon a vast amount of annotated training data. However, manual expert annotation turns out to be tedious and expensive work. There differ- ent approaches to reduce this data bottleneck: fine- tine-tuning large pre-trained models (Devlin et al.) 2019), applying active learning (Sun & Grishman 2012) and semi-supervised learning (Kozareval et al. 2008). However, even if in a reduced amount, these approaches still demand manually anno- tated data. Moreover, constant data re-annotation would be necessary in settings with dynamically changing task specifications or changing data distributions.
Another strategy that does not require any manual data labeling is weak supervision (WS), which allows to obtain massive amounts of labeled training data at a low cost. In weakly supervised setting, the data labeled in an automated process using one or multiple weak supervision sources, such as external knowledge bases (Lin et al. 2016: Mintz et al., 2009) and manually-defined or au- tomatically generated heuristics (Varma & Ré 2018). By applying such rules, or labeling functions (LFss, Ratner et al. 2020). to large unlabeled dataset, one can quickly obtain weak training labels, which are, however, potentially error-prone and need additional denoising (see examples Fig. D].
In this work, we explore methods for improving the quality of weak labels using methods based on the principle of k-fold cross-validation. Intuitively, if some part of the data is left out during training. the model does not overfit on errors specific to that part. Therefore, mismatch between predictions of a model (trained on a large portion of the data set) and labels (of the held-out portion) can indicate candidates of noise specific to the held-out portion.
This idea has motivated different approaches to data cleaning (Northcutt et al., 2021 Wang et al.
2019ch. As they deal with general, non-weakly supervised data, they usually split the data samples into folds randomly and independently on the sample level. However, a direct application these methods weakly labeled data ignores valuable knowledge stemming from the weak supervision

---

Under review as a conference paper at ICLR 2023
process (e.g., which LFss matched in each sample what class each LF corresponds to). In this work, we leverage this additional source of knowledge by splitting the data considering the LFs matched in the samples. We build the intuition that mismatch between predictions of model (trained with a large portion of the LFs) and labels (generated by held-out L.F)) can indicate candidates of noise specific the held-out L.s.. If this LF-specific cross-validation is done for each LF, noise associated with all LFs could be found and corrected. This idea is realized in our extensions to methods proposed in Northcutt al. 202T and Wang al. 2019cc
Apart from that, we use the prin- ciple of weakly supervised cross- validation to break out of the logic
Matched labeling functions
Assigned Tabel
of just repairing the labels, to in- stead repair the L.-to-class assign- ment. This approach is formal- ized in ULF our new method for Unsupervised Labeling Function cor- rection with k-fold cross-vale volsstaali. Its primary goal is to improve the LFss to classes allocation in order to cor- rect the systematically biased label assignments. ULF re-estimates the joint distribution between LFs and class labels during cross-validation based on highly confident class pre- dictions and their co-occurrence with matching LFs. The improved alloca- tion allows to re-assign the weak la- bels for further training. Importantly, ULF also improves labels of the sam- ples with LFs matched, in contrast to other methods that filter them out (Ratner et al.) 2020
(1) if your like drones, pla subscribe to Kamal Tayara, He takes videos with his drone that are absolutely beautiful.
keyword ke subsccrbbb" keyword("ple"
SPAM
(2) It looks so real and my daughter is a big fan and she likes a lot of your songs.
keyword ("my"" keyword ("word("songg
SPAM/ HAM
(3) Follow me on Twitter @mscalifornia95
no metches
Examples of weakly supervised annotation from YouTTub dataset. In (1), both matched LFss correspond to the SPAM class; the sample is therefore assigned to SPAM class. In (2), there is a conflict as one of the matched LFs belongs to the SPAM class, while the other to the HAM class. In (3), no LFs matched, meaning the sample does not get any weak signal.
Figure I:
Overall, our main contributions are: (1) A new method ULF for improving the LFs to classes allocation in an unsupervised fashion. ULF not only detects inconsistent predictions but corrects the process that led to them by re-estimating the assignment of LFs to classes. Training with ULF results in more accurate labels and a better quality of the trained classifier. (2) Two implementations of ULF: feature-based ULF for feature-based learning (without hidden layer), and DeepULF for fine-tuning pre-trained language models. (3) Extensions of two methods for denoising the data using the principle of k-fold cross- cross-validation (Wang et al., 2019c Northcutt et al., (021). Our extensions Weakly Supervised Cross Weigh and Weakly Supervised Cleanlab profit from the WS- specific information and make the denoising of WS data more accurate. (4) Extensive experiments on several weakly supervised datasets order to demonstrate the effectiveness of our methods. To the best of our knowledge, we are the first (1) to adapt k-fold cross- cross-validend validation basod noise detection methods to WS problems, and (2) to refine the LFs to classes allocation in the WS setting.
RELATED WORK
Weak supervision has been widely applied to different tasks in various domains, such as text classi- fication (Ren et al., 2020 Shu et al. 2020), relation extraction (Yuan et al.) 2019: Hoffmann et al.) 2011) named entity recognition (Lan et al., 2020) Wang et al., 2019c). video analysis (Fang et al. 2020 Kundu et al. 2019), medical domain (Fries et al. (021). image classification (Li et al. [202]] and others. Weak labels are usually cheap and casy to obtain, but also potentially error- error-prone, and thus often need additional denoising.
Denoising methods. Among the most popular approaches to improving weakly supervised data is building a specific model architecture or reformulating the loss functions (Karamanolakis et al.) 2021 Hedderich & Klakow] 20018) Goldberger & Ben-Reuven 2017) (Sukhbaatar et al.] [0144) Sometimes, weak labels are combined with additional expert annotations: for example, by adding
2

---

Under review as a conference paper at ICLR 2023
ULF
update
mabrii
US
Crimes comppendrnc
Clean intels
Noky labels
9
C-O
Unteteled camples
Input
Figure 2: ULF: Unsupervised Labeling Function Correction
set of manually annotated data to the weakly labeled one (Mazzetto al. 2021 Karamanolakis Awasthi et al. 2020), or learning from manual user corrections or guidance (Hedderich
eet al.
2021
al. 2021 Chatterjee et al. 2020 Boecking al.. 2021 Lange et al.. (011)) The methods we intro- duce in this paper, on the contrary. do not require any manual supervision and can be used with any classifier. Other approaches aim finding and removing or downweighting erroneous samples by estimating the relation between noisy and clean labels (Wang et al., 2019a Northcutt et al. 2021). Rather than discarding potentially erroneous samples, our method ULF aims label correction, utilizing as much of the weakly supervised data as possible.
Cross-Validatinn Validation. There are several methods that share the idea of using k-fold cross-validation for denoising manually labeled data (Wang et al.) [2099] Northcuut et al.) Teljstett et al., 2015)) Being a stable and reliable method of trained model quality estimation (Wong Yehl 2019). k-fold cross-validation has proven to be fruitful method for detecting errors in data as well. Wang et al. denoiie crowdsoureing annotations by repeatedly training k models and downweighting the samples with disagreeing -ut-of-sample predicted and original labels. Northcutt al. [021 consider the confidence of the predicted class probabilities, identify the mislabeled samples using ranking and prune them. However, being used as-is for denoising the weakly supervised data, both of these methods miss a lot of potentially profitable WS-specific information.
Data Programming. We follow the data programming approach (Ratner al. 2016) which allows encoding the weak supervision sources in form of labeling functions (LFs) which could be viewed as annotators annotating data samples. Such annotations often result in conflicting erroneous labels, which are usually refined by applying more sophisticated LFs aggregation techniques (Ratner et al., 2019 Hoffmann al. (011]) and modifying the LFs (Chatterjee et al. 2019).
ULF: UNSUPERVISED LABELING FUNCTION CORRECTION
Inaccurate and erroneous allocation of LFs to classes often becomes a cause of noise in a weakly supervised setting. For example, in Figure I among the LFs used to annotate the YouTube Dataset, there is a LF "my" that is set to correspond to the SPAM class. The dataset creators (Alberto) al. 2015)) included this LF capture the often encountered spam messages like "subscribe my chanmel" or "follow my page". However, this correspondence is not that straightforward and may be potentially misguiding: it is not hard to come up with dozen non-spam messages containing the word "my". Thus, the weak labeling for Sample 2, where this LF matched alongside another one from class HAM (i.e., non-SPAM class), would lead to tie and, by random tie-breaking, potentially to an incorrect label assignment. However, if the class assignment of "my" were adjusted so that its association with the SPAM class is reduced, the overall HAM probability would dominate, and correct label would be assigned to this sample.
3.1 PRELIMINARIES
Given dataset X with N data samples, X = {11, I2, ] each sample is a sequence of words (i.e., one or several sentences). This set is used for training a classifier with output classes.
3

---

Under review as a conference paper at ICLR 2023
In the weakly supervised setting, there are no known-to-be -ob---cree--- la- bels for the training samples. In- stead, we are provided with a set of LFSs I,, L (1,,2,.-..1.) The LFs can express different weak su- pervision sources, such as knowledge bases, lists of keywords and other human-defined heuristics. We say that a LF 1 matches a sample if some condition formulated Ij holds for 3; (e.g. keyword or regular expression matches); then l maps z to label. For example, in the case of keyword-based LFs, a LF matches sample if the keyword is present in it and, thus, a corresponding label is assigned to this data sample. A set of LFs matched in a sample z; is de- noted by Lx., where [L, E [0, L]. This information can be saved for an entire data set a binary matrix ZNXL, L. where Zij = means that Ij matches in sample x,,
ULF
If
H2
ML
C1
0.8
0.2
0.5
0.3
0.6
Rclasses
Z. tsamplas RLFs
X1
0.9
0.1
orgmaax
1
Xg
1.2
V Yvody #sarmples *Aclasses
V. d.Asampless
Figure 3: Weak annotation encoded with Z and T matrices. Z contains the information about LFs matches in samples. T is matrix with LFs-to-class correspondence. Multiplica- tion of Z and T matrices and applying majority vote results in weak labels Ý. T is the matrix improved with ULF.
Each LF corresponds to some class k;; in other words, I, assigns the corresponding label to sam- ples where 1 matches]] The information about this correspondence is stored in binary matrix TEXK, where Tij means LF l corresponds to class j. The most straightforward way of ob- taining weak labels = {ih, 02, .... Inh. Új E K is to multiply Z and T, apply majority vote, and break the ties.
3.2 OVERVIEW
The main goal of the ULF algorithm is to refine the matrix of LF's to classes assignments T. The main steps are the following: first, the predictions (together with their confidences) are calculated with k-fold moss-validation. Second, these predictions used to build LFs-to-classs confident matrix CLXK. This matrix counts how many samples with a particular LF have highly confident predictions (exceeding threshold) for particular class. C' is then re-normalized to provide cross- validated re-estimation of T. The graphical illustration of the algorithm provided in Figure 3 the method is summarized in Algorithm
3.3 LABEL PROBABILITIES WITH CROSS- VALIDATION
Firstly, class probabilities for each training sample predicted by k-fold cross-validation. For that, use the training set X and weak labels Y obtained by multiplying Z and T matrices. There are three possible ways of splitting the data folds for cross-validation training:
randomly (ULF rnd)) the samples are assigned folds the same way it would be done in standard k-fold cross-validation irrespective of LFs matching;
by LF (ULFgg): the LFs randomly split into & folds {fi,... faJ and each fold fi is iteratively taken as held-out LFs, while others become training LFs. All samples where training LFs match become training samples, the rest used for re-estimation: Xtvain, = Ln N = 0}. Xont. =X Xtrain, (1).
* by signatures (ULF :g)): for each training sample we take the set of matching LFs Lz, as signature and build & folds from them, each of which iteratively be-
'The initial manual class assignment is typically such that a LF corresponds to exactly one class, for which it covers prototypical case. A manual assignment several classes would however he possible in principle and compatible with ULF.
4

---

Under review as a conference paper at ICLR 2023
comes held-out fold, while others used training folds: Xtrain, Xout, = Lz E f(} (2).
{rylless fi),
Secondly, & models are separately trained on Xtrain,, i [1, k] and applied on the held-out folds Xout, to obtain the out-of- predicted probabilities PNXK. From PN*K the out-of-sample labels are derived:
arg max p(ù j; 15JSK
(3)
Importantly, a sample probability p(ū = j; 3it 0) is considered only if it exceeds the class average threshold ty:
Σ Z,EX5-1
p = j: 0)11X=
This allows handling of the sample x, as belonging to the class j if and only if this sample confidently belongs to the corresponding class. If no probability exceeds the class thresholds for sample (e.g., belongs all classes with equally small probabilities), the sample disregarded in further calculations as unreliable.
3.4 RE-ESTIMATE LABELING FUNCTIONS
In order to refine the LFs to classes allocation, ULF re-estimates the joint distribution between matched LFs and predicted labels. For each IF1,, and each class ky. the number of samples is caleu- lated, which have the LF I, matched and were confidently assigned to the class ky. This information saved LFs-confident matrix CLLK that contains the counts of samples:
Ch (xa E X Vs ūj.4 E Tz
(5)
Next, the confident matrix is calibrated and normalized to QL to correspond to the data propor- tion the matrix:
Σ
Qu.0: = Ch.se*
Z
Us
Σ
Clm.83
(6)
m=1
m==1
K
K
where
Eier, Cr, = n, Σ jEK
Σ
1=1
The joint matrix Q can now be used for improving the LFs-to-class matrix T that contains the initial LFs to class allocations. T and Q are summed with multiplying coefficients and - p. where p [0, The value of p balances the initial manual label assignment with the unsupervised estimation and determines how much information from the estimated assignment matrix should be preserved in the refined matrix T
T" p (1--p)-(1-p)-
With the multiplication Z and the newly recalculated 7 matrices, an updated set of labels Y* is calculated. It can be used either for rerunning the denoising process or for training the end classifier.
3.5 BEATURE-BASED ULF VS DEEPULF
We propose two ways of realizing the unsupervised label correction procedure described above. The first variation of the ULF algorithm is feature-based ULF, which uses count-based features. The training steps (i.e., Alg. T lines 7 and 14) are realized with Algorithm where any model for the beature-based prediction can be used. The second one DeepULF fine-tunes pre-trained language models. The fine-tuning can be combined with additional contrastive self-learning step from the Cosine framework (Yu et al. 2021), which leverages unlabeled data and potentially improves the quality of the trained classifier. The training steps (Alg. [ lines 7 and 14) for DeepULF are summarized Algorithm 3 As signature- based data splitting was the best performing method in the experiments with feadure-based ULF, select for the DeepUL.F experiments.
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1: ULF: Unsupervised Labeling Function Correction for Weak Supervision unsupervised training data X, samples to LFs matrix XNXL. LFs to classes matrix TExK K
Algorithm 2: Train (8, X, Y):
Feature-based ULF
Input:
= AdamW(O, X, Y)
Output: Trained 0
Calculate noisy labels Y T
Algorithm 3: Train (8, X, Y): DeepULF; pretrained language model fine-tuning (op- tionally: with Cosine self-training)
for iter 1,2, do
Randomly split LFs L or signatures U into folds Jn,....k fk for 1, ..., k do
Xtab = fx E X 11.1>00
Build X Xtrain, Xout, sets (Eq.
Yiot = {ύ Y Vx Xiab]
Yerain,
[ Y Vz E Xtrain.)
# 1. fine-tune 9 with Xlab and Yiab
Train (0;, Xtrain, Ytraen.)
= AdamW(e, Xlab, Yist)
Calculate p(ū = j; 0:)
# 2. (optional) contrastive self-training of 0 with X
Get labels 5 (Eq. 3) with respect to the
Calculate pseudo labels Uped
thresholds 1 (Eq.
10
Calculate I. LFs-to-class confident matrix
for step = 1, 2, ntmpsttes do
Ct.y (Eq.5) 5)
Select confident samples
Calculate classification loss Le Upsa),,
Estimate a joint matrix Q1. (Eq. 6
I1
contrastive regularizer (0(0, 3psd),
12
Recalculate T matrix (Eq. 7
confidence regularizer R1 (0)
13
Calculate new noisy labels Y with T
L(00, Dpsa) Te + AR1 + R₁
Train (0, X, Y+)
A = AdamW(@. X)
Output: Trained 0
Output: Trained 0
Unlabeled instances. One of the challenges in weak supervision is the data samples where no LF matches; in other words, the samples without any weak signal. In some approaches, such samples are filtered out, while in others they are kept as belonging to a random or the other class. In feature- based ULF, such samples validation training. The number of samples to be used in cross-validation training is defined by hyper-parameter A. After each T"* matrix recalculation, their new labels are calculated directly from the out-of-samples predicted probabilities with respect to the class average threshold (see Eq. B and 4.. In DeepULF combined with Cosine such samples are leveraged during self-supervised learning.
initially assigned with random labels and partly involved in cross-
WSCROSSWEIGH AND WSCLEANLAB
In order to have suitable baselines, consider two other cross-validation-basel methods initially proposed to improve manual data annotations: CrossWeigh (Wang et al., 2019c). and Cleanlab (Northeutt et al. 2021). For a more detailed comparison, introduce extensions of these frame- works for the weakly supervised setting. In contrast to the original methods, our extensions leverage the LF information resulting in more accurate results.
Cross Weigh was proposed for tracing inconsistent labels in the crowdsourced annotations for named entity recognition. The algorithm traces the repeatedly mislabeled entities by training multiple mod- els in a cross-validation fashion on different data subsets, comparing the predictions on unseen data with its crowdsourced labels and reweighting the conflicting ones. Our Weakly Supervised Cross- Weigh (WSCW) splits the data with respect to the LFs (by Eq. I and trains the i separate models on Xtrein.- The labels ý predicted by the trained model for the samples in Xout, are then compared to the initial noisy labels j. All samples X, where ú; # ūj are claimed to be potentially mislabeled; their influence is reduced in further training. The whole procedure of errors detection is performed t times with different partitions to refine the results. More details are provided in Appendix A
The Cleanlab framework allows to find erroneous labels by estimating the joint distribution between the noisy labels and out-of-sample labels calculated by k-fold cross-validation. In our extension Weakly Supervised Cleanlab (WSCL), we follow a similar approach, but adapt it to the weak supervision the same way in WSCW: the cross-ve voliiatiin sets Xtroing and Xout, sets are built considering the matched LFs by Eq. (WSCL yd) or Eq. (WSCL ). These sets are used to train l models, which predict the class probabilities for the held-out samples, which are later transformed

---

Under review as a conference paper at ICLR 2023
YouTube (Acc)
Spouse (F1)
TREC (Ace)
SMS (F1)
Avg
Manually Supervised
Gold
94.0 + 0.0
81.9 + 0.5
91.6 + 0.2
89.3
Weakly Supervised
Majority Vote
86.8 + 0.5
41.2 + 0.2
55.9 0.1
80.0 + 0.2
66.0
Snorkel-DP (Ratner et al.
2020
88.0 + 0.1
41.5 + 0.7
42.2 + 2.2
82.8 + 0.1
63.6
Denoising
Cross Weigh (Wang et al.
2019c)
90.1 + 0.6
41.6+0..
40. 0.1
79.6 + 1.4
62.8
Cleanlab (Northcutt al.
2021
78.0 + 0.2
5..5 4.11
79.5 0.2
65.3
CL-PyTorch
86.9 + 0.7
44.4 1.2
0.3
86.0 + 0.6
68.3
WS Extentions (Ours)
WSCW
90.8 + 0.7
42.0
46.0
0,9
65.7
WSCL !s
87.2 + 0.4
4.74
58.9
86.6
69.3
WSCL agn
88.3 + 0.5
44.6 0.9
5..4
85.1 + 0.3
68.6
ULF (Ours)
ULF mdm
92.8 0.1
58. 0.5
85.7 0.5
70.3
ULF I/
90.8 + 0.9
0.9
55.5
70.0 = 0.4
65.1
ULF
94.6 + 0.2
49.8 + 1.0
58.2 0.2
88.6 + 0.3
72.8
Table I: Feature-based ULF results. The best results are marked bold, the second best underlined. results are averaged over 10 trials and reported with the standard error of the mean.
to the exact labels ý with respect to the class expected value. The correspondence between noisy and out-of-sample predicted labels determines the number of samples to be pruned (the same way as in the default setting of Northcutt et al. 2021). Notably, Cleanlab counts the samples with presumably incorrect noisy labels only, but does not provide any estimate about what LFs assigned these noisy labels (in contrast to ULF). More details are provided Appendix B
5 EXPERIMENTS
In order demonstrate the effectiveness of our approach, we conducted several experiments of ULF and DeepULF on different datasets and tasks. The results are summarized in Tables and implementation details provided in Appendix D
Datasets. We evaluate feature-based ULF on four weakly supervised English datasets: (1) YouTube
Spam Classification dataset (Alberto et al. 2015); (2) Spouse Relation Classification dataset (Corney
et al. 2016); (3) Question Classification dataset from TREC-6 (Li & Roth 2002): (4) SMS Spam Classification dataset (Almeida al.) 2011) For testing the DeepULF, we also added
topic
classification datasets for African languages (5) Yorúbá and (6) Hausa (Hedderich et al.) (020). We use the same LFs and evaluation metrics in previous works for fair comparison. For all datasets except Spouse, the gold manual labels are available but used only for Gold baseline. The datasets and LFs descriptions as well as examples of LFs are provided in Appendix C
Baselines. Feature-basedd ULF is compared against five baselines. (1) Gold: the classifier is trained with gold labels. As this is the only setting where manually obtained gold labels are used, serves as an upper-bound baseline. (2) Majority Vote: the classifier is trained on the data and noisy labels ac- quired with simple majority voting and randomly broken ties. (3) Snorkel-Dr a classifier trained
using both generative and discriminative Snorkel steps (Ratner et al.) 2020), (4) CrossWeige (Wang) al. 2019c), (5a) Cleanlab: the original implementation of Northcutt al. 2021 with Scikit-learn our PyTorch-based reimplementation of the Cleanlab algorithm to provide a fair comparison with our methods implemented with the PoTch library. DeepULF is compared towards seven basclines based fine-tuning of pre-trained language model. In all ex- periments, the pre-trained RoBERTa model (Liu et al. 2019) is used for the English datasets, and
library. (5b) Cleanlab-PyTone
Thee code will be publicly released acceptance.

---

Under review as a conference paper at ICLR 2023
the pre-trained multilingual BERT (Devlin et al. 2018 for Yorùbá and Hausa following Hed- derich et al. 2020, (1) Cold, (2) Majority Vote, and (3) Snorkel-DP: the same baselines as for the ceature-based ULF, but with pre-trained language model as the end model, (4) FS RoBERTa: Flying Squid (Fu et al. 2020) is used as the label model and a pre-trained language model as the end model, (5) Majority Vote Cosine, 6) Snorkel-DP Cosine, and (7) FS Cosine: the same the baselines (2), (3) and (4), but applying the Cosine end model.
Feature-based Experiments. Table I reports the results of feature-basee learning using logistic regression classifier and TF-IDF data representations averaged over ten trials together with the stan- dard error of the mean. Our WSCW and WSCL extensions outperform the corresponding base methods in most cases and lead to consistent improvements com- pared the Majority and Snorkel-DP baselines. These results support our initial hypothesis of L.F' importance in applying cross-validation techniques for weak supervision. At the same time, the ULF algorithm shows the best result overall on most datasets and even outperforms the model trained on YouTube data with gold manual annotations. ULF sgn outperforms the classifier trained on the data with labels chosen by majority voting without any additional denoising by 6.8% on average and Snorkel-DP by 9.2%, mLF by 4.3% and 6.7% respectively. Interestingly, the ULFys demon- strates a worse result compared to ULF and even ULF That could be explained by the fact that more than one LF is matched in significant amounts of data samples in all datasets. As sult, these samples are included in several training and held-out folds (during the folds splitting in the vross-validation training according the matched LFs) and therefore are reestimated multiple times, while ideally each data sample should be considered only once in each denoising round.
CrossWeigt and Cleanlab frameworks
Language model-based experiments. DeepULF results provided in Table 2 form a grid search for energy considerations and resource constraints but random search with ten trials for each model type (DeepULF and baselines) with different hyperparameters. The search space and the best parameter settings are provided in Appendix [ We report the results for the model that was selected based on the performance on the development set For the baselines, we used the Wrench implementations (Zhang et al.) 202T). We evaluated DeepULF in four variants which are determined by the combinations of two additional binary typer-parameters: the first con- trols the model used for cross-validation (Alg. [ line 7), the second controls the end model (Alg. I line 14); in both cases, the options are either simple fine-tuning or followed by additional Cosine training. Results of all cross cross-validatio validation and end model combinations the best ones are included to Table DeepULF outperforms the baselines on five out of six datasets and shows the second-best result for Yorùbá data. Note that both feature-based ULF and DeepULF do not involve manually annotated data or manual correction and cannot be directly compared to the
We did not per-
provided in Appendix E
results reported for settings that include manual annotations (Karamanolakis al. 2021 Awasthi et al. 2020).
YouTube (Ace)
Spouse (FI)
TREC (Ace)
SMS (FI)
Yorùbá (FI)
Hausa (F1)
Avg
Manually Supervised
Gold
98.8
96.6
97.7
67.3
83.5
88.8
Weakly Supervised
Majority Vote
93.2
21.3
68.6
93.0
48.1
43.9
61.4
Snorkel-DP
95.6
32.6
61.8
94.6
58.7
45.7
64.8
FS (Fu et al. 2020
94.0
14.9
35,8
23.7
32,4
45.1
41.0
bosene (Yu et al. 2021)
Majority Vote Cosine
96.4
33.3
65.8
93.6
52.6
45.4
64.5
Snorkel-DP Cosine
96.0
28.1
73.8
96.1
55.0
46.5
65.9
FS + Cosine
95.6
24.9
38.6
90.1
33.3
41.5
54.0
DeepULF (Ours)
96.8
36.9
76.8
96.1
55.8
48.2
68.4
Table 2: DeepULF results. The best results are marked bold, the second best underline.
'Trained models will be publicly released on acceptance.
8

---

Under review as a conference paper at ICLR 2023
6
CASE STUDY
For the case study, we chose a couple of samples with the wrong weak labels (i.e., the annotation done by labeling functions was incorrect) and check their labels after the ULF training. These examples are provided in Figure
Sample is clearly SPAM comment. However, it is misclassified and assigned to the HAM class, because no LF corresponding to the SPAM class matched in it (this sample includes a spam keyword "subscribe": : however, as it is reduced to "sub", the corresponding LF did not match), but a LF short comment (labeling a sample as HAM if it is shorter than 5 words) did. Sample is also SPAM message not covered by any keyword-based LFs (as they are primarily concerned with cases where a user asks to subscribe to the channel, e.g. keywords "subscribe" " "my", while here there is a request to like the comment). Nevertheless, again, signal from the LF short_commer wrongly assigns this sample to the HAM class. Thus, the LF short comment consistently mislabels samples that, although short, should still be classified as SPAM. ULF detects this LF and adjusts its assignment: it now corresponds to the HAM class with only roughly a 40% probability, and not 100% as before; with the remaining 60% it corresponds to the SPAM class. As a result, after label recalculation, both of these samples obtain a correct label SPAM.
In sample 3, a word "subscribe" " (which is usually used in spam comments and, therefore, was leveraged to build a SPAM class LF "keyword subscrrbbe" is mentioned, but in context related to the video; This makes this message not-SPAM in contrast to the assigned label HAM in effect. In the same way, the keyword "password", which is defined as LF corresponding to the SPAM class in SMS dataset (in order to detect spam messages like "Send me your id and password") ") is matched in a HAM sample 4 and annotates it with SPAM label. Again, ULF detects these noisy LFs and changes their assignment, which results in a change of the sample labels.
The samples where two LFs from different classes match can be assigned to either of the two classes with a 50% probability. In sample 5, the tie was broken randomly in favor of a wrong SPAM label. After the ULF reassignments, the LF keyword was related to SPAM class with 70% instead of 100%. Although the assignment of LF keyword Jhhnnk also changed slightly (90% to the HAM class and 10% to the SPAM class), overall the HAM class outweighed, and the sample label was changed to the correct one.
Sample
LPss matched
Noley Label
Corrected Label
True Label
sub my channel for no reason-
short comment
HAM
SPAM
SPAM
2 -):):) Like This Comment s):}:)
short comment
HAM
SPAM
SPAM
3 OMG LOVE YOU KATY PARRY YOUR & SONGS ROCK!!!!!!!!! THATS A TOTAL SUBSCRIBE
keyword subscribe
SPAM
HAM
HAM
4 s...from the training manual it show there is no tech procese:) its all about pessword resot and troubleshooting))
keyword password
SPAM
HAM
HAM
thanks for your message. really approriate your sacrifice. i'm sure the process of direct pay but will find out on my way back from the lest tomorrow. i'm in class now. do lave a wonderful day.
keyword thanks
keyword direct
SPAM
HAM
HAM
Figure Examples of changed labels in YouTube and SMS datasets after denoising with ULF sann-
7 CONCLUSION
In our work, we explored the denoising of weakly supervised data using information from labeling functions that annotate the data. Our original intuition was that the noise specific to some LFs can be detected by training a model that does not use those LFs signals (i.e., the LFs are held-out) and then comparing the predictions of that model to the labels generated by the held-out LFs. A mismatch between such out-of-sample predictions and the weak labels could point to potentially incorrect labels produced by the noisy LF. This way, improving the weak labels can be done without involving any additional source of annotation or external knowledge but with the data itself.
This idea was realized in our new extensions of two k-fold cross-validation frameworks, initially proposed for denoising the manually annotated data Weigh and Cleanlab), as well as in our new method ULF for unsupervised labeling functions denoising. The extensive experiments with feature- -ased ULF and DeepULF demonstrate the effectiveness of our approach and confirm the significant role of labeling functions in weakly supervised data denoising.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Túlio C. Alberto, Johannes V. Lochter, and Tiago A. Almeida. Tubespam: Comment spam filtering on youtube. In 20/5 IEEE 14th International Conference on Machine Learning and Applications doi: 10.1 N..109//CCMLAA205......
(ICMLA). pp. 138-143, 2015.
Tiago A. Almeida, José María G. Hidalgo, and Akebo Yamakami. Contributions to the study of sms spam filtering: New collection and results. In Proceedings of the I 11th ACM Symposium Document Engineering. DocEng 11, pp. 259-262, New York, NY, USA, 2011. Association for doi: 10.1145/20346911220 2202244 URL https: //doi.org/10/ 0111/134//449111 2555/2203469122337423 2034742
Computing Machinery. ISBN 473145038
Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi. Learning from rules generalizing labeled exemplars. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020. URL httt:st://peeeeeeeeeeee net / Eorum? i d= SkeuexBtDr
Benedikt Boecking, Willie Neiswanger, Eric Xing, and Artur Dubrawski. Interactive weak super- vision: Learning useful heuristics for data labeling. In International Conference on Learning Representations, 2021.
Oishik Chatterjee, Ganesh Ramakrishnan, and Sunita Sarawagi. Data programming using con- tinuous and quality-guided labeling functions. CoRR, abs/1911.09860, 2019. URL http:
Oishik Chatterjee, Ganesh Ramakrishnan, and Sunita Sarawagi. Robust data programming with precision-guidet labeling functions. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Confer- ence, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 3397-3404, AAAI Press, 2020. URL https://ojj aaai. org/ index. wttpppptt///aaaataaaeeeeeeeaaaee
D. Comey, M-Dyaa Albakour, Miguel Martinez-Alvarezz and Samir Moussa. What do million news articles look like? In News @ ECIR, 2016.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep
bidirectional transformers for language understanding. CoRR, abs/1810.045 810.04805, 2018. URL
http://arxiv. wwt.o//gaalooo abs 04805
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. N9.1865//1/N119-4432 1423. URL https:
T/Aclantholooo.czr///111
Zhiyuan Fang, Shu Kong, Zhe Wang, Charless C. Fowlkes, and Yezhou Yang. Weak supervision and referring attention for temporal-textual association learning. CORR, abs/2006.1 1 11747, 2020.
URL wttptttss/aaaa.t http:://axxivvvooo org/ abs 20006 11747
Jason A. Fries, Ethan Steinberg, Saelig Khattar, Scott I., Fleming, Jose Posada, Alison Callahan, and Nigam H. Shah. Ontology-driven weak supervision for clinical entity classification in electronic health records. Nature Communicationns, 12(1):2017, 2021. doi: 10. 10.10036611477000-222333 URL http:p:://doo.orgg/// 1038/ /e/////44477-0111220
Daniel Fu, Mayee Chen, Frederic Sala, Sarah Hooper, Kayvon Fatahalian, and Christopher Re. Fast and three-rious: Speeding up weak supervision with triplet methods. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research. pp. 3280-3291. PMLR, 13-18 Jul 2020.
URL htttps::ppooceecnngggg www.ingssmmr..... wtt.isssmm..pr.ee... www.spressse...... www.pprresviiii. vttp//l11///////// fu2 (au2aa.. html
Jacob Goldberger and Ehud Renvne Training deep netwalss using noise adaptation layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
10

---

Under review as a conference paper at ICLR 2023
Michael A. Hedderich and Dietrich Klakow. Training neural network in low-resource setting on automatically annotated noisy data. In Proceedings of the Workshop on Deep Learning Ap- proaches for Low-Resource NLP, pp. 12-18, Melbourne, July 2018. Association for Compu- tational Linguistics. doi: 10. 10.18653/v1// 10//3//1//11-3/0 NWW180422
URL https: /taps://acllc http://aaaaanthhllcc anttalltholy.y loo.oorgyrrroo..................
W18- 3402
Michael A. Hedderich, David Ifcoluwa Adelani, Dawei Zhu, Jesujoba O. Alabi, Udia Markus, and Dietrich Klakow. Transfer learning and distant supervision for multilingual transformer models: A study on african languages. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 2580-2591. Association for Computational www.:0000.mmmlmmmi..2...m 204. URL htttps://doi.crgg
Linguistics, 2020. doi: 10.18653/
18653/v1/2024 www.p-mmiin....... 204
Michael A. Hedderich, Lukas Lange, and Dietrich Klakow. ANEA: distant supervision for low- resource named entity recognition. CoRR, abs/2102.13129, 2021. URL https://arxi>
org htrgabb/21002.. 13129
Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S. Weld. Knowledge- based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. pp. 541-550, Portland, Oregon, USA, June 2011. Association for Computational org.://oooooor//11111/
Linguistics. URL
Giannis Karamanolakis, Subhabrata Mukherjee, Guoqing Zheng, and Ahmed Hassan Awadallah. Self-training with weak supervision. In Kristina Toutanova, Anna Rumshisky, Luke Zettle- moyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tammoy Chakraborty, and Yichao Zhou (eds.). Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAT 2021, Online, June 6-11, 2021, pp. 845-863. Association for Computational Linguistics, 2021. doi: 10.18653/v1/202 .naacl-main.66 URL
naacl ma in. 6.6
Zornitsa Kozareva, Ellen Riloff, and Eduard Hovy. Semantic class learning from the web with hy- ponym pattern linkage graphs. In Proceedings of ACL-08: HLT, pp. 1048-1056, Columbus, Ohio, June 2008. Association for Computational Linguistics. URL htttps://ccarit Drg/P08-1113
Gourab Kundu, Prahal Arora, Ferdi Adeputra, Polina Kuznetsova, Daniel McKinnon, Michelle Che- ung, Larry K. Anazia, and Geoffrey Zweig. Multi-modal content localization in videos using weak supervision. Proceedings of the 36 th International Conference on Muchine Learning, Long Beach, Califormia, PMLR 97, 2019.
Ouyu Lan, Xiao Huang. Bill Yuchen Lin. He Jiang, Liyuan Liu, and Xiang Ren. Learning to contextually aggregate multi-source supervision for sequence labeling. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 2134- 193.
2 2146. Association for Computational Linguistics, 2020. doi: 10. 00.18633/1/2020..e URL https: hte.ddd..oggo.1111e 1tte//0.16653//10 w02222000////20..... www-mmin...... 193
Lukas Lange, Michael A. Hedderich, and Dietrich Klakow. feature-dependess confusion matri- ces for low-resource NER labeling with noisy labels. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence Natural Language Processing (EMNLP-IJCNLP) pp. 3554 3559, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1128 URL httpps://cct lantho ogy. wtt..log..........o////g 1362
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo Mendes, Se- bastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, and Christian Bizer. Dbpe- dia a lcrge-scale, multilingual knowledge base extracted from wikipedia. Semantic Web Journal, 6, 01 2014. doi: 10.3233/SW-14013.
11

---

Under review as a conference paper at ICLR 2023
Jiahui Li, Wen Chen, Xiaodi Huang, Shuang Yang. Zhiqiang Hu, Qi Duan, Dimitris N. Metaxas, Hongsheng Li, and Shaoting Zhang. Hybrid supervision learning for pathology whole slide im- age classification. In Marleen de Bruijne, Philippe C. Cattin, Stéphane Cotin, Nicolas Padoy, Stefanie Speidel, Yefeng Zheng, and Caroline Essert (eds.), Medical Image Computing and Com- puter Assisted Intervention MICCAI 2021 24th International Conference, Strasbourg, France, September 27- October 1, 2021, Proceedings. Part VIII, volume 12908 of Lecture Notes in Computer Science, pp. 309-318. Springer, 2021. doi: 000079978---00-87277777 https: or www.oogggC..... 1000000007777 000---3-0000-------03
URL
Xin Li and Dan Roth. Learning question classifiers. In COLING 2002: The 19th Interna- tional Conference on Computational Linguistics, 2002. URL https:/ /tmpps://wee /www aww.wllweee wel..ee..o. ocl.
anthelogy/c02-1190
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. Neural relation extraction with selective attention over instances. In Proceedings of the 54th Annual Meeting of the Associ- Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 1.1.8666111111612000 URL
ation for Computational Linguistics (Volume 1: Long Papers), pp.
https://aae antho ogy.org/pl6..... 1200
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11192 2019. URL
http//axxxvv....aaaaa/aa ort. /1907.19990...0 11692
Alessio Mazzetto, Dylan Sam, Andrew Park, Eli Upfal, and Stephen Bach. Semi-supervised ag- gregation of dependent weak supervision sources with performance guarantees. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings The 24th International Conference on Arti- ficial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 3196 3204. PMLR, 13-15 Apr 2021. URL https: httts:::/proceedll ngs mlr press/v135/ ma mazzet html
Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. Distant supervision for relation extrac- tion without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pp. 1003-1011, Suntec, Singapore, August 2009. Association for Computational Lin- guisties. URL httph:tt///// aaccc/accaaaacaa anthology........ 0rg/ 1113
Curtis G. Northcutt, Lu Jiang, and Isaac L. Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research (JAIR), 70:1373-1411, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019.
Alexander Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data program- ming: Creating large training sets, quickly. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16. pp. 3574-3582, Red Hook, NY, USA, 2016.
Curran Associates Inc. ISBN 9781510838819.
Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christo- pher Ré. Training complex models with multi-task weak supervision. Proceedings of the AAAI Conference on Artificial Intelligence, 33:4763 4771, 07 2019. doi: 10. 10.1609/aami.v33100 33014763.
Alexander Ratner, Stephen H. Bach, Henry R. Ehrenberg, Jason A. Fries, Sen Wu, and Christo- pher Ré. Snorkel: rapid training data creation with weak supervision. VLDB J., 29(2-3); 709 730, 2020. doi: 10.10077400078-0099005222 URL 600778-019-00555
1007/
12

---

Under review as a conference paper at ICLR 2023
Wendi Ren, Yinghao Li, Hanting Su, David Kartchner, Cassie Mitchell, and Chao Zhang. Denois- ing multi-source weak supervision for neural text classification. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, On- line Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACI., pp. 3739-3754. Association for Computational Linguistics, 2020. doi: 10.
URL
Anastasiaa Sedova, Andreas Stephan, Marina Speranskaya, and Benjamin Roth. Knodle: Mod- ular weakly supervised learning with PyTorch. In Proceedings of the 6th Workshop on Rep-
resentation Learning for NLP (RepLANLP-2021) pp. 100-111, Online, August 2021. Asso-
ciation for Computational Linguistics. doi: 10.18653/v 1118633//12222..rrrg 1000://.........11pp11..... http://ow..r//20.....eeo.m1.pp
URL https:
Kai Shu, Guoqing Zheng, Yichuan Li, Subhabrata Mukherjee, Ahmed Hassan Awadallah, Scott Ruston, and Huan Liu. Leveraging multi-source weak social supervision for early detection of fake news. CoRR, abs/2004.01732, 2020.
Snorkel.
Detecting spouse mentions in sentences. Accessed: 15 May 2022.
https://wwis http://wwwwsnorke..coo
use lsse-cases/spooseeeoe
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir D. Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. arXiv: Computer Vision and Pattern Recognition, 2014.
Ang Sun and Ralph Grishman. Active learning for relation type extension with local and global data views. In Xue-wen Chen, Guy Lebanon, Haixun Wang, and Mohammed Zaki (eds.), 2 ACM International Conference on Information and Knowledge Management, CIKM 12, Maui, HI, USA, October 29 November 02, 2012, pp. 1105-1112 ACM, 2012. doi: 10.1145/2396761. 2398409. URL https:::////.. http://................232333333 2398409
org 10. 01..0..1442223
Christopher Teljstedt, Magnus Rosell, and Fredrik Johansson. semi-automatic approach for label- ing large amounts of automated and con-automated social media user accounts. In 2015 Second European Network Intelligence Conference, pp. 155-159, 2015. doi: 10.1 1119//NIC..00.....
Paroma Varma and Christopher Ré. Snuba: Automating weak supervision to label training data. Proc. VLDB Endow, 12(3):223-25 November 2018. ISSN 8097. doi: 10.14778/329164. 3291268. URL http::::::/ooi..r///0/00 Tet.....1777772788 32912699228
Hao Wang, Bing Liu, Chaozhuo Li, Yan Yang. and Tianrui Li. Learning with noisy labels for sentence-level sentiment classification. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat- ural Language Processing (EMNLP-IJCNLP), pp. ber 2019a. Association for Computational Linguistics. doi: 10.18653/w1/D11--155 URL https:://aclane logy www.oog...oogg1oe//ee
Hong Kong, China, Novem-
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 322- 330. IEEE, 2019b. doi: N0.1109//CC.22019. 00041. URL https::/ www.aii.orgg/l/
ICCV 2019.00990000
Zihan Wang, Jingbo Shang, Liyuan Liu, Lihao Lu, Jiacheng Liu, and Jiawei Han. Crossweigh: Training named entity tagger from imperfect annotations. In Kentaro Inui, Jing Jiang, Vin- cent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Lan- guage Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 5153- 5162. Association for Computational Linguistics, 2019c. doi: 10.19653/1/DD1--1599, URL wtw.:iioogg1.... wttp://....1553/11/11
1519
Tzu-Tsung Wong and Po- Yang Yeh. Reliable accuracy estimates from k -fold cross validation. IEEE Transactions on Knowledge and Data Engineering, PP:1-1, 04 2019. doi: 10.1109/TKDE.2199, 2912.15.
13

---

Under review as a conference paper at ICLR 2023
Yue Yu, Simiao Zuo. Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang. Fine-tuning pre- trained language model with weak supervision: A contrastive self traininan approach. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer. Dilek Hakkani- Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 1063- 1077. Association for Computational Linguistics, 2021. doi: 10.18865/v/2222 htt.:/2222...aannnnammmm Lnacc--main84. URL https: / wtt.o//.......//////./ 20/5533/112222// nww.canccc..cccc.cccc..aaac..aaa 1 -ma in. 84
Changsen Yuan, Heyan Huang, Chong Feng, Xiao Liu, and Xiaochi Wei. Distant supervision for relation extraction with linear attenuation simulation and non-iid relevance embedding. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January /t199/aaii.33101133331744
27 February 1, 2019, pp. 7418 7425. AAAI Press. 2019. doi: URL https: doi org. 10. 1609/ aaai. 133101.333331. 33017418
Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang. Mao Yang, and Alexander Ratner. WRENCH: A comprehensive benchmark for weak supervision. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. URL https:::
A WEAKLY SUPERVISED CROSSWEIGH
The original CrossWeigh framework (CW, Wang et al. 2019c) was proposed for tracing inconsistent labels in the crowdsourced annotations for the NER task. After randomly splitting the data into k folds and building training and hold-out sets, Cross Weigh additionally filters the training samples that include the entities matched in hold-out folds samples. The intuition behind this approach is: if an entity is constantly mislabeled, the model would be misguided; but the model trained without would be rid of this confusion.
We consider this approach quite promising for detecting unreliable LFs in weakly supervised data in similar way. If potentially erroneous LF systematically annotates the samples wrongly, reliable model trained data without it will not make this mistake in its prediction, and, thus, the error will be traced and reduced. Our new Weakly Supervised Cross Weigh method (WSCW) allows splitting the data not entirely randomly but considering the LFs so that all LFs matched in a test fold are eliminated from the training folds.
More formally, we, firstly, randomly split labeling functions L into k folds if..... fx Then, we iteratively take LFs from each fold fa test LFs and the others training L.s. So, all samples where no LFs from hold-out fold match become training samples, while the rest are used for testing.
Xtrain; = nf 0}
X Xaut, X X Xtrain,
After that we train the & separate models on Xtrain, and evaluate them on Xout, In the same way as in the original CrossWeigh algorithm, the labels predicted by the trained model for the samples in the hold-out set ú are compared to the initial noisy labels y. All samples X, where ú * yj are claimed to be potentially mislabeled; their influence is reduced in further training. The whole procedure of error detection is performed times with different partitions to refine the results. The sample weights Way are then calculated as Way es,, where ej the number of times a sample 2r was classified as mislabeled, 0 S Cj I t, and e is a weight reducing coefficient.
B WEAKLY SUPERVISED CLEANLAB
The second method we introduce is Weakly Supervised Cleanlab (WSCL)- an adaptation of Clean- (orrmeeorkNNohhuuuttt
lab framework Northcutt et al. (2021) I) for weak supervision. In the same way in WSCW, not data samples, but the labeling functions L are split into folds (f...., f& and used for building the Xtroing and X ond; sets, i < k for training k models.
14

---

Under review as a conference paper at ICLR 2023
In contrast to WSCW, for each sample the label is not directly predicted the Xout, but the probability vector of class distribution p(y = ji 2; f E is considered. The exact labels à are calculated later with respect the class expected self-confidence value ł, (see Northcutt et al. 2021]
Ex, p(y = -jxx,, 1j
XXyl
where X3 {r, E _
sample z; is considered to confidently belong to class j E if the probability of class j is greater than expected self-confidence for this class 1j the maximal one in the case of several classes is probable:
91 arg max
ply = ))
JE[K]:
Ao=jia.
The samples with probability that would exceed the thresholds have decisive label and do participate in further denoising.
After that, a class-to-class confident joint matrix Cy. is calculated, where:
Cg.bbilk = kz, X y; = j. -
Notably, Cy.s contains only the information about correspondence between noisy and out-of-sample predicted labels (the same way in Northcutt et al. [02T]. So, it gives the idea about the number of samples with presumably erroneous noisy labels y but does not give us any insights about the erroneous labeling functions that assigned this noisy label this sample (in contrast to the ULF approach we present in Section 3)
The confident matrix Cy.ū is then calibrated and normalized in order to obtain estimate matrix of the joint distribution between noisy and out-of-sample predicted labels Qy. which determines the number of samples to be pruned. We perform the pruning by noise rate following the Cleanlab default setting: Qu.s, i + samples with max(p(y (y- = j) p(y = i)) are eliminated further training.
C DATASETS
YouTube (Alberto et al. 2015) spam detection dataset was collected from the YouTube video comments. The samples that not relevant to the video (e.g. advertisement of user's channel ask for subscription) are classified as SPAM, while others belong to the HAM class. We use the same created using keywords, regular expres-
labeling functions in (Ratner al. 2020); they sions, and heuristics. For example, there is a labeling function keyword dlmy which corresponds to class SPAM, meaning that sample contains the word "my", will be assigned the SPAM class. Examples of other labeling functions are ZEYWORD SUBSCRIBE, KEYWORD LEASE, KEYWORD _SONG (all are the syyoord-based., short, it is probably not spam; thus, samples less than 5 words long would be classed as HAM) and so on.
(i.e., if YouTube comment
Spouse (Corney et al.) 2016) relation extraction dataset based on the Signal Media One-Million News Articles Dataset, which main task is define whether there is spouse relation in sample.
We use the Snorkel annotation (Snorkel)); the labeling functions were created based on keywords (e.g. spouse, husband), spouse relationships extracted from DBPedia (Lehmann et al. 2014), and language patterns (c.g. check whether the person mentioned in a sample have the same last name).
(Li & Roth 2002) A question classification dataset that maps each data sample to of classes. The labeling functions were generated based on keywords (Awasthi et al. 2020), e.g. whi ch, what, located, situated keywords relate sample to the class T.CCATTON.
TREC
15

---

Under review as a conference paper at ICLR 2023
SMS
(Almeida al. 2011) A spam detection dataset comprised of text messages. The annotation
Awasthi et al., 2020) includes keyword based and regular expression- based labeling functions. For example, regex- based labeling function:
( (2) (won won) ["\w] * [s]+ ) (claim, Iclaim) ["Nw] *
($)
corresponds class SPAM (e.g. in sample 449050000301 You have won ??2,000 price! To claim, call 09050000301.).
Yorùbá and Hausa (Hedderich al., 2020) Topic classification datasets of the second (Hausa) and the third (Yorùbá) most spoken languages in Africa comprised of news headlines. The weak keyword rules are provided by the authors (Hedderich et 2020).
YouTube
Spouse
TREC
SMS
Yorûbá
Hausa
Train Data
1586
22254
4965
4502
1340
2045
Valid Data
150
2711
500
500
189
290
Test Data
250
2701
500
500
379
582
#Classes (K)
2
6
2
#LFs (L)
10
9
68
73
19897
18624
#Unlabeled
195
16520
242
2719
0
Avg I. Hits
1.6
33.7
1.7
0.5
3.0
2.9
LF Accuracy
81% + 2.0
53% + 0.6
50% + 2.6
60% + 1.6
55% + 1.5
54% + 0.3
LF Coverage
87%
25%
85%
40%
100%
100%
Table 3: Statistics of all the datasets. The LF accuracy metrics are calculated with majority vote without any model training reported across runs with standard deviation in order reduce the instability caused by randomly broken ties.
D IMPLEMENTATION DETAILS
Feature-based ULE The feature-based ULF is realized with logistic regression model; the train- ing data are encoded with TF-IDF vectors. For training, we set the number of epochs to 20 and applied the early stopping with patience = Each experiment was run times with different initializations. Development set is used for early stopping as well as to estimate the number of iter- ations initially, is set to 20, but if training labels do not change after three iterations, the algorithm stops, and the last saved model is used for final testing. The actual number of iterations training of the best performing ULF model ULF alongside other hyperparameter values found using grid-search, can be found in Table 10] the search space is provided in Table
Hyperparameter
Values
Multiplying coefficient p Learning rate Ir
0.1, 0.2, 0.3, 0.4, 0.5, 0.9 0.9 le-1, le-2, le-3, le-4
Number of folds k
3, 8, 10, 15, 20 (with respect to the overall number LFs)
Non-labeled data rate
0, 0.5, 1, 2,
Table 4: Feature-based ULF: hyperparameter values tried in grid search.
16

---

Under review as a conference paper at ICLR 2023
YouTTbee
Spouse
TREC
SMS
Multiplying coefficient p
0.5
0.2
0.3
0.1
Learning rate Ir Number folds k
le-2
le-2
le-1
le-1
3
3
10
Number of iterations I Non-labeled data rate À
1
0
3
1
0.5
Table 5: ULF selected hyperparameters.
DeepULF.
The pre-trained language models were downloaded from HuggingFace" To encode the features, we used multilingual pre-trained BERT mode for Yoruba and Hausa datasets and pre- trained RoBERTa mode for others and followed the Wrench encoding method (Zhang al. 2021 )
The gradient- scheduler, and the early stopping applied based the evaluation metric values the validation set. For the baselines and Cosine training, we took the parameters listed by zhang2021 wrench. Additionally, we include the label prediction parameter: if equals "soft", the probabilistic labels are used for training; otherwise ("hard") label is the one-hot encoding of the most probable class (the ties broken randomly). ULF-specific parameter search space was defined heuristically, All parameters are provided in Table O
optimization was performed with Adam Optimizer and linear learning rate
In order to reduce computational load we used the random parameter search instead of grid search. Specifically, we tried 10 random parameter combinations from search space and selected the one which performed the best on the development set.
Hyperparameter
Values
Multiplying coefficient p Learning rate Ir Number of folds
0.1. 0.3, 0.5, 0.7, 0.9
lc-4, 1e-5, le-6
5,5, 7 (with respect to the overall number of LFs)
Number of iterations I
1, 3,
Confident regularization weight À
0.01, 0.1
The confident threshold
0.2, 0.4, 0.6, 0.8
Label prediction
soft, hard
Table 6: DeepUL.F: hyperparameter values tried in random search.
YouTube
Spouse
TREC
SMS
Yorŭbá
Hausa
Multiplying coefficient p
0.5
0.1
0.1
0.3
0.1
0.1
Learning rate ir Number of folds &
lc-4
le-06
le-05
lc-05
lc-06
le-06
3
3
3
Number of iterations
2
1
1
2
Confident regularization weight
0.1
0.01
0.1
0.1
0.1
0.05
Confident threshold &
0.8
0.2
0.6
0.8
0.2
0.8
Label prediction
soft
soft
soft
soft
soft
soft
Table 7: DeepULFFr.rt selected hyperparameters.
"nuts://uuggingfaeecc//cmm co/modelss
"httss//huggingfacc. http:/aacccccccc..aeaeee cow/eet-aaee-mulllinguu-aeeeeeee "hups://huginngface.
htw.r/nnnnn..cccoooreetaaaaae co/roberta-base
17

---

Under review as a conference paper at ICLR 2023
YouTube
Spouse
TREC
SMS
Yorùbá
Hausa
Multiplying coefficient
0.5
0.1
0.3
0.3
0.1
Learning rate Ir Number of folds k
le-4
1e-06
1e-06
le-05
1e-05
1e-05
2
3
5
3
2
Number of iterations Confident regularization weight À
3
1
1
2
0.1
0.01
0.01
0.1
0.1
0.05
Confident threshold {
0.8
0.2
0.8
0.8
0.2
0.4
Label prediction
soft
hard
soft
soft
soft
soft
Table 8: DeepULFFT.com selected hyperparameters.
YouTube
Spouse
TREC
SMS
Yorùbá
Hausa
Multiplying coefficient
0.1
0.1
0.1
0.3
0.1
0.1
Learning rate Ir Number of folds k
le-06
le-06
1e-05
le-06
1e-06
1e-05
3
3
3
5
7
Number of iterations Confident regularization weight À
1
1
2
0.1
0.01
0.1
0.01
0.05
0.05
Confident threshold g
0.2
0.2
0.6
0.4
0.4
0.4
Label prediction
soft
hard
soft
soft
soft
soft
Table 9: DeepULFcos FT selected hyperparameters.
YouTube
Spouse
TREC
SMS
Yorùbá
Hausa
Multiplying coefficient
0.7
0.1
0.3
0.1
0.1
0.1
Learning rate Number of folds &
le-05
le-06
le-06
le-05
1e-06
le-05
5
3
5
7
7
Number of iterations
2
2
Confident regularization weight À
0.01
0.01
0.01
0.05
0.05
0.05
Confident threshold g
0.2
0.2
0.8
0.4
0.4
0.4
Label prediction
soft
hard
soft
hard
soft
hard
Table 10: DeepULFcos cos selected hyperparameters.
Both feature-based ULF and DeepULF were implemented with Python and PyTTorch (Paszke al., 2019) in the setting of the weak supervision framework Knodle (Sedova et al. 2021). By providing an access all WS components Knodle allowed us implement and benchmark all algorithms described above. ULF experiments were performed on machine with CPU frequency of 2.2GHz with cores; DeepULF experiments were run on asingle Tesla V100 GPU on Nvidia DGX-1. The full setup took average 20 hours for each dataset for ULF and 96 hours for DeepULF.
E DEEPULF RESULTS
YouTube (Acc)
Spouse (F1)
TREC (Ace)
SMS (F1)
Yorûbá (F1)
Hausa (F1)
DeepULFFT.FT
96.8
22.0
68.2
96.1
54.6
43.0
DeepULF Fyr.cos
94.4
36.9
76.6
96.1
54.2
48.2
DeepULFco.
95.2
21.3
68.6
96.1
55.8
43.6
DeepULF Cos Cos
94.8
33.0
76.8
96.1
54.2
44.5
Table 11: DeepULF results. All possible cross-validation and end model combinations.
18


Reviews:
[{"review_id": "chRsJ10tc6", "reviewer": "Reviewer_ECG9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors investigate noise reduction techniques for weak supervision based on the principle of k-fold cross-validation.  More specifically, the authors propose a new algorithm ULF for denoising weakly annotated data which uses models trained on all but some LFs to detect and correct biases specific to the held-out LFs. The experimental results help validate the effectiveness of  ULF in weakly supervised learning setting. ", "summary_of_the_review": "The authors propose a method ULF, which could denoise weakly annotated data by refining the allocation of LFs to classes. The motivation of the proposed method is clear and the proposed design is reasonable. The experiments are conducted on several datasets to validate the effectiveness. Some of concerns are included in the Strength And Weaknesses section.", "strength_and_weaknesses": "Strength:\n1. The authors work on a practical and important problem.\n2. The authors introduce their method with a clear description and motivation\n3. The proposed method mainly aims to refines the allocation of LFs to classes, which is a reasonable and less studied design.\n \nWeakness:\n1. The [1] shares a similar motivation, which also aims to learn the weights of LFs to data. \n2. The reported performance on TREC from [2] is 82.59, which is higher than that of the proposed method and its implementation in Table 2.  It is suggested to add more experiments on the some datasets from  [2] to carefully validate the progress.\n3. The motivation is clear but corresponding empirical analysis are missing. What are the refine results of the allocation of LFs to classes? \n\n[1] Giannis Karamanolakis, Subhabrata Mukherjee, Guoqing Zheng, and Ahmed Hassan Awadallah. Self-training with weak supervision.  NAACL 2021. \n[2] Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, Chao Zhang. Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach NAACL 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X1hp6vwUNpo", "reviewer": "Reviewer_Cajh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method called ULF to correct the mapping between labelling functions and classes in weakly supervised learning. In this setting, initially a labelling function maps instances to some specific classes. As this mapping can be noisy, ULF iteratively updates the (soft) mapping by estimating how correct the mapping is. This estimation is based on cross-validation-based confidence learning proposed by Northcutt et al (2022). The paper demonstrates that ULF outperforms several weakly supervised baselines (including Snorkel-DP) on 4 text classification datasets.  ", "summary_of_the_review": "In general I like the paper, both the ideas and the experimental results. It should be a welcomed contribution to the community whose work is on learning from noisy data. \n\nAlthough I recommend to accept the paper, there are several points that the paper can be improved: \n* the writing can be clearer (especially the description for cross-validation-based confidence estimation), some parts can be shorten for analyses. \n\n* the paper will be stronger if ULF can be framed in a theoretical framework. \n\n* more analyses are needed to understand the quality of T* and Y*. ", "strength_and_weaknesses": "ULF is an interesting and effective way to use confidence estimation in weakly supervised learning setting. \n\n* It is well motivated, for why labelling labels can be noisy and how correcting the noise can be helpful. Cross-validation-based confidence estimation is well argued as a promising way to overcome the problem. However, using this confidence estimation is not straightforward (as pointed out in the experiments -- comparison against wscrossweigh and wscleanlab). \n\n* The idea of incrementally updating the mapping between labelling functions and classes seems to have a nice connection with the noisy-channel-based approaches (e.g. Goldberger & Ben-Reuven 2017) where confusion matrices play a key role in estimating the labelling noise. \n\nThe paper however lacks a theory to support the idea. Incrementally updating the matrix T* and noise labels Y* sounds reasonable but can be dangerous as we don't know to where they will converge (or will they converge at all). In fact, there's even no objective for ULF to work on. (On the contrary, the noisy-channel matrix is part of the loss to be minimized.) Would it be possible to view T* and Y* as latent variables and put them in a theoretical framework (as in Snorkel or noisy-channel)?\n\nWhen comparing ULF to data programming Snorkel, it is unclear whether ULF is able to deal with dependency between labelling functions. \n\nThe use of development sets in experiments for early stopping and hyper param tuning is typical. However, if these sets are clean (annotated labels are not noisy), then they don't seem to be used efficiently. For instance, why not using them to estimate the confidence directly? Or, at least they maybe more helpful than only to select p. Also, having clean dev. sets can be tricky as one can train a model directly on these sets in a supervised learning fashion. \n\nAs estimating T* and Y* is the core of ULF, the paper should have some analyses on whether T* and Y* are estimated more correctly after each iteration. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Wb_ywZCPsUl", "reviewer": "Reviewer_jWdL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a method for noise reduction in cases of automatic annotation using pre-defined labeling functions. They propose to do the noise reduction using the principle of k-fold cross validation, and through the iterative process are able to automatically denoise the entire dataset in an unsupervised fashion. Their major contribution is in more accurate labels and better quality of the trained classifier from the labels, and in providing a feature-based method (for feature based learning without a hidden layer) and deep learning method (for fine-tuning pre-trained language models). ", "summary_of_the_review": "The work is quite interesting and the idea is simple, but I would like to see more justification and examples for the intuition and why it makes sense. ", "strength_and_weaknesses": "Strengths: \n- The paper has strong performance improvements compared with the baseline and competitor methods, as listed in table 1 and table 2\n- The method is relatively simple to implement for denoising automatic labels and could have large scale impact if well generalizable \n\nWeaknesses: \n- The method is based on an intuition that a “mismatch between predictions of a model and labels can indicate candidates of noise specific to all held out labeling functions.” This intuition needs stronger support and explanation beyond a few examples mentioned in Figure 1. \n- It is unclear on whether this method can be applied beyond the language modality, and if so, how would that be implemented \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4Gsggy0d0fK", "reviewer": "Reviewer_SFmE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents ULF (Unsupervised Labeling Function) which uses k-fold cross-validation to correct errors in labeling functions (LFs) by using cross-validation to identify confident sample estimates and update LF class label assignments. They explore 2 variants of ULF: a count-based and language model / DeepULF based method. The evaluate on up to 6 benchmark tasks and explore a wide range of baseline methods, including two other cross-validation-based methods and a host of label model configurations used to train RoBERTa + Cosine (an existing self training method for incorporating unlabeled training data). The manuscript reports that ULF in it's variations outperforms existing methods.  ", "summary_of_the_review": "Unfortunately given the clarity in presentation and the inconsistency with reported results (which would erase ULF's purported benefits), I don't think this paper is ready for acceptance. ", "strength_and_weaknesses": "Strengths\n-  Viewing LF improvement through the lens of data quality is interesting and methods to fuse techniques that can modify and repair LFs is a valuable research focus\n- The authors use the WRENCH benchmark, which addresses an endemic weakness in weak supervision papers where the labeling functions are not consistent across methods, confounding performance insights. \n\nWeaknesses\n- The stated goal of the paper is to \"primary goal is to improve the LFs to classes allocation in order to correct the systematically biased label assignments\". This targets fundamentally misspecified labeling functions, since a core assumption of vanilla DP is that LFs perform better than random chance. These can happen in practice, but I'm not convinced flipping the label (essentially) is always beneficial especially in rare/infrequent classes settings.\n- The paper suffers from clarity issues. For example, the feature-based and deep ULF methods need more methods elaboration and details \n- From the Table 2 results, which use the WRENCH benchmark implementations (https://arxiv.org/pdf/2109.11377.pdf) the reported performance numbers don't seem to match what was reported in the original WRENCH paper. See their Table 12 the best reported best results for RoBERTa + Cosine given various label model classes. I've repeated the key parts below. This paper's implementation is underperforming and DeepULF does not outperform a tuned model. My concern is that the perform benefits might simply wash out when using a tuned end model. \n| Dataset/Metric | WRENCH/Label Model | This Paper | DeepULF |\n|----------------|--------------------|------------|---------|\n| YouTube/Acc    | 97.60 / MV         | 96.4       | 96.8    |\n| Spouse/F1      | 40.50 / MV         | 33.3       | 36.9    |\n| TREC/ACC       | 77.96 / MV         | 65.8       | 76.8    |\n| SMS/F1         | 96.67 / MV         | 93.6       | 96.1    |\n- There are stronger baseline label models than Flying Squid and the default Data Programming (Snorkel-DP) model See Mazzetto et al 2021. \"Semi-Supervised Aggregation of Dependent Weak Supervision Sources With Performance Guarantees\"\n- Ideally this type of paper would be more systematic in exploring properties of the task themselves, including low frequency class problems, higher cardinalities. It's hard to draw any strong insights from the experiments as presented. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject