ULF: Unsupervised Labeling Function Correction using Cross-Validation for Weak Supervision
Authors: Anastasiia Sedova
Benjamin Roth
Venue: EMNLP 2023 Main (2023)
Keywords: weak supervision
cross-validation
denoising methods
TL;DR: We propose a method ULF that builds on cross-validation to adjust the assignment of labeling functions to classes for weakly supervised learning.

Abstract: A cost-effective alternative to manual data labeling is weak supervision (WS), where data samples are automatically annotated using a predefined set of labeling functions (LFs), rule-based mechanisms that generate artificial labels for the associated classes. In this work, we investigate noise reduction techniques for WS based on the principle of k-fold cross-validation. We introduce a new algorithm ULF for Unsupervised Labeling Function correction, which denoises WS data by leveraging models trained on all but some LFs to identify and correct biases specific to the held-out LFs. Specifically, ULF refines the allocation of LFs to classes by re-estimating this assignment on highly reliable cross-validated samples. Evaluation on multiple datasets confirms ULF’s effectiveness in enhancing WS learning without the need for manual labeling.

ULF: Unsupervised Labeling Function Correction using Cross-Validation for Weak Supervision
Anastasiia Sedova and Benjamin Roth
Faculty of Computer Science, University of Vienna, Austria
Vne Doctoral School Computer Science, University of Vienna, Austria Faculty of Philological and Cultural Studies, University of Vienna, Austria rotheinniioooeeeiiiiiii ac.at
(anastasi ia. sedova,
Abstract
Sample
Matelted Labeling Funesiora
Audigned
CHECK MY CHANNEL PLEASE. SINGING COVERS
leyyoed Jny,
A cost-effective alternative manual data la- beling is weak supervision (WS), where data samples are automatically annotated using predefined set of labeling functions (LFs), rule- based mechanisms that generate artificial la- bels for the associated classes. In this work, we investigate noise reduction techniques for WS based on the principle of k-fold cross- validation. We introduce new algorithm ULF for Unsupervised Labeling Function correc- tion, which denoises WS data by leveraging models trained on all but some LFs to identify and correct biases specific to the held-out LFs. Specifically, ULF refines the allocation of LFs to classes by re-estimating this assignment highly reliable cross-vass validaaddd samples. Eval- uation on multiple datasets confirms ULF's ef- fectiveness in enhancing WS learning without the need for manual labeling.'
lacyiood
chicss
Helka! I'm Marian
wanted to
my own pop and pop-rodk songs. II would moan lot If you could have a look at my channel subscribe to XOXO THANKS!!
leyward
knywant sulteriiee beyword song
SPAM/ HAM
if u like.
toxbblob jubbjestivitty
so
my daughter big
keywoodd my sinngs
fan likow lot your songs.
HAM
Twitter ImsscliforniaS5
no matches
Table 1: Examples of WS annotation for YouTube dataset (Alberto et al., 2015). (1) is classified as spam as matched LFs belong to the SPAM class. In (2) and (3), there is conflict matched LFs belong different classes. In (4), no LFs matched; such samples are usually filtered out.
leaving out portion of the data during training, the model avoids overfitting to enrors specific that part. Hence, a mismatch between predictions of a model trained on a large portion of the dataset and the labels of the held-oul portion can indicate potential noise specific to the held-on out portion. Pre- vious pross-validation
1 Introduction
denoising approaches (Northcutt et al., 2021; Wang et al., 2019b) split the data samples into folds randomly; a direct applica- tion of these methods to WS data ignores valuable knowledge stemming from the WS process. In our work, we leverage this knowledge by splitting the data based on matching LFs in the samples. The intuition is the following: a mismatch between predictions of a model trained on a large portion of the LFs and labels generated by held-out LFs can indicate noise specific to the held-out LFs. By performing cross-validation for each LF, noise as- sociated with all LFs can be identified.
A large part of today's machine learning success rests upon large amounts of annotated training data. However, collecting manual annotation (even in a reduced amount, e.g., for fine-tuning large pre- trained models (Devlin et al., 2019), active learn- ing (Sun and Grishman, 2012), or semi-supervieed learning (Kozareva et al., 2008)) is tedious and expensive. An alternative approach is weak super- vision (WS), where data is labeled in an automated process using
multiple WS sources such keywords (Hedderich et al., 2021), knowledge bases (Lin et al., 2016), and heuristics (Varma and Ré, 2018), which are encoded as labeling functions (LFs, Ratner et al. 2020). LFs are applied to unla- beled datasets to obtain weak training labels, which are cheap. but often conflicting and error-prone, re- quiring improvement (see Table 1).
In contrast to correcting labels as in previous work for supervised settings, we utilize the cross- validation principle in a WS setting to adjust and improve the LF-to-clan assignment. In some cases, an LF correctly captures some samples but misla- bels others. For example, one of the LF' used annotate the YouTube dataset (Alberto et al., 2015) is the keyword my, which is effective in identify- ing spam messages such as subscribe to my chan-
We focus enhancing the quality of weak la- bels using k-fold cross valsd-aali Intuitively, by
We make our code availahle within the Knodle framework (Sedova et al., 2021): https//githuh..om/keeeo

---

Unlabeled dataset
Labeling functions (LFs)
ample, such assignment reduces the association of the LF "my" with the SPAM class, resulting in dominant HAM probability in Samples 2 and 3 in Figure 1. Moreover, ULF successfully labels samples with no LFs matched, as, e.g., Sample 4, unlike other methods that filter them out (Ratner et al., 2020). We conduct extensive experiments us- ing feature-based and pre-trained models to demon- strate the effectiveness of method. To the best of our knowledge, we are the first to adapt cross- validation denoising methods to WS problems and refine the LFs-to-class allocation in the WS setting.
TigAar expressonss
words
knowladge basea
Z
LF1
LF3
T
XQ
3
LF2
0
ULF
update
Noisy labels y
LF-to-class estimation Q
2 Related Work
train
Weak supervision (WS) has been widely applied to different tasks across various domains, such as text classification (Zeng et al., 2022), relation extraction (Datta and Roberts, 2023), named entity recogni- tion (Wang et al., 2022), video analysis (Chen al., 2023), medical domain (Fries et al., 2021), image classification (Yue et al., 2022). Weak annotations are easy obtain but prone to errors. Approaches to improving noisy data include building specific model architecture (Karamanolakis et al., 2021). us- ing additional expert annotations (Mazzetto et al., 2021), identifying and removing or downweighting harmful samples (Northcutt et al., 2021; Sedova et al., 2023), learning from manual user guid- (Boecking al., 2021; Chatterjee al., 2020). ULF compatible with any classifier and do not re- quire any manual supervision; instead of removing the samples, ULF corrects the labels, utilizing as much WS data as possible. K-fold cross-validation, a reliable method for assessing trained model qual- ity (Wong and Yeh, 2019), is also often used to de- tect errors in manual annotations (Northcutt et al., 2021; Wang et al., 2019b,a; Teljstedt et al., 2015), but has not been applied WS setting. We pro- pose WS extensions to some of these methods in Appendix and use cross-validation ULE.
predict
train
Predicted labels ÿ
predict
Clean labels
Figure 1: ULF. Noisy training labels y* are obtained by multiplying the matrices Z and T. 'Tee most con- fident predictions are calculated using k-fold cross- validation and used estimate new LFs-to-class corre- spondence and update the T matrix. The clean labels are obtained by multiplying the updated and Z matrices.
nel or check my channel out (see Table 1). How- ever, considering this LF as solely indicative of the numerous
spam class would be unwarranted, non-spam messages also contain the word my (e.g., Sample 3). The weak labeling of such samples can results in tie (i.e., one vote for SPAM and one vote for HAM), potentially leading to incorrect label assignments through (random) tie-breaking. To address such cases, we introduce new method ULF: Unsupervised Labeling Function correction (Figure 1), which comprises both erro- neous labels detection and correction by leveraging weakly supervised knowledge. ULF re-estimates the joint distribution between LFs and class la- bels during cross-validation based on highly confi- dent class predictions and their co-occurrence with matching LFs. Importantly, this reestimation is performed out-of-sample, meaning it is guided by the data itself without involving additional man- ual supervision. Instead of hard assignment of naive WS (i.e., an LF either corresponds to the class or not), ULF performs a fine-adjusted one, which helps to correct the label mistakes. For ex-
3 ULF: Unsupervised Labeling Function Correction
In this section, we present the key elements of ULF. More details can be found in Appendix A, the pseudocode is provided in Algorithm 1.
Given a dataset X 22,,, to be used for K-class classifier training. In WS setting, we do not have any gold training labels, but only a set of LFs = (4.,42,...,44) An LF 1F matches a sample 2; some condition formulated in I

---

ULF
to the held-out folds, resulting in outoffsammpe predicted probabilities PN*K. The out-of-sampe label i/s for each sample determined by selecting the class with the highest probability, provided that this highest probability exceeds the class average threshold ty.
If,
Ifz
f22
MLL
Iff
X1
C1
OK
X2
1f:
0.8
0
1f.
0
12
0.2
0.5
Toxe #LFs #classes
0.3
0.6
1 = Ex,e P(ū = j; 24
Toot #LFs #classes
Ta Asamples x #LFss
[Xj=sl
That is, a sample z; is confidently assigned class j if the out-of-sample probability of belonging to class j is higher than the average sample probability of all samples initially assigned to this class. If no probability exceeds the class thresholds (e.g., all probabilities are equally small), the sample is disregarded as unreliable for further calculations.
X1
0.2
0.9
X5
Ok
argmaa
Xa
0.1
X2
X3
0.5
1.2
X3
C1
0.6
yanaka; #samples x #classes
Visi #samples
Figure 2: Weak annotation encoded with and T matri- with Z containing LFs matches in samples, and T representing LF-to-class mapping. 7' is an improved version of 7 with ULF. Applying and (or T*) ma- trices multiplication and majority vote yields labels Y.
These assignments are used build an LFs-to- classes confidence matrix CLXK, which estimates the joint distribution between matched LFs and predicted labels. For each LF 4 and each class kj, the confidence matrix CLxX is populated by counting the number of samples that have LF I matched and confidently assigned to class ky:
holds for 24. Following Sedova al. (2021), store this information in binary matrix Z where Zzzf
1 means that LF lj matches sam- ple Tj. A set of LFs matched sample x; de- noted by Lx,- where Lr, C L and Lr,1 E [0, LI]. The LFs to class correspondence is stored in a bi- nary matrix Tixk, where Tiş
Ca.br = 1(2 E = =9 E Los)l.
(3)
Algorithm 1: ULF: Unsupervised Labeling
Function Correction for Weak Supervision Input: unsupervised training data X, samples to LFs matrix Z LFs to classes matrix TLxk, CV model g (<; 0o), end model h (-:0)
1 means the LF 4 corresponds to class (i.e., l assigns the samples to the class j)?. The weak training labels y= (ύn, 82, ..., Dn). . Dj K are obtained by mul- tiplying Z and T. apply majority vote, and break the ties randomly. The main goal of the ULF al- gorithm is to refine the T matrix. The graphical explanation provided in Figure 2.
Calculate noisy labels Y + ZT for iter 1,2, / do Split the data into & folds fi. fi..... fx
3
First, class probabilities are predicted for each sample using k-fold cross- cross-validation on the training set X and weak labels Y. We propose different ways of splitting the data into & folds f1.... fr- with the most reliable method being splitting by signatures (refer to Appendix A for details and other possible splitting methods). The samples' signatures, i.e., the sets of LFs matched in each sample, are collected, split into & folds, and used to create data folds: Xtrain,
4
for f., E 2k.., do Build Xtrain, Xout, sets (Eq. 1) Yiruin, E Vr E Xtrain,) -==rain(Xxxee)
5
Y(ruin.)
7
8
Calculate p(ū j; 24 0.) for Va;; E X Xout, S < K
Calculate labels in w.r.t the thresholds
ty (Eq. 2)
Calculate LFFs-tocllass confidence matrix Ci.g (Eq. 3)
[zz & fi). Xout; = Lz, E fi} (1). Next, & models are separately trained on each of k 1 folds and applied
Estimate Qt.5 joint matrix (Eq. 4)
Recalculate T matrix (Eq. 5)
Calculate improved labels y <- 2T
13
'The initial manual class assignment is typically such that each LF corresponds to one class, covering a prototypical case. However, assigning an I.F to multiple classes is theoretically possible and compatible with ULF.
0' ttain(XX, Ý)
Output : Trained 0'

---

YouTube Spouse
TREC
SMS
Yorübá
Hausa
Avg
Gold
98.8
96.6
97.7
67.3
83.5
88.8
Majority Vote (MV)
93.2
21.3
68.6
93.0
48.1
43.9
61.4
MeTaL (Ratner al., 2019)
96.0
19.6
55.8
89.2
58.6
41.6
60.1
Snorkel-DP (Ratner et al., 2020)
95.6
32.6
61.8
94.6
58.7
45.7
64.8
FlyingSquid (Fu et al., 2020)
94.0
14.9
35.8
23.7
32.4
45.1
41.0
WeaSEL (Cachay et ul., 2021)
96.0
14.9
64.4
23.6
49.6
43.2
48.6
FABLE (Zhang et al., 2023)
94.8
27.8
54.6
91.1
23.2
18.6
51.7
MV Cosine (Yu al., 2021)
96.4
33.3
65.8
93.6
52.6
45.4
64.5
MeTaL + Cosine
95.6
26.9
67.4
80.7
62.0
45.5
63.0
Snorkel-DP Cosine
96.0
28.1
73.8
96.1
55.0
46.5
65.9
FlyingSquid Cosine
95.6
24.9
38.6
90.1
33.3
41.5
54.0
FABLE- Cosine
94.0
33.9
70.6
97.7
60.1
44.7
66.8
ULF (Ours)
96.8
36.9
76.8
96.2
55.8
48.2
68.4
Table ULF experimental results with pre- are-trained language models. Accuracy is reported for YouTube and TREC, Fl score presented for other datasets account for class imbalance. The Gold baseline is not applied to the Spouse dataset due to the absence of gold labels. Hyper-parameters were obtained through random search with 10 initialization for energy considerations (not a grid search, as in, c.g., Zhang et al. 2021).
Subsequently, CLXK is calibrated and normal- ized to QLEK in order to align with the data pro- portions of the Z matrix:
CV Final YouTube Spouse
TREC
SMS
Yorübá Hausa
FT FT
96.8
22.0
68.2
96.
54.6
43.0
Cos
36.9
76.6
96.2
54.2
48.2
Cos FT 95.2
21.3
68.6
96.
55.8
43.6
Cos Cos
94.8
33,0
76.8
96.1
54.2
44.5
Table 3: Results of all ULF combinations of cross-
Q1.0: Cru
Σ m=1
Σ
Clmiss
validation and final model (simple fine-tuning (FT) or followed by additional Cosine (Cos) training).
(4)
Experiments
K
where EiEL, Chi = n, E Zim JEK
Qm..ss
Datasets and baselines. We evaluate ULF on four WS English datasets: (1) YouTube Spam Classification (Alberto et al., 2015); (2) Spouse Relation Classification (Corney et al., 2016); (3) Question Classification from TREC-6 (Li and Roth, 2002); (4) SMS Spam Classification (Almeida et al., 10, and two topic classification WS African datasets: (5) Yorùbá and (6) Hausa (Hed- derich et al., 2020). For all datasets, we utilize the LFs provided by dataset authors.
This calibration ensures that QL*K sums up the total number of training samples, and the sum of counts for each LF is the same in the original Z matrix; thus, QLEK can be utilized as a cross- validated re-estimation of T. Finally, a refined T matrix calculated as follows:
T p Q (1 TT
(5)
Here, the hyperparameter p, P [0, 1). determines the extent which information from the original T matrix should retained. The resulting 7* matrix is utilized to generate improved labels for addi- tional ULF iterations or training the final classifier.
We compare our results towards the (1) Gold baseline (the only classifier which exploits gold labels) and the most popular and recent WS base- lines: (2) Majority Vote, (3) MeTaL (Ratner et al..
2019), (4) Snorkel-DP (Ratner et al., 2020), (5) Fly- ing Squid (Fu al., 2020), (6) WeaSEL (Cachay et al., 2021), and (7) FABLE (Zhang al., 2023). More details are provided in Appendices and D.
Unlabeled samples. ULF also takes advantage of unlabeled samples that do not have any LFs matched. A portion of these samples, determined by the hyperparameter X, is randomly labeled and included in cross- validaaii training, with reesti- mation in subsequent iterations. To leverage all unlabeled samples in fine-tuning-bart -bsed setting, we also include the optional Cosine self-training step (Yu et al., 2021), which can be executed during cross-valioee valss-validation and/or final classifier training.
Results. We run experiments using RoBERTa fol- lowing Zhang al. (2021) for English datasets and mulitlingual BERT following Devlin et al. (2019) for others (more implementation details are pro- vided in Appendix F). Table 2 presents the results of the best combination of cross-validation and final models; each them can be either simple

---

fine-tuning or followed by Cosine training step (Yu al.. 2021). On average, ULF outperforms the baselines and achieves better results on four out of six datasets. The weakest performance was ob- served on the Yorùbá dataset, which is explained by the extremely high number of labeling functions (19897) and the smallest training dataset size (1340 samples) when compared to the other datasets. Results of other combinations are provided in Table 3. Two out of the four combinations achieve average scores of 67.6 and 67.1, demonstrating a better performance compared to the baselines. Although the Cosine contrastive self- training con- siderably improves the results, the ULF high per- formance does not rely solely dependent on it. This is evident in the fact that the most effective config- uration for all other datasets except TREC incor- porates the use of Cosine in only one of the two model training steps. Moreover, FT FT_ FT setting, which does not involve Cosine at all, also demon- strates compatible results across all datasets.
aeeeeooeee link
HAM SPAM
HAM
SPAM
HAM
SPAM
subscribe
0
0.19
0.81
0.20
0.80
weywordd please
0.16
0.84
0.15
0.85
keyword, song
0
1
0.25
0.75
0.27
0.73
regex, check out
0
1
0.18
0.82
0.25
0.75
short, correent
0
0.59
0.41
0.49
0.51
has person, nip textblob, polarity
0
0.12
0.88
0.09
0.91
textblab subjectivity
0
0.75
0.25
0.79
0.21
0
0.68
0.32
0.67
0.33
0.69
0.31
0.65
0.35
0
0.71 0.29
0.74 0.26
Figure 3: Transformation matrix with ULF after first and second iterations.
plcs where improved and gold labels do not match (i.e., where ULF, strictly speaking, failed). How- ever, these samples are quite controversial: e.g., (3) might be a spam message if the link was different (our model does not check the link's content), while (4) can be interpreted differently and perceived spam comment. Finally, (5), not covered by any LFs and initially randomly assigned to the HAM class, has been corrected to the SPAM class.
Case Study. We provide a YouTube dataset case study. Figure 3 shows the imitial and adjusted T matrices after two ULF iterations in FT FT setting. Some LI's underwent minimal adjustments (such Reyword_subscribe and regex_ ctwck_chccuuukeee which clearly corresponded one class), while contentious LFs (like short_comment, i.e., short comments are non-spam) were significantly ad- justed. The adjustment was slightly improved after the second iteration of ULF: however, a single it- eration was already sufficient for most of the set- tings, as demonstrated by our experiments (see Ap- pendix F). Table 4 shows mislabeled samples and their corrected labels after ULF application. In (1), the original equal voting was changed 1.58 for HAM and 2.42 for SPAM after T matrix correction, explicitly determining the label as SPAM. Simi- larly, labels assigned by a clear majority vote, such as (2), were also corrected. Next, there are sam-
Conclusion & Future Work
In our work, we focused on denoising WS data by leveraging information from LFs. Our approach assumes that the noise specific some LFs can be detected by training a model that does not use those LFss signals and then comparing its predictions to the labels generated by the held-out LFs. This idea is used in our method ULF, which improves the weak labels based on the data itself, without lever- aging external knowledge. Extensive experiments validate the effectiivness of our approach and sup- port our initial hypothesis of the significant role of LFs in denoising WS data. In future work, we plan to try ULF for other tasks, such as sequence tag- ging and image classification, and perform more experiments on weakly supervised datasets with different peculiarities and in different languages.
Sample
LF's matched
Noiey
Corrected Label
Gold Label
Label
Hello! I'm Marian wanted to play my own pop and pop- rock songs. It would mean a lot you could a look at my channel if u like, subscribe to it! XOXO THANKS!!
keyword my keyword subscribe
HAM
SPAM
SPAM
keyrorord song
textblob subjectivity
Nice song Sec my new track.
keywoord song
my
HAM
SPAM
SPAM
textblob subjectivity
'HAPPY BIRTHDAY KATY :) http:/ / wwwpg.cch..mcccctecccmmm /gtfts////////////ee//oee.csssss /gips//brttfaay... birthday- Gowers-happy-gvss YuuqiaX KaQxS/ fullscreen... ...
keyoord link
HAM
SPAM
HAM
textblob subjectivity
The little PSY is suffering Brain Tumor and has 6 more
months live. Please pray him and the best lucks.
keyword please subjectivity
HAM
SPAM
HAM
Follow me on Twitter @mscalifornia95
HAM
SPAM
SPAM
Table 4: Examples of label changes in the YouTube dataset after applying ULF

---

Limitations
References
Túlio C. Alberto, Johannes V. Lochter, and Tiago A. Almeida. 2015. Tubespam: Comment spam filtering on youtube. In Proceedings of the 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA).
In our work, did not focus on the task of creating labeling functions. Rather, our primary objective is to improve the model performance with a fixed set of already provided labeling functions, and to enable better generalization to new data.
Tiago A. Almeida, José María G. Hidalgo, and Akebo Yamakami. 2011, 1. Contributions to the study of sms spam filterin: New collection and results. In Pro- ceedings of the 11th ACM Symposium on Document Engineering, DocEng '11
All the datasets and their corresponding label- ing functions used in our experiments are weakly supervised datasets that have been extensively uti- lized in previous research. The provided labeling functions for these datasets, as well as other well- known weakly supervised datasets, considered reliable. ULF does not require the majority of LFs have high precision; however, if we consider a significantly different setting where the majority of labeling functions are highly unreliable (e.g., gener- ated by a noisy automatic process), cross-validation as done in ULF may not be as effective as in more standard WS setting.
Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi. 2020. Learning from rules gener- alizing labeled exemplars. In Proceedings of the 8th International Conference on Learning Representa- tions, ICLR 2020.
Benedikt Boecking. Willie Neiswanger, Eric P. Xing, and Artur Dubrawski. 2021. Interactive weak super- vision: Learning useful heuristics for data labeling. In Proceedings of the 9th International Conference on Learning Representations, ICLR 2021.
In our experiments, we restricted ourselves to NLP datasets and tasks, as creating labeling func- tions for weak supervision is particularly intuitive for language-rele related tasks. We leave the explo- ration of other data modalities for future research.
Salva Rühling Cachay, Benedikt Boecking, and Artur Dubrawski. 2021. End-to-end weak supervision. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Pro- cessing Systems 2021, NeurlPS 2021.
Oishik Chatterjee, Ganesh Ramakrishnan, and Sunita Sarawagi. 2020. Robust data programming with cerision-guided labeling functions. In Proceedings of the Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Thirty-Second Innova- tive Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educa- tional Advances in Artificial Intelligence, EAAI 2020.
Ethics Statement
While our method can lead to better and more help- ful predictions by the models trained on the noisy data we cannot guarantee that these predictions are perfect and can be trusted the sole basis for decision- making, especially in life-critical applica- tions (e.g. healthcare). Machine learning systems can pick up and perpetuate biases in the data, and if our algorithms are used for real-world applica- tions, the underlying data and the predictions of the resulting models should be critically analyzed with respect to such biases. We build our work on previously published datasets and did not hire annotators.
Siqi Chen, Jun Xiao, and Long Chen. 2023. Video scene graph generation from single-frame weak su- pervision. In The Fleventh International Conference on Learning Representations, ICLR 2023.
David P. A. Comey Dyaa Albakour, Miguel Martinez- Alvarez, and Samir Moussa. 2016 What do a million news articles look like? In Proceedings of the First International Workshop on Recent Trends in News Information Retrieval co-located with 38th European Conference on Information Retrieval ECIR 2016.
Acknowledgement
Surabhi Datta and Kirk Roberts. 2023. Weakly super- vised spatial relation extraction from radiology re- ports. JAMIA Open, 6(2). Ooad027.
We thank the anonymous reviewers for their constructive feedback. This research has been funded by the Vienna Science and Tech- nology Fund (WWTT))L047377//RGG000 "Knowledge-iffused Deep Learning for Natural Language Processing".
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers).

---

Jason A. Fries, Ethan Steinberg, Saelig Khattar, Scott L. Fleming. Jose Posada, Alison Callahan, and Nigam H. Shah. 2021. Ontology-driver weak supervision for clinical entity classification in electronic health records. Nature Communications, 12(1).
Curtis G. Northcutt, Lu Jiang, and Isaac Chuang. 2021. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research (JAIR), 70:1373-1411.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner. Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, higl--peformance deep learning library. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché-Bus Buc, E. Fox, and R. Garnett, editors, Advances in Neurul Information Processing Systems 32. Curran Associates, Inc.
Daniel Fu, Mayee Chen, Frederic Sala, Sarah Hooper. Kayvon Fatahalian, and Christopher Re. 2020. Fast and three-rious: Speeding up weak supervision with triplet methods. In Proceedings of the 37th Inter- national Conference on Muchine Learning, volume 119.
Michael A. Hedderich, David Ifcoluwa Adelani, Dawei Zhu, Jesujoba O. Alabi, Udia Markus, and Dietrich Klakow. 2020. Transfer learning and distant supervi- sion for multilingual transformer models: study on african languages. In Proceedings of the 2020 Con- ference Empiricul Methods in Natural Language Processing, EMNLP 2020.
Alexander Ratner, Stephen H. Bach, Henry R. Ehren- berg. Jason A. Fries, Sen Wu, and Christopher Ré, 2020. Snorkel: rapid training data creation with weak supervision. VLDB J., 29(2-3)
Michael A. Hedderich, Lukas Lange, and Dietrich Klakow. 2021. ANEA: distant supervision for low-resource named entity recognition. CoRR, abs/2102.13129.
Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christopher Ré. 2019. Training complex models with multi-task weak supervision. In Proceedings of the Thirty-Thine AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019.
Giannis Karamanolakis, Subhabrata Mukherjee. Guo- qing Zheng, and Ahmed Hassan Awadallah. 2021. Self-training with weak supervision. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021.
Anastasiaa Sedova, Andreas Stephan, Marina Speran- skaya, and Benjamin Roth. 2021. Knodle: Mod- ular weakly supervised learning with PyTorch. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepLANI.P-202)
Zorntssa Kozareva, Ellen Riloff, and Eduard Hovy. 2008, Semantic class learning from the web with hyponym pattern linkage graphs. Proceedings of ACI--08:
HLT. Association for Computational Linguistics.
Anastasiia Sedova, Lena Zellinger, and Benjamin Roth. 2023. Learning with noisy labels by adaptive gradient-based based outlier removal. In Machine Learn- ing and Knowledge Discovery in Databases: Re- search Track, pages 237-253, Cham. Springer Nature Swizzerland.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo Mendes, Sebastian Hell- mann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, and Christian Bizer. 2014. Dbpedia- a large- scale, multilingual knowledge base extracted from wikipedia. Semantic Weh Journal, 6.
Snorkel. Detecting spouse mentions in sen- tences. https://www.snort snorkel. wwwoorklllorgguuueeee org/use-casess spouse-dema, Accessed: February 2022.
Xin Li and Dan Roth. 2002. Learning question clas- sifiers. In Proceedings of the 19th International Conference on Computational Linguistics, COLING 2002.
Ang Sun and Ralph Grishman. 2012. Active learning for relation type extension with local and global data views. In Proceedings of the 21 ACM International Conference on Information and Knowledge Manage- meni.
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016. Neural relation extraction with selective attention over instances. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics [Volume I: Long Papers).
Christopher Teljstedt, Magnus Rosell, and Fredrik Jo- hansson. 2015. A aami-automatic approach for label- ing large amounts of automated and non-automated social media user accounts. In Proceedings of the 2015 Second European Network Intelligence Confer- ence.
Alessio Mazzetto, Dylan Sam, Andrew Park, Eli Upfal, and Stephen Bach. 2021. Semi-superviee aggre- gation of dependent weak supervision sources with performance guarantees. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Re- search. PML.R
Paroma Varma and Christopher Ré. 2018. Snuba: Au- tomating weak supervision to label training data. Proc. VLDB Endow., 12(3).

---

Sophia Y. Wang, Justin Huang. Hannah Hwang. Wendeng Hu, Shiqi Tao, and Tina Hernandez- Boussard. 2022. Leveraging weak supervision to perform named entity recognition in electronic health records progress notes to identify the ophthalmology exam. International Journal of Medical Informatics, 104:40
A.1 A detailed explanation of Z, T, and Y matrices
Figure 2 represents the weak annotation encoded with Z and T matrices. The matrix ZNxL rep- resents the information regarding the matches of labeling functions (LFs) in samples. For instance, in the case of keyword-base b-aaa LFs, an LF matches a sample if this keyword is present in this sample (this sample is then assigned to the class associated with this LF). This matrix is binary: if an LF l matches a sample 2;, Zij 0.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jin- feng Yi, and James Bailey. 2019a. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the 2019 JEEF/CVF International Conference on Computer Vision, ICCV.
Zihan Wang. Jingbo Shang. Liyun Liu, Lihao Lu, Ji- acheng Liu, and Jiawei Han. 2019b. Crossweigh: Training named entity tagger from imperfect anno- tations. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing, MANLL-IJCNLL 2019.
The matrix TLxk signifies the correspondence between LFs and classes. Original T matrix is binary: each element Twi represents whether an LF I corresponds to class 1. T& = 1 indicates that an LF 1 corresponds to class and assigns this class to all samples where it matches. For example, the keyword subscribe corresponds the class SPAM; any sample containing the word subscribe will receive one vote for the SPAM class.
T'zu-Tsung Wong and Po- Yang Yeh. 2019. Reliable ac- curacy estimates from k -fold cross validation. IEEE Transactions on Knowledge and Data Engineering. PP:1-1.
By multiplying the matrices Z and T and apply- ing majority vote, we obtain weak labels Y. The T* matrix denotes the improved version of the T matrix achieved through ULF. Note that the 7 matrix not binary anymore: instead of hard as- signments, contains soft ones. The improved, clean labels can be obtained by multiplying the improved matrix T with the original matrix Z.
Yue Yu, Simiao Zuo, Haoming Jiang. Wendi Ren, Tuo Zhao, and Chao Zhang. 2021. Fine-tuning pre- trained language model with weak supervision: contrastive- regulaiiieeegll self-training approach. In Proceedings of the 2021 Conference of the North American Chapter the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLT 2021.
Jun Yue, Leyuan Fang, Pedram Ghamisi, Weiying Xie, Jun Li, Jocelyn Chanussot, and Antonio Plaza. 2022. Optical remote sensing image understanding with weak supervision: Concepts, methods, and perspec- tives. I.E. Geoscience and Remote Sensing Maga- zine, 10(2).
A.2 Data Splitting into Folds for Cross-Validation
First, the training data is split into k folds for cross- validation training. We analyzed three possible ways of splitting:
Ziqian Zeng. Weimin Ni, Tianqing Fang. Xiang Li, Xinran Zhao, and Yangqiu Song. 2022. Weakly su- pervised text classification using supervision signals from language model. In Findings of the Associa- tion for Computational Linguistics: NAACL 2022.
randomly (ULFFrdm')) the samples are as- signed to folds the same way as would be done in standard k-fold cross cros-vvaldatiin irre- spective of LFs matching;
Jieyu Zhang. Linxin Song, and Alexander Ratner. 2023. I Leveraging instance features for label aggregation in programmatic weak supervision. In In Proceedings of the 26th International Conference Artificial Intelligence and Statistics (AISTATS) 2023.
by LF (ULFg)): the LFs are randomly split into ls folds {fi, fx. fx} and each fold fi is iteratively taken as held-out LFs, while others become training LFs. All samples where train- ing LFs match become training set Xtraing; the rest build hold-out set and are used for re-estimation X
Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yam- ing Yang, Mao Yang, and Alexander Ratner. 2021. WRENCH: A comprehensive benchmark for weak supervision. In fhirty-fifth Conference on Neural Information Processing Systems Datasets and Bench- marks Track.
A Details on ULF method
this section, we give a more formal description of the ULF algorithm as well as discuss its details.
Xtrain, Lez Ao =
(7)
Xout, = X Xeraing (7)

---

by signatures (0LF for each training sample 2i, we define its signature La; as the set of matching LFs. For instance, if LFs Ifi, and lfT match in a sample Di, its sig- nature is represented as (1, 4, 7). Next, the signatures are split into l folds, each of which becomes in turn test fold, while others com- pose training folds. All samples with signa- tures present in the training folds are consid- ered training samples Xtriin while the re- maining samples form the hold-out set Xout.-
Next, the confident matrix calibrated and nor- malized to QL x to correspond the data propor- tion in the Z matrix:
Qlais =
Crusss
Σ m=l
Zt ml
Σ
Clmmis
m==1
(11)
where EiEr, Cl,,d, n, Σ jEK
Zt
mm
2s
j=1
Algorithm 2: Train (X, Y) in feature-
based ULF
Xtrain; = {zi|La $ fi} Xout, = Lay E fi}
0 = AdamW(O, X, Y) Butput:Traines 0
After the data is split into folds, models are separately trained on Xtrain, E [1. k] and applied on the held-out folds Xout, to obtain the out-of- sample predicted probabilities PNxK.
Algorithm 3: Train (X, Y) in ULF with pretrained language model fine-tuning (optionally: with additional Cosine self- training step)
A.3 Out-of-sample Labels Calculation
The predicted probabilities P used to cal- culate the out-of-sample labels ÿ. The class with the highest probability selected:
Xiab = {i E X
LTx, 0}
Yos (ύ E Vo Xiab)
# 1. fine-tuue @ with Xiat and Yiao
0 AdamW(B, X Yint)
# 2. (optional) contrastive self-training of with X
0: arg max p(ū - o;; ISICK
Calculate pseudo labels !psd
if and only if the probability plū J; 0; ex- ceeds the class j average threshold ty:
for step ..., nun nun_steps do Select confident samples
6
:=== Σ NEXXy=j
0)/ (i==
7
Calculate classification loss Le ypaa),
(10)
contrastive regularizer Rı (0, Upsa),
confidence regularizer Rí (0)
In other words, the class average threshold I; is calculated by summing up the probabilities of the class for samples initially assigned to that class and then dividing it by the number of those samples. A sample 2; considered as belonging to the class j and only if this sample confidently belongs to the corresponding class. no probability exceeds the class thresholds for a sample (c.g., it belongs to all classes with equally small probabilities), is disregarded in further calculations unreliable.
(see Yu et al. (2021) for exact formulas and explanation)
T(0,, Upsa) Tug XR1 + R
@ = AdamW(0, X)
Output: Trained : 0
A.5 T Matrix Update
The joint matrix Q is used for improving the LFs-to- class matrix T that contains the initial AFs-to-class allocations. T and Q are summed with multiplying coefficients p and 1-p. where p E [o, 1]. The value of p balances the initial manual label assignment with the unsupervised re-estimation and determines how much information from the estimated assign- ment matrix Q should be preserved in the refined matrix T"
A.4 LFs-to-Class Estimation Matrix
To refine the LFs to class allocation, ULF re- estimates the joint distribution between matched LF's and predicted labels. For each LF I, and each class kj. the confidently assigned to the class k- samples with the LF 4 matched are calculated; the counts are saved as a L.Fs-confdent matrix CLxc:
===+Q++00 +
(12)

---

With the multiplication of Z and the newly re- calculated T matrices, an updated set of labels Y* is calculated. can either be used for rerunning the denoising process or training the end classifier.
each fold fi as test LF' and the others as training LFs. So, all samples where no LFs from hold-out fold match become training samples, while the rest are used for testing.
Xout X X Xtrain;
B Weakly Supervised Extension of Denoising Models
After that we train the A separate models on X Xtrain; and evaluate them on Xout. In the same way, as in the original CrossWeigh algorithm, the labels predicted by the trained model for the sam- ples in the hold-out set i are compared to the initial noisy labels y. All samples X; where Ú + Yj are claimed to be potentially mislabeled; their influ- ence is reduced in further training. The whole pro- cedure of error detection is performed t times with different partitions to refine the results. The sam- ple weights Wax then calculated as Wa, E where ej is the number of times sample 2j was classified as mislabeled, 0 S cj S t, and is a weight reducing coefficient.
To validate our hypothesis regarding the impor- tance of considering labeling functions in noise de- tection in WS data, we adopt two cross cross-validalee volssdaalitt based methods originally designed for denoising manually annotated data: Cross (Wang et al., 2019b) and Cleanlab (Northcutt et al., 2021). We adapt these methods for the weakly supervised set- ting and introduce our extensions: WSCrossWeigh and WSCIeanlab. A key difference between the original methods and our extensions is the approach used to split the data into folds for cross-validation training. Instead of random splitting, we split the data based on the funcllon that match in the samples.
functions
B.2 Weakly Supervised Cleanlab
In this section, we outline the original methods and introduce our WS extensions. Additionally, we conducted experiments, which are detailed in Appendix E.
The second method we introduce is Weakly Super- vised Cleanlab (WSCL) an adaptation of Clean- lab framework (Northeutt et al., 2021) for weak supervision. In the same way as in WSCW, not data samples, but the labeling functions into folds (fi...., fx} and used for building the X Xtrain; and Xout, sets, < i < k. for training models. In contrast to WSCW, for each sam- ple, o; the label is not directly predicted on the Xout,, but the probability vector of class distribu- tion p(y j; zi, 0).j C K is considered. The exact labels á are calculated later on with respect to the class expected self-confidence value t, (see Northcutt al. 2021):
B.1 Weakly Supervised Cross
split
The original Cross framework (CW, Wang et al. 2019b) was proposed for tracing inconsis- tent labels in the crowdsourced annotations for the named entity recognition task. After randomly splitting the data into ie folds and building & train- ing and hold-out sets, Cross Weigh additionally fil- ters the training samples that include the entities matched in hold-out folds samples. The intuition behind this approach is that if an entity is constantly mislabeled, the model would be misguided, but the model trained without it would eliminate this con- fusion. We consider this approach quite promising for detecting unreliable LFs in weakly supervised data similarly. If a potentially erroneous LF sys- tematically annotates the samples wrongly, reliable model-trained data without it will not make this mistake in its prediction, and, thus, the error will be traced and reduced. Our new Weakly Supervised CrossWeigh method (WSCW) allows splitting the data entirely randomly but considering the LFs that all LI's matched in a test fold are eliminated from the training folds. More formally, firstly, we randomly split labeling functions L into folds (fi,..,, f-) Then, we iteratively take LFs from
ty: Ex, ply j: 22,
(13)
where X,- - (x, E X
A sample is considered to confidently belong to class E K if the probability of class is greater than expected self-confidence for this class t, the maximal one in the case of several classes is probable:
01 = arg max p(y - jz (14) [E[A] #1y=j:r,0)24,

---

The samples with no probability that would ex- ceed the thresholds have no decisive label and do not participate in further denoising.
News Articles Dataset, which main task is to define whether there spouse relation in a sample. We use the Snorkel annotation (Snorkel); the labeling functions were created based on keywords (e.g.- husband), spouse relationships extracted from DB- Pedia (Lehmann et al., 2014), and language pat- terns (e.g., check whether the people mentioned in sample have the same last name).
After that, a class-to-class confident joint matrix Cyg is calculated, where:
Casbbilkk] Hr, E Xlu = =
Notably, Cz. contains only the information about correspondence between noisy and out-of- sample predicted labels (the same way as in North- cutt et al. (2021)). So, it gives the idea about the number of samples with presumably erroncous noisy labels but does not give us any insights about the erroneous labeling functions that as- signed this noisy label this sample (in contrast the ULF approach we present in Section 3).
(Li and Roth, 2002) A question classifi- cation dataset that maps each data sample to one of classes. The labeling functions were gener- ated based on keywords (Awasthi et al., 2020), e.g. which, what, located, situated keywords relate sample to the class LOCATION.
TREC
SMS (Almeida et al., 2011) A spam detection dataset comprised of text messages. The annota- tion proposed by Awasthi al. (2020) includes keyword and regular expression based label- ing functions. For example, a regex-based labeling function:
The confident matrix Cy.j then calibrated and normalized order to obtain an estimate matrix the joint distribution between noisy and out-of- sample predicted labels Qy.-- which determines the number of samples to be pruned. We perform the pruning by noise rate following the Cleanlab default setting: Qu..ū;? # j samples with mar(p(y j) p(y i)) are eliminated in fur- ther training.
I*) (won|won) [No]] ([*ss]+ )*
(claim, (claim)([\\]]H [$)
corresponds to class SPAM (e.g.. as in sample 449050000301 You have won ??2,000 price! To claim, call 09050000301.).
Datasets
In this section, we give a brief overview of the dataset and the examples of labeling function we used in our experiments. The dataset statistics is provided in Table 5.
Yorùbá and Hausa (Hedderich et al., 2020) Topic classification datasets of the second (Hausa) and the third (Yorùbá) most spoken languages Africa comprised news headlines, The weak key- word rules are provided by the authors.
YouTube (Alberto et al., 2015) A spam de- tection dataset was collected from the YouTube video comments. The samples that are not rel- evant to the video (e.g. advertisement of user's channel or ask for subscription) are classified as SPAM, while others belong to the HAM class. We use the same labeling functions as in Rat- ner et al. (2020); they were created using key- words, regular expressions, and heuristics. For example, labeling function KEYWORD_MY corre- sponds to class SPAM, meaning that if a sam- ple contains the word "my" will be assigned to the SPAM class. Among other labeling func- tions are SWWWRRDSBBSCRIBEE KEYWORD__SO LOYNG (keyword-based), CHORT_CCOMENNT (if comment is short, is probably not spam; thus, samples less than 5 words long would be classed HAM).
D Baselines
We compare our method ULF towards the most recent seakly-supervvseed baselines. Note that ULF does not use manually annotated data and can- not be directly compared to models that do (Kara- manolakis al., 2021; Awasthi al., 2020).
Gold A classifier is trained using the gold, man- ual labels. is the only model which is trained with manual labels in our experiments.
Majority Vote A classifier is trained using weak labels obtained by applying labeling functions to the samples, selecting the class with majority vot- ing, breaking the ties randomly.
SEYYWORD_PLEASS
MeTaL (Ratner et al., 2019) A classifier is trained with labels which are obtained by combin- ing signals from multiple labeling functions and training hierarchical multi-task network.
Spouse (Corney et al., 2016) A relation extrac- tion dataset based on the Signal Media One-Million

---

YouTube
Spouse
TREC
SMS
Yorúbá
Hausa
Train Data
1586
22254
4965
4502
1340
2045
Valid Data
150
2711
500
500
189
290
Test Data
250
2701
500
500
379
582
#Classes (K)
2
2
6
2
#LFs (L)
10
68
73
19897
18624
#Unlabeled
195
16520
242
2719
Avg LF Hits
1.6
33.7
1.7
0.5
3.0
2.9
LF Accuracy
81% 1 + 2.0
53% + 0.6
50% + 2.6
60% + 1.6
55% + 1.5
54% + 0.3
LF Coverage 87%
25%
85%
40%
100%
100%
Table 5: Statistics of the datasets. The LF accuracy metrics are calculated with a majority vote without any model training reported across 10 runs with standard deviation reduce the instability caused by randomly broken ties.
Snorkel-DP (Ratner et al., 2020) A classifier is trained using generative and discriminative Snorkel steps.
E Feature-based Experiments
In Section 4, we presented the results of our fine- tuning-based ULF implementation. In addition to it. also provide feature-based ULF implemen- tation that does not rely on pre-trained language models but can be run with an arbitrary model for the feature- based prediction.
FlyingSquid (Fu al., 2020) A classifier is trained using noisy labels that rectified exploit- ing an Ising model by triplet formulation.
We run the feature-based ULF experiments for four datasets: YouTube, Spouse, TREC, and SMS. The datasets selection motivated by previous work that includes feature-based methods (Zhang et al., 2021, 2023) and run experiments with these datasets. This choice allows for direct comparison of our results with those studies. (For the fine- tuning-based experiments, which typically yield better results, we expanded our experimental setup and included the low-resource language datasets Yorùbá and Hausa, Section 4.)
WeaSEL (Cachay et al., 2021) A classifier is trained using probabilistic encoder and a down- stream model combined with specifically defined noise-aware loss function. As WeaSEL is end- to-end system, were unable to include the Co- sine step it; that is the reason why (potential) WeaSEL+Cosine baseline is absent in Table 2.
FABLE (Zhang et al., 2023) A classifier is trained using noisy labels that are inferred lever- aging the instance features and the mixture coeffi- cients of the EBCC model.
We compare the (eature-based ULF approach to the same weakly supervised baselines used for com- paring the tme-tuning-based ULF approach. Those are: Majority Vote. MeTTaL (Ratner et al., 2019), Snorkel-DP (Ratner et al., 2020), FlyingSquid (Fu et al., 2020), FABLE (Zhang et al., 2023), and WeaSEL baseline (Cachay et al., 2021), which also uses logistic regression in the setting presented the original paper. Additionally, we include two methods for learning with noisy labels we already discussed in Appendix B Cross Weigh (Wang et al., 2019b) and Cleanlab (Northcutt et al., 2021), also reature-basee in the original setting together with weakly supervised extensions WSCross- Weigh and WSCleanlab. The feature-based ULF method was run in all possible data splitting set- tings: randomly (ULF)nam). by LFs (ULF)) and by signatures (ULF sgn). see Appendix A for more
In all baseline runs, we adhered to the hyper- parameters and their associated search spaces as suggested by the methods' authors. However, for some of the baselines, we have change the setting proposed in the original papers in order to provide a fair comparison. For instance, FABLE, we fine-tuned the hyper-parameters of the final clas- sifier training, similar to how we did for other baselines, even though the authors did not conduct such fine-tuning in their experiments and did not assume the presence of gold validation set. It is also important to note that due energy considera- tions and resource constraints, we retrieved the best hyper-paramett values by random search, rather than grid search. Consequently, for some of the results, direct comparison is not possible.

---

YouTube (Acc)
Spouse (FI)
TREC (Acc)
SMS (FI)
Avg
Gold
94.0 0.0
81.9 +
91.6 0.2
89.3
Majority Vote (MV)
90.7 + 1.4
24.0 + 0.1
57.6 + 5.0
90.8 + 1.0
65.8
MeTall (Ratner al., 2019)
89.8 0.8
21.8 0.8
59.5 +.8
89.1 0.9
65.1
Snorkel-DP (Ratner et al., 2020)
90.0 1 0.8
24.8
6..1
2.74
51.1
FlyingSquid (Fu et al., 2020)
87.7
2..
34.2
6.50
54.1
WeaSEL (Cachay et al., 2021)
5.2
2.4
2..6
2.10
28.4
FABLE (Zhang al., 2023)
91.1 0.2
2..6
46.0
64.7
56.3
CrossWeigh Weigh (Wang et al., 2019b)
90.1
0..6
40.1
79.6
62.8
Cleanlab (Northcutt al., 2021)
8.7
1.2
5..8
86.0 0.6
68.3
WSCrossWeiis Weigh
0.7
4.0
4..
65.7
WSCleanlab
87.2 = 0.4
0.7
58.9
86.6 0.3
69.3
WSCIcanlab 1gn
88.3 0.5
1.9
56.4
85.1 1 0.3
68.6
ULF endm
92.8 0.1
44.8 0.5
5.2
85.7 + 0.5
70.3
ULF Ifs
90.8 0.9
44.0 0.9
55.5
70.0 + 0.4
65.1
ULF sga
94.6 0.2
49.8 + 1.0
58.2
88.6 0.3
72.8
Table Results of the deature-basee ULF compared towards the feature based baselines. All results are averaged over 10 trials and reported with the standard error of the mean. The results marked with are taken directly from Zhang et al. (2021).
details. The feature-based ULF and the correspond- ing baselines realized in our experiments with logistic regression model; the training data are en- coded with TF-IDF vectors. The results of feature- based experiments are provided in Table
strated the best performance for feature-based ULF, was chosen for the fine-tuning- based ULF experi- ments (see Table 2).
F Implementation Details
Results. Our weakly supervised extensions to CrossWeigh and Cleanlab methods consistently outperform the base versions and most of other baselines, supporting our hypothesis of LFs' im- portance in applying cross-validation techniques to weakly supervised settings. Feature- based ULF also shows the best result overall on most datasets and even outperforms the model trained on YouTube data with manual annotations.
All our experiments used the validation set for hyper-parameter tuning, early stopping, and model selection. The gradient-based optimization was per- formed with AdamW Optimizer and linear learning rate scheduler. All results are reproducible with the seed value 1111.
ULF-specific parameter search space was de- fined heuristically. All parameter search spaces provided in Table 7. The number of iterations I was also estimated based on the validation set: ini- tially, is set
We also use feature-base based ULF to compare the data splitting strategies. UL.Frdn incorporates a standard cross-validation with random data split- ting, disregarding any signal originating from the weak annotation. This approach can be viewed for estimating the ULF performance alone, indepen- dent of any weak signals. The lower performance performance of ULFy, compared to other configu- rations may due multiple LF matches in many data samples, leading to multiple overlaps during cross-validation training (i.e., the samples were reestimated multiple times). In ULF wigns on the contrary, each data sample is considered only once in each denoising round resulting in better per- formance. However, even in the worst-performing settings (ULFy and ULF our (eature-basel ULF outperforms the majority of the baselines.
20, but if training labels do not change after three iterations, the algorithm stops, and the last saved model used for final testing. The actual number of iterations, alongside other hyper-paramette values, can be found in Tables 8 and 9. These tables show that single iteration yields the optimal result in most scenarios, with two iterations being the second most commonly selected option.
In fine- tme-tuning--asse ULF, in addition to other prper-parameters, we include the label prediction parameter: if it equals "soft", the probabilistic la- bels are used for training; otherwise ("hard") a label is the one-hot encoding of the most probable class (the ties are broken randomly).
In order to reduce computational load, we per- formed the random parameter search instead of the
The signature-bart based splitting, which demon-

---

grid search in all our experiments. Specifically, we tried 10 random parameter combinations from search space and selected the one which performed the best on the validation set. For feature- based ULF and corresponding baselines, the model with the retrieved best hyper- values was rum ten times with different initializations; the aver- age values with the standard enror of the mean reported. For fine-tuning-based ULF and corre- sponding baselines, the model with the retrieved best pyper-parameter values was run once on the test set; this value reported. The retrieved hyper- parameters are provided in Table for the best feature-based ULF setting, ULI'sng and in Table for all fine-tuning and cosine combinations of the fine-tuning-based ULF.
Feature-based ULF experiments were performed on a machine with a CPU frequency of 2.2GHz with 40 cores. Fine-tuning-based ULF experiments were run on single Tesla V100 GPU on Nvidia DGX-1. The full setup took 20 hours on average for each dataset for Seature-basee settings and 96 hours for based settings.
Both feature-based and iin-ttuiigg-baeed ULF were implemented with Python and PyTorch (Paszke al., 2019) in the setting of the weak su- pervision framework Knodle (Sedova al., 2021). By providing access to all WS components Kno- dle allowed us to implement and benchmark all algorithms described above. The pre-trained lan- guage models were downloaded from HuggingFace (https//huggingffcccco ctt.://cccccoooosss.....oo We followed the Wrench (Zhang al., 2021) encoding method and used their implementation for most of the baselines (apart from Cleanlab and Cross Weigh which are not included the Wrench framework).
Hyperparameter
Values
Feature-based ULF
Multiplying coefficient p
0.1, 0.2, 0.3, 0.4, 0.5, 0.7, 0.9 le-1, 1e-2, 1e-3, le-4
Learning rate Ir
Number of folds It
3,5,8, 8, 10, 15, 20 (w.nt. #LFs)
Number of iterations Non-labeled data rate
1,2, 3, 10
0, 0.5, 1,2.3
Fine-tuning base-d ULF
Multiplying coefficient
0.1, 0.3, 0.5, 0.7, 0.9 le-4, 1e-5, 1e-6 3,5,7 7 (w.nt. #LFs)
Learning rate Ir Number of folds Ie
Number of iterations I
2,
Confident regular. weight À
0.01, 0.1
The confident threshold
0.2, 0.4, 0.6, 0.8
Label prediction
soft, hard
based Table 7: Hyperparameter values tried a grid search feature-based ULF and fine-tuning based ULF.

---

YouTube
Spouse
TREC
SMS
Multiplying coefficient p
0.5
0.2
0.3
0.1
Learning rate Ir
le-2
le-2
le-1
le-1
Number of folds
3
3
10
Number of iterations I
1
Non-labeled data rate
3
0.5
Table 8: Feature-based Ul.Fsny selected hyperparameters.
YouTube
Spouse
TREC
SMS
Yorùbá
Hausa
Fine-tuning- ULF FT_FT selected hyperparameters.
Multiplying coefficient p
0.5
0.1
0.1
0.3
0.1
0.1
Learning rate Ir
le-4
le-06
le-05
le-05
le-06
le-06
Number of folds Number of iterations I
2
3
3
3
2
2
1
Confident regularization weight A
0.1
0.01
0.1
0.1
0.1
0.05
Confident threshold g
0.8
0.2
0.6
0.8
0.2
0.8
Label prediction
soft
soft
soft
soft
soft
soft
Fine- teee-tuning--ae -aased ULF FT cos selected hyperparameters.
Multiplying coefficient p
0.5
0.1
0.3
0.3
0.1
0.1
Learning rate Ir Number of folds Number of iterations I
lc-4
lc-06
lc-06
le-05
lc-05
lc-05
2
3
5
3
3
1
Confident regularization weight X
0.1
0.01
0.01
0.1
0.1
0.05
Confident threshold g Label prediction
0.8
0.2
0.8
0.8
0.2
0.4
soft
hard
soft
soft
soft
soft
Fine-tune tine-tuning-base ULF cos. FT selected hyperparameters.
Multiplying coefficient p
0.1
0.1
0.1
0.3
0.1
0.1
Learning rate Ir
le-06
le-06
le-05
le-06
le-06
le-05
Number of folds Number of iterations I
3
3
3
5
2
1
1
2
Confident regularization weight X
0.1
0.01
0.1
0.01
0.05
0.05
Confident threshold & Label prediction
0.2
0.2
0.6
0.4
0.4
0.4
soft
hard
soft
soft
soft
soft
Fine-tuning-based ULF cos selected hyperparameters.
Multiplying coefficient p
0.7
0.1
0.3
0.1
0.1
0.1
Learning rate Ir Number of folds &t
1e-05
le-06
le-06
1e-05
le-06
le-05
5
3
5
7
7
Number of iterations I
4
2
2
Confident regularization weight À
0.01
0.01
0.01
0.05
0.05
0.05
Confident threshold
0.2
0.2
0.8
0.4
0.4
0.4
Label prediction
soft
hard
soft
hard
soft
hard
Table 9; Fine-tuning-bused ULF selected hyperparameters.


Reviews:
[{"review_id": "nZWkGSNMYA", "reviewer": "Reviewer_D61f", "summary": "", "questions": "See weaknesses.", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! \n\n1. In line 71, we describe the majority voting as the simplest label aggregation approach and discuss its downsides in order to motivate *our* label aggregation approach ULF. We use majority vote *only* as a baseline and compare our method to it and 5 other label aggregation methods: MeTaL, Snorkel-DP, FlyingSquid, WeaSEL, and the most recent FABLE. One cannot compare ULF to ASTRA (Karamanolakis et al., 2021), SPEAR (Maheshwari et al., 2020), or ImplyLoss (Awasthi et al., 2020) as these methods use and rely on manually labeled data; those are semi-supervised methods that leverage weakly annotated data in addition to manually labeled data. On the contrary, ULF does not make use of any manual supervision at all and is applicable in actual weakly supervised settings, where a classifier can only be trained *exclusively* on the automatically (weakly) labeled data. We also point it out in lines 805-808: *\"...Note that ULF does not use manually annotated data and cannot be directly compared to models that do (Karamanolakis et al., 2021; Awasthi et al., 2020)...\"*. Therefore, we compare ULF towards the label aggregation methods that also do not use manually labeled data in order to provide a fair comparison. We will make this more clear in the final version of the paper. Also note that the methods we compare ULF to are more recent: WeaSEL (Cachay et al., 2021) and the most recent SOTA method FABLE (Zhang et al., 2023). \n2. We report all hyperparameters search range and selected values for all experiments, as well as other implementation details and computational requirements in Appendix G, Tables 7-9. We will consider using the additional page for moving this information to the main sections upon acceptance. \n3. Thank you for your suggestion to move Figure 2 into the Appendix, we will consider it for the final version. Figure 3 in the Appendix gives an additional illustration of the type of changes induced by the algorithm - since space is not a critical factor in the Appendix, we decided to show it there as we hope it can aid the understanding of ULF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Reasons to accept are:\n1. The paper is well-described and written.\n2. Results provided in the paper state that except one dataset, it has performed better than other approaches. However, there are many other methods (e.g. ASTRA, SPEAR, ImplyLoss) that should be added in comparison tables as these are the new state-of-arts.", "reasons_to_reject": "Reasons to reject are:\n1. In line 71, the authors only try majority voting as the label aggregation approach. However, many other label aggregation approaches, e.g. ASTRA, SPEAR, Imply Loss etc., choose the LFs sensibly in case of a tie by weighing the LFs based on previously correctly assigned labels. There is no comparison of the proposed approach with them as well.\n\n2. Experimental section does not have any information about computational requirements, and hyperparameters used and it just contains the description of the dataset. The section on the case study could be made small and more details about hyperparameter choices and the computational requirements could be added.\n\n3. Authors really need to manage the space as there are multiple redundant figures (e.g. figure 3 solely gives the interpretation of the changes in the label and figure 2 could be moved to the appendix or vice versa).", "questions_to_authors": "See weaknesses.", "paper_topic_and_main_contributions": "The topic of the paper is:\nUnsupervised Labeling Function Correction using Cross-Validation for Weak Supervision.\n\nThe main contributions of the paper are:\nThe paper proposes a method to denoise the weakly supervised data by providing a kind of weighting mechanism for Labelling functions. It optimizes the assignment of labeling functions (LFs) to classes by recalculating this allocation using well-established cross-validated samples known for their high reliability. However, there are many label aggregation approaches (e.g. ASTRA, SPEAR, ImplyLoss) already available that perform a similar kind of task. Hence the contribution might be incremental.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Not found."}, "extra_text": {"Missing_References": "1. Karamanolakis, Giannis, et al. \"Self-training with weak supervision.\" arXiv preprint arXiv:2104.05514 (2021).\n2. Maheshwari, Ayush, et al. \"Semi-supervised data programming with subset selection.\" arXiv preprint arXiv:2008.09887 (2020)."}}, {"review_id": "vrTP2Vz9i3", "reviewer": "Reviewer_jCVR", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review!\n\nThe results of the ablation experiments (i.e., the experiments with different combinations of cross-validation and final models) are provided in Table 4. For the best-performing setting, we also provide the information about the impact of each labeling function separately: see Figure 2 with the exact changes made by ULF (e.g., how each labeling function allocation was changed) and Table 3 with the recalculated final labels. \n\n\nTo offer deeper understanding regarding the influence of labeling functions in a different scenario, we now provide the transformation of the T matrix in another setting (i.e., cross-validation is done with the Cosine training step): \n\nT matrix after the first ULF iteration: \n\n| HAM | SPAM | \n|------|------| \n| 0.05 | 0.95 | \n| 0.06 | 0.94 | \n| 0.02 | 0.98 | \n| 0.03 | 0.97 | \n| 0.98 | 0.02 | \n| 0.03 | 0.97 | \n| 0.99 | 0.01 |\n| 0.99 | 0.01 |\n| 0.98 | 0.02 |\n| 0.95 | 0.05 |\n\nT matrix after the second ULF iteration: \n\n| HAM | SPAM |\n|------|------|\n| 0.09 | 0.91| \n| 0.10 | 0.90| \n| 0.02 | 0.98| \n| 0.07 | 0.93| \n| 0.91 | 0.09| \n| 0.10 | 0.90| \n| 0.93 | 0.07| \n| 0.94 | 0.06| \n| 0.94 | 0.06| \n| 0.90 | 0.10| \n\nHere, same as in Figure 2, each row corresponds to one labeling function (see Appendix B for the examples of labeling functions for this and other datasets) and each column - to one class.\nUsing a more powerful Cosine training makes the corrections to the T matrix much stronger and more intense; in our case, these stronger corrections end up causing more problems (compared to the simple fine-tuning) instead of improvements, as also demonstrated by the numbers in our experiments. Conversely, incorporating Cosine training into the end model training proves advantageous and enhances the final results. This analysis will be added in the camera-ready version. \n\nAlso, please note that in order to ensure the reproducibility of our experiments, we report the settings of parameters (i.e., all hyperparameters search range and selected values for all experiments) in Appendix G, Tables 7-9.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper proposes a novel approach to enhancing the quality of weak labels using k-fold cross-validation.  \n- They introduce a method called Unsupervised Labeling Function(ULF) correction that successfully labels samples with no labeling functions matched. \n- The work clearly outlines the motivation for re-estimate the joint distribution between labeling functions and class labels and carry out experiments evaluating the same. \n- The authors conduct extensive experiments using feature-based and pre-trained models to demonstrate the effectiveness of their method \n- The results show that the proposed solution performs comparably or better than the current solutions/baselines. \n- Overall, the paper is well-written, and the authors provide a clear and concise explanation of their approach and results. ", "reasons_to_reject": "- The work does not have any ablation experiments to understand the impact of different types of labeling functions \n- The work would benefit from a discussion section that analyzes the results.  ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposes a novel approach to enhancing the quality of weak labels using k-fold cross-validation. Specifically, the authors introduce a new method called ULF (Unsupervised Labeling Function correction) that successfully labels samples with no LFs matched. The authors conduct extensive experiments using feature-based and pre-trained models to demonstrate the effectiveness of their method. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zTjeuH6W54", "reviewer": "Reviewer_PGPZ", "summary": "", "questions": "author mentioned cosine training on page 4, but not explanation or reference given. \n\nAuthor replied to this.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review!\n\nWe do not reference the Cosine paper on page 4 because we have already referenced it on page 3, line 199. The explanation of the Cosine training setup is provided in Appendix B, Algorithm 3. We will consider moving it to the main paper when we have an additional page.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Although with limitations,  overall they have presented a study with clear and comprehensive explanation of their methodology and experimental setup. The authors have provided a thorough comparison of their approach with baselines and existing methods, which adds value to the paper and strengthens its contributions.  i think this work contributes to the field with novel findings that are likely to be of interest to the scientific community.", "reasons_to_reject": "no", "questions_to_authors": "author mentioned cosine training on page 4, but not explanation or reference given. \n\nAuthor replied to this.", "paper_topic_and_main_contributions": "This paper is about modeling soft labeling function (or label function denoising) for a weakly supervised setting, i.e., to learn/update the label functions using k-fold cross validation. They normally started with hard label functions like keyword-based, or regular expression-based or heuristics).  The assumption is the noise specific to some label functions (e.g., keywords, regular-expression-based) can be detected by training a model that does not use those label functions signals and then comparing its predictions to the labels generated by the held-out label functions.    \nThey proposed an interesting new method, i.e., the first to adapt cross-validation denoising methods to WS problems and refine the LFs to class allocation in the WS setting.  I appreciate the authors' efforts in providing detailed and well-document appendix to ensure the reproducibility of their work", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "for cosine training, any reference?", "Typos_Grammar_Style_and_Presentation_Improvements": "overall is ok, easy to read and follow."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces an approach for denoising weakly supervised data, particularly through the introduction of a weighting mechanism for labeling functions and the Unsupervised Labeling Function (ULF) correction. Based on the discussions during the rebuttal phase, below is the summary of the main aspects of the work. 

Reasons to Accept:

Clarity and Presentation: The paper is well-described and well-written. The clear and organized presentation enhances the overall understanding of the research, making it accessible to a broad audience.

Competitive Performance: The paper presents compelling results, demonstrating that, with the exception of one dataset, the proposed approach consistently outperforms other existing methods. This highlights the effectiveness and competitiveness of the proposed method.

Motivated Research: The work is well-motivated and provides a clear rationale for the need to re-estimate the joint distribution between labeling functions and class labels. The experiments conducted effectively evaluate this motivation, strengthening the paper's credibility.

Extensive Experimentation: The authors have conducted a comprehensive set of experiments, including feature-based and pre-trained models, to demonstrate the effectiveness of their method. This thorough experimentation adds significant value to the research. The appendices provide extra analyses and results.

Comparative Performance: The results consistently indicate that the proposed solution either performs comparably or surpasses current solutions and baselines. This underscores the competitiveness and utility of the proposed approach.

Reasons to Reject:

Deeper Discussion Section Needed: While the paper excels in its description and presentation, it would significantly benefit from a more extensive discussion section. A deeper analysis of the results, including their implications and nuances, is essential to provide readers with a more comprehensive understanding of the research.

Future Directions and Next Steps: As a short paper, considering the inclusion of a section that outlines potential future research directions and next steps would be valuable. Identifying areas where the proposed approach could be extended, improved, or applied would enhance the paper's significance.

In conclusion, there is convergence on the reviews considering that the paper exhibits substantial promise with its clarity, strong presentation, innovative approach, motivated research, extensive experimentation, and competitive performance. To further elevate its contribution and overall quality, it is recommended to expand the discussion section and consider adding a section on future research directions.
Decision: Accept-Main