Characterizing the Impacts of Semi-supervised Learning for Weak Supervision
Authors: Jeffrey Li
Jieyu Zhang
Ludwig Schmidt
Alexander Ratner
Venue: NeurIPS 2023 poster (2023)
Keywords: Weak Supervision
Semi-supervised Learning
Learning From Limited Labels

Abstract: Labeling training data is a critical and expensive step in producing high accuracy ML models, whether training from scratch or fine-tuning. 
To make labeling more efficient, two major approaches are programmatic weak supervision (WS) and semi-supervised learning (SSL). More recent works have either explicitly or implicitly used techniques at their intersection, but in various complex and ad hoc ways. In this work, we define a simple, modular design space to study the use of SSL techniques for WS more systematically. Surprisingly, we find that fairly simple methods from our design space match the performance of more complex state-of-the-art methods, averaging a 3 p.p. increase in accuracy/F1-score across 8 standard WS benchmarks. Further, we provide practical guidance on when different components are worth their added complexity and training costs. Contrary to current understanding, we find using SSL is not necessary to obtain the best performance on most WS benchmarks but is more effective when: (1) end models are smaller, and (2) WS provides labels for only a small portion of training examples.

Characterizing the Impacts of Semi-supervised Learning for Programmatic Weak Supervision
Jeffrey Li", Jieyu Zhang', Ludwig Schmidt', Alexander Ratner University of Washington, Snorkel Al
(jw12162, jieyuz2, schmidt, ajratner)0cs washington edu edu
Abstract
Labeling training data is critical and expensive step in producing high accuracy ML models, whether training from scratch or fine-tuning. To make labeling more efficient, two major approaches are programmatic weak supervision (WS) and semi- supervised learning (SSL). More recent works have either explicitly or implicitly used techniques at their intersection, but in various complex and ad hoc ways. In this work, we define a simple, modular design space to study the use of SSL techniques for WS more systematically. Surprisingly, we find that fairly simple methods from our design space match the performance of more complex srate-of-the-ee methods, averaging 3 p.p. increase in accuracy/FI-scor across 8 standard WS benchmarks. Further, we provide practical guidance on when different components are worth their added complexity and training costs. Contrary to current understanding, we find SSL is nol necessary to obtain the best performance on most existing WS benchmarks but is more effective when: (1) end models are smaller, and (2) WS provides labels for only a small portion of training examples.
Introduction
Learning with limited labels is fundamental challenge in machine learning (ML) applications To address the significant costs of hand-labeling training sets, programmatic weak supervision (WS) has emerged as a high impact research area (25] B1), where the aim is to learn from multiple cheaper sources of noisy labels. Meanwhile, semiisupervised learning (SSL.) is more classical direction with similar high-level motivations. Instead of generating larger quantities of noisy labels, SSL aims to directly leverage additional unlabeled data. It seems natural that these two fields could be applied productively with one another, yet their intersection has not been systematically studied.
12
35
In this work, we anchor WS approaches and study whether they can be enhanced using techniques from SSL. At a high-level, most WS methods consist of two steps. First, a label model aggregates set of weak label sources to noisily label training set. Commonly, these sources take the form of wrirteritten heuristics (e.g., for sentiment analysis, a user may check for the keyword "great" to provide the label "positive")). Second, an end model is learned on this training set. Crucially, weak sources can often abstain (e.g., the absence of "great" may not imply "negative"". leaving certain examples remain unlabeled and thereby unused. This presents a natural opportunity use SSL.
Indeed, this approach of using SSL in WS settings has motivated several recent methods 30 26 IT [
Though these works often attribute their observed improvements to their usage of unlabeled data B3 30 26). they also incorporate various algorithmic components in addition to SSL. Thus, we lack clarity about the precise contributions of SSL and whether simpler methods might also suffice. Also, while varying the amount of unlabeled data is crucial when evaluating SSL techniques
(22), previous WS benchmarks contain little diversity along this key dimension. Here, conduct more systematic study of how useful SSL is in various WS settings, as well as how and when to best employ it.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Specifically, we first organize the intersection between SSL and WS by proposing an explicit design space, centered around disentangling the following key methodological considerations:
(1) Thresholding: What to treat as (un)labeled? Because WS uses heuristics for labeling, often only provide labels for subset of examples, leaving the rest as unlabeled. Further, it can also be beneficial to additionally remove some (likely) incorrect labels provided by WS, shown by [5]. Since unlabeled data can result in multiple ways when using WS, we view "what to treat as unlabeled" as a non-trivial and first-class axis in our design space.
(2) SSL Technique: How to use unlabeled examples? After deciding what data should be treated as unlabeled, one can leverage these examples by simply using any existing SSL technique. Most recent proposals do so via self eeff-training, which uses the end model to periodically provide labels for unlabeled examples; we also try other out-of-the- the--oxeeoxx SSL methods.
(3) Re-labeling: Whether to update weak labels during end model training? Because some labels from WS are incorrect, it can be helpful re-label a training set with the end model during training. This approach, increasingly used by WS methods [30 [T 4. is often packaged as part of self- sefftrraniigg-bseee SSL.. Here, we identify that re-labeling and SSL can be independently employed, and we aim to disentangle their impacts.
With our design space, we can organize previous works and modularly generate a variety of methods. We test these methods on several standard WS benchmarks, finding that our design space is sufficient for matching the performance of more complex ofte-of-theear methods. We then compare methods within our design space to ablate the importance of each axis and so provide guidance on when each is worth using (and thus more carefully exploring). We summarize our key findings as follows:
By searching over our design space, we identify two methods that at least match all previous baselines on of WS benchmarks, averaging a 3 p.p. increase in accuracy/F1-score
While previous works emphasize utilizing data left unlabeled by WS sources, we find that on 6 of 8 benchmark tasks, SSL is actually not necessary for achieving high performance: thresholding and re-labeling can recover 90% of the gains achieved by also using SSL.
To explain SSL's lack of impact, we find that the small amounts of unlabeled data in these 6 benchmarks (i.c., < <11%) are largely unnecessary for the underlying tasks; when using clean instead of weak labels, ignoring unlabeled examples drops test accuracy by 2.5 p.p.
In contrast, when WS sources leave more data as unlabeled (i.e., > 65%), SSL is generally worth prioritizing: using it can improve upon thresholding and re-labeling by up to 16 p.p.
2 Related Work
SSL. for WS tasks. Methods in WS have increasingly turned to SSL to improve end model training. This includes DENOISE (26). which incorporates the temporal ensembling SSL algorithm [14, as well as COSINE 33 and KeyClass ()) which both use self- self-training [16). Other works [TT 18
6 20
19 23
apply SSL when learning from weak labels plus small set of clean labels. However, these split based on cleanly versus weakly labeled examples. Further, integrating clean labels into WS enables a greater variety of specialized strategies, so we do not consider this setting in our study. Likewise, [2] assumes additional supervision via heuristics for which examples share the same labels, similar in spirit to consistency-based SSL methods. Finally, we defer a background on SSL in its own right to Section 3
methods fundamentally differ in their use of SSL, i.e., they define a
SSL for learning with noisy labels. SSL. techniques have been regularly applied in the literature concerning learning from noisy labels [17 7 [3]. Though this setting is similar to WS, its main difference is that label noise comes from single "black-box" noising process instead of from multiple explicit WS sources. Thus, the resulting label noise patterns, often also artificially injected in input-independent ways (28), may differ significantly from those in WS. Furthermore, the WS setting contains some examples with no labels since WS sources may abstain.
Subset selection in WS. [15] showcases the broad utility of more carefully selecting subsets of weak labels before end model training. In our work, we consider subset selection in the greater context of two other trends in the WS literature, applying SSL and re-labeling. Compared to the core method of [5), we also try simpler baseline, similar to [9], based on thresholding the existing confidences produced for weak labels. We find that this method offers competitive alternative on most datascts.

---

Labeled Set L (2/ P3)
End Model (DM)
Label Model (LM)
Weskly Labeled Set D [xp, 3,111
Thresholding
Unlabeled Set U [x(] [
SSL Technique
Re-labeling
Thresholding
Coverage, Confidencs, Out-baaed)
SSL Technique
None, EM, VAT, UDA, Seif-Training, Self-Training (DM as LF)
Re-labeling
None,, directy,, LF!
Figure I: Overview of our design space. In WS, a label model produces a weakly labeled training set D which is used train a discriminative end model. Our design space overlays three decision points this pipeline: (1) thresholding, which filters out and unlabeled sets L and U: (2) SSL technique, which defines how the end model can still use U during training; (3) re-labeling, which uses the end model to update previously used weak labels.
weak labels in 1 to return labeled
3 Design Space
In this section, we first formalize WS and SSL. Then, we describe how our design space overlays on WS, detailing its three key axes along with the specific instantiations of each that explore in experiments. Finally, we contextualize which parts of existing work fit within our framework.
3.1 Problem Formalization
Weak supervision (WS) starts with an unlabeled training set D {x: n drawn from an underlying distribution (æx, y) ~ Labels are provided by set of labeling functions (LFs) (A₂) (x,)
where each Xy : (M) (0} either labels or abstains (denoted as ,y(x,,) 9) on each 2;. The goal is to train discriminative end model (DM) f X + y that performs well on P. Canonically, WS
methods contain two components. First, label model (L.M) observes (D, {},})=]] and outputs
weakly labeled training set Ó {(x;, D:) Second, is trained using D. Often, the LM models i/ probabilistically, allowing D: to soft-label, vector of probabilities over )
Problematically, ò may contain several uncovered examples, where x (xx) for all Ay. Thus, the default practice is then to train ff only on covered examples that received at least one non-abstaining LF vote. Concurrently, may also contain several incorrect labels, where ūr * (or in the soft-label case, arg m [4[c] * yt where ÿ is the probability assigned to class c). Both issues may lead to sub-optimal performance compared to standard supervised learning. Here, we focus on whether SSL lets us more effectively learn from Ó in light of these challenges.
Semi-supervised learning (SSL) assumes access labeled dataset T. {(ff, 1O as well as an unlabeled dataset = {xx] 10| The goal is to obtain a model that performs well on L's underlying distribution despite the limited size of L. Though not strictly required, it is often assumed that | << [U] and both are drawn from the same test distribution. Generally, each SSL method makes assumption about how P(x) relates to P(y(z). Popular categories of methods include mutropy-minimization [TO, 16 which assumes P(y|z) is uncertain only when P(x) is small, and
consistency regularization 21 27 14 which assumes local smoothness of P(y(æ) when P(x))
3.2 A Simple Design Space
By considering the standard WS pipeline, we anchor our design space on three natural decision points as shown in Figure (1) thresholding strategy, (2) SSL technique, and (3) whether to re-label previously used weak labels. Importantly, our design space agnostic to the specifications of label model and end model, though particular choices for each could affect which methods work best.

---

Method
Thresholding
SSL Technique
Re-labeling F.
Novel LM
Add. Reg.
Add. Labels
Vanilla WS
Coverage
Cuistar [1]
Cut-based (onc-time)
Denoise [2]
Coverage
Tomp Ensembling
Self-Train LM
Weasel
Agrsement-based
Cosine [4]
Conlidence (dyn.)
SelF-Train
DM directly
KeyClass [5]
Confidence (dyn.)
Self
directly
ASTRA 11
Coverage
Teil-Train (DM as LF)
DM LF
LPN-WL 22
Label Progugation
SPEAR 18
Entropy Min
Agreement--ased
Table I: Contextualizing methods within (middle columns) and outside (rightmost) our design space. "Novel LM" refers a method introducing its own label model. "Add. Reg." refers using additional regularizations, such as contrastive losses and soft-label re-normalization. "Add. Labels" refers to assuming an additional set of clean labels. For methods using clean labels, we do not compare to their results directly, but we can still place elements of their approaches in our design space.
Thresholding. To apply SSL, we must first define which examples in D should be considered as part of L and U, respectively. Though this is baked into the problem definition in traditional SSL settings, it is a non-trivial choice in WS settings. As a default, standard WS pipelines select based on labeling function coverage. ignoring uncovered examples on which all X abstain. However, as shown by (15), removing additional examples from D can help if they are more likely to be incorrectly labeled. In our work, we consider the following strategies for partitioning δ into L and U:
Coverage-based (default): This removes uncovered points {(1; (t) Ay(xx)) vj)-
Confidence-baat -aaee: Since most label models can output probabilistic labels, a basic yet under- explored approach tried by [9] is to remove examples that have low estimated confidence, here defined as the highest probability assigned to a class. Formally, this removes the examples (( ūt): max.ey D: [c] < }} for some threshold | < <<<
Cut-based: This the method of [33, which at high-level removes examples whose labels differ most from those of their nearest neighbors in some pre-trained embedding space.
SSL Techniques. After partitioning î into I, and U. we next consider how to still make use of the examples in U to train better end models. By default, WS methods ignore U altogether, but we may instead apply a variety of strategies, such as plugging in any existing SSL technique. Like [22,, we limit ourselves to SSL techniques that add unsupervised loss terms during training since these methods tend to achieve the best performance on traditional SSL benchmarks. We also pick representative methods from their taxonomy of approaches:
Entropy minimization (EM): A classical approach, EM penalizes uncertain predictions unlabeled examples, aiming to place decision boundary in low-density regions of P(z).
Self-training (ST): Self-training, along with the closely related pseudo-labeling [66] 3) are traditional SSL methods that itcratively use current predictions on unlabeled examples as true labels. Uniquely in the WS setting, we also consider self-training the label model and end model together. Inspired by [1. we try one way to do so by feeding the end model as an additional labeling function Am+1 used to re-fit the LM. We call this ST (DM as LF).
Consistency regularization: These methods encourage similar predictions on (realistic) per- turbations of unlabeled examples. We use VAT (2 does not rely on data augmentations, which are less straightforward for text datasets, the majority of WS tasks. On some tasks, we are also able to use UDA 29 generating perturbations via En-De backtranslations.
Re-labeling L. Traditional SSL assumes that all labels in L are correct. However, in WS, we may also consider using the end model to correct labels in L as it trains. This increasingly popular technique in WS methods 4 30 TT 18 is often packaged with traditional self-training as an overall "SSL method" (30 III instead of using the current model to label just U, these methods also do so for L. However, this makes it difficult to discern whether these methods improve performance because they leverage U (i.e., apply SSL) or simply because they clean up labels in L. Therefore, in our study, we explicitly decouple the use of SSL and re- labeling L as two independent decisions; i.e., we can re-label I. regardless of whether we self-train on U, or indeed use any specific SSL technique.
Specifically, we consider two forms of re-labeling analogous to the two types self-training for SSL:: (i) using the end model's predictions directly; (ii) using an LM re-fit with the end model as an LF. For

---

Datasets
Train
Test
# L.Fs
Coverage
Snork. Precision
Metric
IMD6
87.6%
74.4%:
Yelp
2
30400
3800
8
82.8%
75.4%
Acc.
Youtube
2
1586
250
10
87.7%
87.0%
Acc.
NgNews
4
96000
12000
9
69.1%
82.5%.
Act.
Tree
6
4965
500
68
95.1%
60.0%
Acc.
Spouse
22254
25.8%
65.6% Val)
(binary)
Chemprot
10
1286
1607
26
85.6%
58 58.0%
Acc.
Census
10083
16281
83
99.1%
58.0%
(binary)
Table 2: Statistics for the WS benchmarks that we use. Groupings are by task type: text, relation, and
tabular classification. Coverage is the percentage of training examples on which at least one LF does not abstain. "Snork. precision" refers to the accuracy of the Snorkel LM on the covered set of inputs.
tractability, however, we only consider re-labeling L in manner (i) when using SSL, except when the SSL technique is already ST (DM as LF). Also, since re-labeling effectively loops the two-stage WS pipeline back onto itself. N rounds of re-labeling could in principle be paired with N + 1 separate choices for thresholding and SSL. However, to reduce this search space, we only ever apply the same SSL method across all rounds. For thresholding, we try two possible schedules: dynamic, which applies the same thresholding after each re-labeling round and one-time, which fixes the L/U split for all rounds of end model training. Finally, some recent methods re-label by using specialized LM that can jointly learned with the end model 4 18 We do not include this type of agreement-bur based re-labeling in our study, instead focusing on methods agnostic to the form of L.M.
3.3 Contextualizing previous works
With our design space, we can contextualize several recent WS methods, as shown in Table Overall, this table demonstrates the lack of systematic exploration. For instance, few works perform any thresholding beyond coverage-based, and only [T3] and [9] threshold initial LM outputs. Furthermore, assessing the impact of SSL in WS settings is muddled because methods often incorporate several different techniques. A salient example is that COSINE (30) was found by WRENCH [33] to obtain state-of-te theth--rt performance, with both works championing the usage of unlabeled data as a key driver of improvements; however, what COSINE refers to as self-training actually involves pseudo- labeling the whole dataset, thereby performing both SSL. and re-labeling. Further, many methods use techniques outside of our design space entirely, such as novel LMs [26]
[8] contrastive learning [30), and soft-label re- n--ommllization [30 9. As a result, it remains unclear whether our three axes are necessary or even sufficient to achieve optimal performance. Though our design space is by no means exhaustive, we believe it offers a useful starting point to answer such questions.
4 Results
We begin by describing our experiment setup and various baselines in Section 4.1 Then, Section 4.2 explores how our design space yields methods that perform least as well as the aforementioned baselines. In Section 4.3 and Section [4.4] we conduct extensive ablations on our three axes, finding that SSL is surprisingly unnecessary on most existing WS benchmarks. Finally, Section 4.5 explains this phenomenon and explores settings in which SSL is more helpful.
4.1 Experiment Setup
Datasets and models. We use classification datasets (see Table 2 and largely follow the end model configurations from WRENCH (33) with a few changes the hyperparameter grid (Appendix (A). For NLP tasks, we both fine-tune RoBERTa pre-trained models and train MLP classification heads on (frozen) RoBERTa embeddings, deferring all results for the latter to the appendix. For tabular tasks, we just train MLPs on raw features. We tune all methods on a shared hyperparameter budget of 50 trials for full RoBERTa fine-tuning and 300 trials for MLPs. All reported test performances are then averages over over three additional test runs, while all error bars are the standard deviations over these runs. Finally, though our design space is compatible with any LM,, we use the soft-labels produced by the Snorkel LM from 24 However, we test robustness to this choice by also trying Majority Voting when comparing with existing methods. From the conclusions of 33 these were most consistently the best LMs across many benchmark tasks.
5

---

0.06
IMDD RoBERRa (Acc)
0.06
Yelp-RoBERTa (Acc)
0.83
RoBRib (Acc)
0.03
AGNews ERTa (Acc)
0.02
0.01
0.00
0.03
0.00
0.00
-0.02
TREC-ROBERTa
RoBERTa binary)
Chemprot-R@EERT
0.15
MLP (F1,
0.10
0.28
0.00
2.03
0.13
0.52
0.05
0.02
0..5
Design Space Conf + ST (DM as LF) Re-label
COSINE Snorkel
Weasel
Figure Test performance for LM Snorkel. We compare the best single method found (for each end model) from our design space (green) with the best methods found per-dataset (turquoise, narrower error caps) as well as four recent methods from the literature. All performances are plotted relative to that of vanilla training (deferring absolute metries to Table O in Appendix [B. Note that for Census, tabular task, the best single method different since we report the one we found for MLPs.
Baselines. We consider five baselines, corresponding to the WRENCH implementations of existing state-of-the-art methods. As Table shows, each incorporates some aspects of our design space, while not necessarily being contained completely within it. We list these methods as follows:
Vanilla: applies the default WS pipeline, using coverage thresholding and SSL re-labeliig.
Cutstat (Snorkel, MVJ: uses the thresholding method from [15], with no alterations. Notably, this method is contained within our design space as pure thresholding method.
COSINE (Snorkel, MV): modifies [30] in ways when appropriate for fair comparison: the LM used is Snorkel instead of MV except as shown in Table 7 and the DM is an MLP in Table
Denoise + /Snorkel, MV/: applies 26 except using Snorkel instead of MV to initialize their label
aggregator as shown in Table 7 Also, we use RoBERRaa-based end models instead BERT.
Weasel: applies the method from
except to train our RoBERTa- end models for text datasets.
4.2 Does our design space yield competitive methods?
We first demonstrate that simple methods from our design space can at least match the performance of all previous methods. When the end model is RoBERTa, our best single method, shown in Figure 2 is the combination of (dynamic) confidence thresholding, ST (DM LF), and re-labeling, This method at least matches the other baselines on 5 of the 7 tasks. On Chemprot, only Cutstat results in better point estimate. However, we may also swap cut-based thresholding into our method, which results -fate-of-the-ee art accuracy of 0.601 (0.008) (as shown in Appendix B this method is similarly strong when the LM is instead Majority Voting). Finally, FigureBillustin space exhaustively per while not necessarily practical-can be even more powerful, improving RoBERTa fine-tuning performance by 1.6 p.p. on average and closing 16% the remaining gap between our baselines and fully supervised training. Significantly, our methods are strong despite not using additional techniques employed by recent works (as shown in Table [ This provides empirical justification for focusing only on methods from within our design space in subsequent analyses.
that searching our design
4.3
Which axes are most important on current benchmarks?
Though our design space is sufficient for strong performance, we now show that SSL is, by and large, not necessary current benchmarks. Specifically, we conduct an extensive ablation comparing all eight possible subsets of including (or ignoring) each axis, presenting the results in Tables and As an example, for the row "Thresh SSL," every combination within the cross pross-product of

---

Method
IMDb
Yelp
Youtube
AGNews
TREC
Spouse
Chem.
Mean
w/o Spouse
Vanilla
0.874
0,925
0,867
0.646
0.215
0.572
0.805
(0.002)
(0.017)
(0.009)
(0.008)
(0.01 l)
(0.012)
(0.002)
(0.004)
(0.004)
0.885
0.946
0.951
0.882
0.706
0.263
0.587
0.746
0.826
Thresh Alone
(0.002)
(0.006)
(0.002)
(0.001)
(0.015)
(0.071)
(0.005)
(0.010)
(0.003)
SSL Alome
0.885
0.941
0.955
0.885
0.660
0.343
0.593
0.752
0.820
(0.002)
(0.008)
(0.005)
(0.002)
(0.074)
(0.075)
(0.010)
(0.015)
(0.013)
0
0,944
0.952
0.877
0,700
0.332
0.577
0.755
0.825
Re-label Alone
(0.013)
(0.004)
(0.006)
(0.002)
(0.014)
(0.105)
(0.010)
(0.015)
(0.004)
Thresh SST.
0.885
0,959
0.943
0.887
0.732
0.529
0.597
0.790
0.834
(0.018)
(0.002)
(0.008)
(0.005)
(0.014)
(0.057)
(0.009)
(0.009)
(0.004)
Thresh Re-label
0.905
0
0.952
0,883
0.748
0.236
0.600
0.755
0.841
(0.011)
(0.002)
(0.006)
(0.005)
(0.023)
(0.085)
(0.006)
(0.013)
(0.005)
SSI. Re-label
0.894
0,953
0,951
0.882
0.702
0.365
0.583
0.761
0.828
(0.008)
(0.008)
(0.002)
(0.001)
(0.035)
(0.073)
(0.003)
(0.0122
(0.006)
Thresh + SSL + Re-label
0.907
0,965
0.949
0,885
0.765
0.531
0.601
0.800
0.845
(0.004)
(0.002)
(0.010)
(0.003)
(0.012)
(0.039)
(0.008)
(0.000)
(0.003)
Emtree Design Space
0.907
0,965
0.952
0.885
0.765
0.531
0.601
0.801
0.846
(0.004)
(0.002)
(0.006)
(0.002)
(0.012)
(0.039)
(0,008)
(0.006)
(0.003)
Fully supervised
0.932
0.967
0
0.966
0.894
0.943
(0.005)
(0.001)
(0.013)
(0.006)
(0.004)
(0.012)
(0.003)
Table 3: Axis ablation for LM- = Snorkel, DM = RoBERTa. Each entry corresponds to picking a specific method within our design space using validation tuning. Blue numbers are the highest for given dataset, while bold numbers are those within error bars of the best result.
all non-default thresholding and SSL techniques (i.e., (Conf-based, Cut-based) x (EM, VAT, UDA, ST, ST w/ DM)) and then select the hest of these combinations using the validation set. For "Entire Design Space," we perform this method selection procedure over all methods from the design space [
From this analysis, we observe that SSL helps only in limited scenarios. Including SSL on Spouse yields 0.2 increase in Fi-score compared to not using any form of SSL. However, on all the other six tasks, "Thresh Re-label" performs at least within a standard deviation of the best method, making up 90% of the gap between "Vanilla" and "Thresh SSL Re-label." One explanation for this result is the distinctly lower coverage LF set for Spouse, a factor we explore in depth in See 4.3] Further, model size may also play role. In the corresponding Table Tablep (Appennixx (B) for MLPs, "Thresh + Re-label" can make up only 68% of the gap between "Vanilla" and "Thresh + SSL + Re-label."
4.4 What are the best instantiations of each axis?
Having compared the three axes at macro-level, now zoom in on each. Specifically, we compare all implementations of a given axis when paired with the best possible setting of the other two. For thresholding and SSL, we find that simple or previously underexplored approaches are worth using.
Thresholding. Examining Table 4 we see that thresholding is significantly help- ful in most cases. This extends the con- clusions of [15]. showing that remov- ing labels still helps even when also us- ing SSL and re-labeling. Interestingly, the simpler confidence- based threshold matches the cut-based method (within er- ror bars) on all datasets except Chemprot.
IMDb
Yelp
AGNews
TREC
Spouse
Chem.
0.901
0.953
0.885
0.702
0.365
0.593
Cov
(0.013)
(0.008)
(0.002)
(0.035)
(0.073)
(0.010)
0.907
0.965
0,887
0.765
0.531
0.605
(0.004)
(0.000)
(0.005)
(0.012)
(0.007)
(0.002)
Conf
0.906
0.965
0.884
0.741
0.531
0.593
(0.005)
(0.002)
(0.002)
(0.029)
(0.039)
(0.010)
Table 4: Closer look at thresholding for DM = RoBERTa. We report the best performance of any method that uses cach type of thresholding. Blue indicates the best result while bold indicates being within error bars of the best.
SSL and Re-labeling. For these two axes, we observe largely similar trends in Tables (T)l and in Appendix C.2 SSL and re-labeling are each only strictly nec- essary for minority of datasets, outperforming "no SSL" and "no re-labeling" (beyond error bars) on just one and two datasets, respectively. As such, all SSL methods also tend to perform similarly. For MLP models, though, where SSL is more useful, see from Table 14 that VAT and ST (DM as LF) are the most consistent. No other method comes within error bars on four datasets.
'This differs from "Thresh + SSL Re-label" because methods for that row must employ non-default choices.
7

---

4.5 When is SSL more useful?
To explain why SSL is largely redundant on most existing WS benchmarks, we show that the unlabeled data in each task is generally unnecessary for learning strong models. However, when using lower coverage LF sets, setting not captured by these benchmarks, ignoring unlabeled data can significantly compromise performance, allowing SSL more room to be impactful.
Data gaps in WS. We can think of any WS training set L = {(zf, D)) S D as suffering from:
(1) Limited size: Since not all examples are labeled, the quantity of labels | may be insufficient. Coverage bias: Since LFs abstain based feature-dependent rules, the inputs in L represent a biased subpopulation of the true distribution that the test set is drawn from.
Label noise: Since LFs just heuristics, labels Y be biased towards incorrect classes.
We hypothesize that SSL adds more unique value within our design space when gaps (1) and (2) are more significant. When gap (3) is the only dominating factor, there is less reason to expect SSL to outperform thresholding and re-labeling since the latter two more directly attempt to address label noise. However, these two axes still do not use the unlabeled examples, which cause gaps (1) and (2).
Label noise (not unlabeled data) is the main gap on existing benchmarks.
To measure the relative importance of the three data gaps, we compare the following models:
Impacts of [4] and Coverage Blas
1.00
GT (Cov): the model trained on the ground-truth labels for only LF-covered inputs, removing gap (3) but retaining gaps (1) and (2).
GT (Pull) Test Acc
Imdb
GT (Full): the model trained with a clean and fully labeled version of D, removing all gaps.
yelp youtube
agnews
Given our hypothesis, we would expect these models to perform similarly on WRENCH benchmarks. Indeed, as shown Figure the largest gap between them is < 2.5 p.p
trae
0.90
chemprot
SSL helps more when coverage is lower. While SSL's ineffectiveness on existing bench- marks corresponds to label noise being the pre- dominant data gap, we would ideally also show that SSL is more useful when the other two gaps are significant, i.e., when GT (Cov) performs markedly worse than GT (Full). One natural way to explore this would be to test on lower coverage LF sets, which result in more unla- beled data. However, most existing benchmarks have high coverage, often above 80% shown in Table To overcome this limitation, we first explore a wider range of coverage levels by subsampling generating LFs existing datasets. We also create two new WS text classification tasks based on publicly avaialble datasets, Massivel8 [8] and Banking [3]: these tasks have larger label spaces than all WRENCH tasks and therefore require more effort (i.e., LFs) to obtain high coverage. Finally, explore the less frequently studied tabular setting, using Mushroom, Spambase, and Phishing Websites in setup similar that of [32). We defer details on all these data/L.F sets to Appendix D
0.85
0.90
0.95
1.00
(Cov) Acc
Figure 3: Measuring the importance of unlabeled data WS benchmarks. Assuming no label noise, removing unlabeled data for GT (Cov) decreases performance minimally compared to GT (Full).
From this analysis, we first see that smaller coverage levels (within the ranges we test) do not always cause GT (Cov) to perform poorly. On some tasks (top two rows of Figure 4 including the three tabular and our two new ones, GT (Cov) performs significantly worse as coverage decreases In contrast, on IMDb, Yelp, and AGNews (bottom row of Figure 4) GT (Cov) surprisingly comes within 2% of GT (Full) even when coverage drops to 10-20%.
Importantly, the partitioning of datasets based on the performance of GT (Cov) also corresponds to the effectiveness of using SSL over not using it. For each LF set used, we run the methods within
2 Unfoortunately, Spouse does not have ground-truth training labels, preventing us from adding to Figure 3 In Appendix show that coverage bias (not limited size) is the main cause of these performance drops.
8

---

Semeval
Chemprot
Massive18 (Ours)
Banking (Ours)
0.95
0.90
0.90
0.85
0.85
3 0.85
0.80
0.80
< 0.80
3 0.75
0.7
0.75
0.70
0.75
0.70
0.05
0.65
0.60
0.55
0.55
0.50
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Coverage
Coverage
Coverage
Coverage
Mushroom (Tab.)
Spambase (Tab.)
Phishingwebsites (Tab.)
1.0
1.0
1.00
0.9
Test Acc
0.8
0.7
0.90
0.6
0.5
0.2
0.6
0.6
1.0
0.7
0.0
0.4
0.6
2.0
0.85
0.0
0.6 Coverage
Caverage
Coverage
Imdb
Yelp
Agnews
0.94
0.92
0.88
0.94
0.86
0.90
0.92
0.89
0.90
0.82
0.80
0.88
Coverage
Coverage
Coverage
+ SSL
Thresh Re-label
GT (Full)
GT (Cov)
Vanilla WS
Figure 4: Impacts of coverage level on data gaps and the effectiveness of SSL. Plotting GT (Full) (gray-dashed) and GT (Cov) (blue), we measure the impacts of removing label noise across nested LF subsets. We also compare the effectiveness of Thresh SSL (green) to Thresh Re-label (red). For datasets more impacted by coverage bias and limited size (larger gaps between dashed and blue lines), SSL adds more unique value. "(Ours)" refers to datsets we introduce. "(Tab.)" refers to tabular tasks.
reduced version of our design space: allow for confidence thresholding and try both versions of self-training to perform SSL (i.e., by labeling points in U' or to re-label (i.e., by labeling points in L). For the tabular tasks, we also try VAT for the SSL technique because of its effectiveness for MLPs. We plot the best performing methods for both "Thresh SSL" and "Thresh + Re-label" in Figure 4 As shown in the top two rows, "Thresh + SSL" enough consistently outperform "Thresh + Re-label" at lower coverage levels. In contrast, in the bottom row, where GT (Cov) drops in performance by < p.p.. "Thresh + SSI." continues to perform within errors of "Thresh Re-label."
Overall, these results suggest that SSL can indeed be useful in WS settings that differ from those captured by standard benchmarks. At lower coverage levels (i.e., <35%), SSL is consistently worth trying; in practice, SSL may allow users to reduce the number of LFs they need to write in order to achieve particular target performance. On our tabular tasks, these savings seem especially impactful; SSL allows one to achieve performance equivalent to having all 40 LFs (highest coverage plotted) with at most 20 LFs (second lowest coverage plotted).
5 Conclusion
We constructed design space for combining SSL and WS, using it to both contextualize and match the performance of state-of-the -ht WS methods. We show that on existing WS benchmarks, using the unlabeled data is surprisingly not essential. However, SSL can be more useful when training MLPs and when the LFs cover fewer examples. Some limitations of our work include: (1) a lack of finer-grad grained heuristics for more efficiently navigating relying on ground-trutt labels to measure the relative importance of different data gaps; (3) using automated ways generate low coverage LF sets which may not fully represent those written by real users. We see all of these limitations as seeding important directions for future work.
design space when given new task; (2)
'Unlike in Section 4.3 here also allow "Thresh" in "Thresh (SSL, Re-label]" to be coverage-based.

---

References
[1] Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi. Learning from rules generalizing labeled exemplars. In International Conference on Learning Representations. 2020.
[2] Benedikt Boecking and Artur Dubrawski. Pairwise feedback for data programming. In Proceedings of NeurlPS '19 Workshop on Learning with Rich Experience (LIRE '1), December 2019.
[3] M. Herbert C. Rosenberg and H. Schneiderman. Semi-supervised learning by entropy mini- mization. In Semi-Supervised Self-Training of Object Detection Models, 2005.
[4] Salva Cachay, Benedikt Boecking, and Artur Dubrawski. End-to-end weak supervision. In Advances in Neural Information Processing Systems, volume 34, 2021.
[5] Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38-45, Online, July 2020. Association for Computational Linguisties.
[6] Mayee Chen, Benjamin Cohen-Ware Wang, Stephen Mussmann, Frederic Sala, and Christopher Re. Comparing the value of labeled and unlabeled data in method-of-momente latent variable estimation. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 2286-3294 PMLR, 13-15 Apr 2021.
[7] Y. Ding, L. Wang, D. Fan, and Gong, semi-supervised two-stage approach to learning from noisy labels. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1215-1224, Los Alamitos, CA, USA, mar 2018. IEEE Computer Society.
[8] Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Lauric Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. Massive: A Im-example multilingual natural language understanding dataset with 51 papplogically-diverss languages, 2022.
[9] Chufan Gao, Mononito Goswami, Jieshi Chen, and Artur Dubrawski. Classifying unstructured clinical notes via automatic weak supervision. In Proceedings of the 7th Machine Learning for Healthcare Conference, volume 182 of Proceedings of Machine Learning Research, pages 673-690. PMLR, 05-06 Aug 2022.
[10] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems, volume 17. MIT Press, 2004.
[11] Giannis Karamanolakis, Subhabrata Mukherjee, Guoqing Zheng, and Ahmed Hassan Awadallah. Self Self-training with weak supervision. In Proceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 845-863, Online, June 2021. Association for Computational Linguistics.
[12] Jan Kocoñ, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło. Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, Anna Kocoñ, Bartlomiej Koptyra, Wiktoria Nielsszreenoo-Kowsewiizz, Piotr Milkowski, Marcin Oleksy, Maciej Piasecki, Fukasz Radliński, Konrad Wojtasik, Stanislaw Wožniak, and Przemystaw Kazienko. CharGPT: Jack of all trades, master of none. Information Fusion, 90:101861, 2023.
[13] Kyeongbo Kong, Junggi Lee, Youngchul Kwak, Minsung Kang. Seong Gyun Kim, and Woo-Jin Song. Recycling: Semi-supervised learning with noisy labels deep neural networks. IEEE Access, 7:66998-6777 67005,, 2019.
[14] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In Interna- tional Conference on Learning Representations, 2017.
10

---

[15] Hunter Lang, Aravindan Vijayaraghavan, and David Sontag. Training subset selection for weak supervision. In Advances in Neural Information Processing Systems, volume 35, pages 16023-16036. Curran Associates, Inc., 2022.
[16] D.H. Lee. The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop on Challenges in Representation Learning, 2013.
[17] Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations, 2020.
[18] Ayush Maheshwari, Oishik Chatterjee, KrishnaTeeja Killamsetty, Rishabh K. Iyer, and Ganesh Ramakrishnan. Data programming using semi-supervision and subset selection. CoRR, 0bs/2008.098877 2020.
[19] Alessio Mazzetto, Cyrus Cousins, Dylan Sam, Stephen H Bach, and Eli Upfal. Adversarial multi class learning under weak supervision with performance guarantees. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7543. PMLR, 18-24 Jul 2021.
[20] Alessio Mazzetto, Dylan Sam, Andrew Park, Eli Upfal, and Stephen Bach. Semi-supervised ag- gregation of dependent weak supervision sources with performance guarantees. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages
PMLR, 13-15 Apr 2021.
[21] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training:
A regularization method for supervised and semi-supervised learning, 2017.
[22] Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In Advances in Neural Information Processing Systems, volume 31, 2018.
[23] Rattana Pukdee, Dylan Sam, Pradeep Kumar Ravikumar, and Nina Balcan. Label propagation with weak supervision. In The Eleventh International Conference on Learning Representations. 2023.
[24] Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christopher Ré. Training complex models with multi-task weak supervision. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):4763347777 Jul. 2019.
[25] Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
[26] Wendi Ren, Yinghao Li, Hanting Su, David Kartchner, Cassie Mitchell, and Chao Zhang. Denoising multi-source weak supervision for neural text classification. In Findings of the Association for Computational Linguisties: EMNLP 2020, pages 3739-3754, Online, November 2020. Association for Computational Linguistics.
[27] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[28] Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In International Conference on Learning Representations, 2022.
[29] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data aug- mentation for consistency training. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 0256-6268. Curran Associates, Inc., 2020.
[30] Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang. Fine-tuning pre- trained language model with weak supervision: A contrastive-regulaizee self-training approach. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1063-1077, 2021.
11

---

[31] Jieyu Zhang, Cheng- Yu Hsich, Yue Yu, Chao Zhang, and Alexander Ratner. A survey on programmatic weak supervision. arXiv preprint Xiiv:22020054533 2022.
[32] Jieyu Zhang, Haonan Wang, Cheng- Hsieh, and Alexander Ratner. Understanding pro- grammatic weak supervision via source-aware influence function. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Process- ing Systems, volume 35, pages 2862 2875. Curran Associates, Inc., 2022.
[33] Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner. WRENCH: A comprehensive benchmark for weak supervision. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2021.
[34] Wenxuan Zhou, Hongtao Lin, Bill Yuchen Lin, Ziqi Wang, Junyi Du, Leonardo Neves, and Xiang Ren. Nero: A neural rule grounding framework for label-efficien relation extraction. In Proceedings of The Web Conference 2020, WWW '20, page 2166-2176. Association for Computing Machinery, 2020.
[35] Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. Can chatgpt reproduce muman-generates labels? a study of social computing tasks, 2023.
12

---

A Experiment Details
A.1 Search Spaces
Method
Hyperparameters
Description
Range
learning rale
1e.5.ee-4le333ee2..ed
Snurkel LM
veight_decay
weight decay
1e.5.le-.le-3le-222ee
num _en_epoch
the number of training epochs
5.10,50,100,200
batch_ size
hatch size
32,128,512
learning
Ie-4.1e-3,Ie-2
dropout
dropout probability
0.2. 0.0 0.0
ML.P
weight_decal
weight decay
nun_layes
the number hidden layers
hidden_size
hidden size of MLP layers
256,
batch_
batch size
32
BERT
leaming rate weight decay
1e-5,3c-5,50-5
weight_decay
le-4
T
period updating pseudo-labels confidence threshold
50,100,200
0.1, 0.3, 0.5, 0.01,0.05.0.1
COSINE
weight for confidence regularization weight for contrastive regularization margin for contrastive regularization
p
momentum term for temporal ensembling size of hidden layer
0.6
d
26 -
Denoise
c1
coefficient of denoiscr loss
0.1.0.3.0.55007..0..
coefficient classifier loss
0.I.0.3.0...0....0..
coefficient unsuperviseds self-training
1-c2-c1
7
temperature
1.0, 0.33
WcaSEL
size hidden layer dropout prob
26
dropout
0.3
Confidence Thresh
threshold
minimum confidence to keep
up. linspace( (>) 0.1,
Cutstat Thresh
percentile
what percentile as cut-off
np. mp.1inepace(0) 0.9, 9)
Self- Self-Training
T
period for updating lahudo-lahels
50, 100
Self-Train (DM LF)
T
period for updating pseudo-labels
50, 100
EM
A
coefficient EM loss
1.0.5, Ic-1, 1c-2, le-3
A
coefficient for VAT unsupervised loss
1.0.5, le-1, 1e-2,
VAT ip
Iterations of the power method for VAT
VAT
{
Finite difference for approximation in VAT
0.05, 0.5, 1,
VAT Perturbation distance
1c-6, le-3, 2.5.5
A
the cocfficient for UDA unsupervised loss
2. 1 0.5, le-1 none, log 1, 0.855,
UDA
tea_sched
schedule for training signal anncaling temperature for prediction sharpening
Table Search spaces organized by type of method: label models, end models, baselines, threshold- ing, and SSL. techniques.
A.2 Implementation Details
Here provide high-level details on the various components of our design space. For further details, please see our codehase at //ttts:///githue:: com//e/frreyypp1//S1......
Label Models. Of the two label models that we tune over the search space in Table|3Jonce per dataset; we then fix the hyperparameters and seed for fitting Snorkel to ensure the same exact weak labels are given all methods. This seed is chosen from a pool of three, also based on the validation set.
must tune parameters only for Snorkel. We
End Models. For MLPs, we increase the range of hidden sizes compared to WRENCH and tune for dropout instead of weight decay. For RoBERTa, we do not tune the batch size in order to reduce the search space, observing minimal differences compared to using alternative batch size of 16. Following WRENCH, we perform early stopping based on validation performance for all methods. Specifically, we use patience of 1000 steps for MLPs and 100 steps for RoBERTa.
13

---

Thresholding. Whenever applying dynamic versions of thresholding, we use the same threshold value across different rounds of end model training for simplicity. Setting separate thresholds for different stages of learning (or adaptively) could be an interesting direction for future work.
SSL Techniques. We make note of some important details relevant specific SSL techniques:
For the two self-training methods, we use training schedule similar to that of COSINE. whereby we divide training into two stages: In the first stage, the end model is trained on weak labels as is done in vanilla WS end model training. In the second stage, new pseudo-labels are generated upon reaching pre-specified update period of (50,100) steps.
For self-training, we also allow for applying thresholds to the pseudo-labels given to unlabeled data. When combining this approach with thresholding the initial weak labels, we again use the same threshold value.
When using ST (DM as LF), we do not re-tune the label model-specific hyperparameters for Snorkel in subsequent self-training rounds. We default to instead using the same hyperparameters found from the initial search.
For EM, VAT, and UDA, we sample batches with equal numbers of labeled and unlabeled examples when taking each step but tune a weighting parameter to balance their losses.
When applying VAT to RoBERTa models, calculate perturbations with respect to the RoBERTa-based features (which are continuous) instead of the raw text inputs.
We try UDA on the text classification datasets where translations are straightforward to apply (i.e., do not require extra annotation effort). This includes Youtube, Trec, AGNews, and Yelp. For Yelp, where some examples are too long, we split examples at the sentence-level and backtranslated each sentence before concatenating following (9 For the other text datasets, which are relation classification, we did not use UDA as it requires non-trivially annotating the entity spans in the backtranslations.
Re-labeling, For re-labeling, we again use a two-stage training schedule where the second stage contains a label update period (as explained when discussing self-training). When combining re- labeling with thresholding, we consider the set of examples with candidate labels (i.c., which are possibly removed when performing dynamic thresholding) thresholding (of LM outputs). Thus, re-labeling provides new pseudo-labels only for examples that survived the initial thresholding. This is so as to cleanly differentiate from SSL.
fixed after the initial round of
Compute. We ran all MLP experiments on AWS, using up to four g4dn.4xlarge EC2 instances at one time. Each instance allowed for running up 10 different experiments (i.e., here considered hyperparameter sweep for any specific method from our design space) in parallel. For all RoBERTa training runs, we ran our experiments on a fleet of up to 15 NVIDIA-A40 GPUs hosted on the University of Washington Hyak cluster. Each experiment fits on a single GPU, which is large enough use our batch sizes without needing gradient accumulation.
'ImdB had a similar length issue, but unfortunately the version used in WRENCH had been stripped of punctuation, rendering sentencc-spliting non-trivial.
14

---

B Additional Results for Baseline Comparisons
B.1
Additional Plots
Yelp-MLP
(Acc)
A Teat Perf.
0.22
0.82
0.62
0.02
0.91
8.02
0.01
0.01
-0.91
-0.0
-0.01
0.00
0.12
TREC. MLP (Acc)
6:18
Spouse-MLP [F] binary)
0.07
Chemprot MUP (Acc)
0.12
Census-MLP [F] binary)
in Test
0.04
6.06
0.01
0.04
0.50
0.00
0.00
0.00
Best of Design Space
Cutstat Snorkel
Denaise
Snoricel
Figure 5: Test performance for LM Snorkel, DM MLP. We compare selected method from design space (green) with four recent proposals from the literature. We plot all performances relative that of vanilla training (see Table for the absolute metries).
BNDD-BERT-VY (Acc)
Yelc-BERT-MV (Acc)
Youtube BERT-MV (Acc)
AGNews AGNeww--ERTMMt BERT-MV (Acc)
8.06
0.06
0.03
0.03
Part.
8.00
0.00
0.00
0.09
TREC BERT MY (Acr)
BOOUs--BERT-VV binary)
0.06
Chemprot BERT. MV (Acc)
5.30
5.36
0.04
Cut + ST (DM as LF) Re-abel
To
Dennise NV
0.00
Weasel
2.05
5.00
0.02
Figure 6: Test performance for LM = MV, DM RoBERTa We compare selected method from our design space (green) with four recent proposals from the literature. We plot all performances relative to that of vanilla training (see Table for the absolute metries). For this setting, we did not run an exhaustive search over the whole design space so do report "Best of Design Space."
MILP Results. While fully fine-tuning RoBERTa end models provides better overall performance across text-based tasks, we also provide results for MLP end models for completeness. Here, the best single-method across datasets was (one-time) confidence thresholding + VAT re-labeling. This method achieves the best point estimate on all 8 datasets compared to previous baselines.
Majority Voting Results. We also ablate the specific choice of label model be Majority Voting for the RoBERTa experiments. Here, we mostly use the same method as for when the label model Snorkel. However, found that swapping in cut-based thresholding tended perform better than sticking with confidence-based This behavior can perhaps be explained by the relatively cruder confidence estimates for Majority Voting soft-labels; instead learning a probabilistic model, the soft-labels for Majority Voting simply use the ratios of LF votes (c.g. for a binary task, if an example received 2 negative and 3 positive votes, the soft-label would be [0.4, 0.6]). Overall, we find that this method at least comes within error bars of previous baselines on all datasets though the point estimate is 8.4 p.p. lower on Spouse. Notably though, the best overall performance on Spouse still obtained by applying our design space on top of the Snorkel label model (see Tables 6 anddd]
15

---

B.2 Tables with Absolute Performance
Method
TMDb
Yelp
Yontube
AGNews
TREC
Spouse
Chemprot
Vanilla
0.874
0,925
0.948
0.867
0.646
0.215
0.572
(0,002)
(0.009)
(0.011)
Cutstat Snorkel
0.871
0,939
0,951
0.381
0,707
0.242
0.587
(0.006)
(0.006)
(0.002)
(0,001)
(0.015)
(0.023)
(0.005)
0,888
0.954
0,947
0.882
0.711
0.531
0,581
Cont ST (DM LF) Re-label
(0.013)
(0.006)
(0.012)
(0.002)
(0.030)
(0.039)
(0.012)
Best of Design Space
0.907
0.965
0,952
0.765
0.531
(0.004)
(0,002)
(0.006)
(0,002)
(0.012)
(0.039)
(0,008)
0.884
0.938
0.952
0.875
0.675
0.269
0.573
COSINE (RoBERTa) Snorkel
(0,004)
(0.009)
(0.003)
(0.004)
(0.002)
(0.106)
(0.005)
Denoise (RoBERTa) + Snorkel
0.867
0,922
0.949
0.867
0.633
0.287
0,544
(0.007)
(0.018)
(0.006)
(0.008)
(0.014)
(0.202)
(0.005)
Wensel (RaBERTa)
(0.009)
(0.017)
(0.008)
(0.001)
(0.044)
(0.117)
(0.007)
Table 6: Test performance for LM = Snorkel, DM = RoBERTa. Blue referes to the best performance
of single method (i.c., excluding "Best of Design Space", which reports the best method per dataset)
while bold refers to being within standard deviation error bars of the best method. Red indicates
where "Best of Design Space" outperforms the best single method outside of error bars.
Method
IMDb
Yelp
Youtube
AGNews
TREC
Spouse
Chemprot
0.839
0,960
0.669
0,248
0.374
Vanilla
(0.004)
(0.017)
(0,014)
(0.004)
(0.012)
(0.030)
(0.003)
0.861
0.953
0.968
0.881
0.731
0.279
0.592
Cuistat MV
(0.013)
(0.005)
(0.066)
(0.005)
(0.027)
(0.013)
(0.010)
Cusstat ST (DM LF) Re-label
0.882
0.954
0.969
0.882
0.743
0.347
0.606
(0.007)
(0.004)
(0,044)
(0,004)
(0,049)
(0,052)
(0,004)
0.884
0,945
0.959
0.876
0.716
0.431
0.589
COSINE (RoBERTa) MV
(0.004)
(0.004)
(0,006)
(0.011)
(0.067)
(0.034)
(0.010)
0.866
0,905
0.956
0.873
0.678
0.172
0.576
Denoise MV
(0.013)
(0.029)
(0.010)
(0.006)
(0.025)
(0.205)
(0.010)
Weasel (RoBERTs)
0.866
0,928
0.958
0.871
0.629
0.293
0.586
(0.009)
(0.017)
(0.008)
(0.001)
(0.044)
(0.117)
(0.007)
Table 7: Test performance for LM = Majority Voting, DM = RoBERTa. Blue referes to the best
performance of single method while bold refers to being within standard deviation error bars of the best method.
Method
IMDb
Yelp
Youtube
AGNews
TREC
Sponse
Chemprot 0.544
Census
0.828
0.917
0.925
0.854
0.623
0.228
0.511
Vanilla
(0,01))
(0.004)
(0.002)
(0.004)
(0.013)
(0.002)
(0.012)
Cutstat Snorkel
0.827
0.913
0.919
0.865
0.662
0.228
0.553
U.587
(0.002)
(0.001)
(0.005)
(0.001)
(0.017)
(0.003)
(0.004)
(0.001)
0,850
0.924
0.929
0,870
0.678
0,330
0.558
0,593
+ Re label
(0.002)
(0.004)
(0.005)
(0.000
(0.023)
(0.005)
(0.005)
(0.005)
0,850
0.677
0.561
0.605
Best Design Space
(0.002)
(0.004)
(0.005)
(0,000)
(0.016)
(0.006)
(0.005)
(0.006)
COSINE (MLP) Snorkel
0,833
0.915
0,921
0.853
0.641
0.248
0,536
0.520
(0.001)
(0.002)
(0.002)
(0.002)
(0.012)
(0.028)
(0.004)
(0.027)
0.806
0.895
0.568
0.535
0.569
Denoise (MLP) Snorkel
(0.011)
(b.006)
(0.000)
(0.001)
(0.003)
(0.207)
(0.004)
(0.007)
Weasel (MLP)
0.826
0.446
0.551
0.552
(0.003)
(0.005)
(0.016)
(0.005)
(0.117)
(0.017)
(0.007)
(0.051)
Table 8: Test performance for LMM Snorkel, DM MLP. Color-coding is done in the same way that described in Table 6.
16

---

Additional Ablation Results
C.1 Axis Ablation for MLP
Method
IMDb
Yelp
Youtube
AGNews
TREC
Spouse
Chemprot 0.544
Census
Vanilla
0.828
0.917
0.925
0.854
0.625
0.228
0.511
0.679
(0.001)
(0.001)
(0.004)
(0.002)
(0.004)
(0.013)
(0.002)
(0.012)
(0.002)
0.835
0,924
0,155
0,865
0.678
0.228
0.547
0.587
0.697
Thresh Alone
(0.002)
(0.002)
(0.002)
(0.001)
(0.027)
(0.003)
(0.004)
(0.001)
(0.003)
SSI. Alome
0.841
0.932
0.877
0.605
0.518
(0.004)
(0.002)
(0.000)
(0.001)
(0.008)
(0.029)
(0.003)
(0.015)
(0.004)
Re-label Alone
0.833
0.862
0.631
0.549
0,526
(0.001)
(0.002)
(0.014)
(0.001)
(0.024)
(0.003)
(0.007)
(0.027)
(0.005)
Thresh + SSL
0.838
0.926
0.929
0,878
0.691
0.394
0.560
0.605
0.728
(0.004)
(0.004)
(0.002)
(0.001)
(0.012)
(0.006)
(0.001)
(0.006)
(0.002)
0.837
0,925
0.917
0.870
0.677
0.270
0.558
0.602
0.707
Thresh Re-label
(0.002)
(0.003)
(0.007)
(0.001)
(0.016)
(0.007)
(0.002)
(0.0022
(0.002)
0.835
0.925
0.935
0.875
0.662
0.347
0.557
0,542
0.710
SSL Re-label
(0.001)
(0.006)
(0.005)
(0.002)
(0.011)
(0.043)
(0.004)
(0.025)
(0.006)
Thresh SSL Re-label
0.850
0,930
0.929
0.880
0.671
0.347
0.561
0.6016
0.722
(0.002)
(0.001)
(0,005)
(0.000)
(0.034)
(0.043)
(0.005)
(0.001)
(0.007)
0.850
0.926
0,929
0.880
0.677
0.394
0.561
0.605
0.728
Entire Design Space
(0.002)
(0.004)
(0.005)
(0.000)
(0.016)
(0.006)
(0.005)
(0.006)
(0.002)
Table 9: Axis ablation for LM Snorkel, DM MLP. Each entry corresponds picking a specific method within our design space using validation tuning. Blue numbers are the highest for a given dataset, while bold numbers those within error bars of the best result.
C.2 Axis Instantiations
C.2.1 RoBERTa
Note that we exclude Youtube from these tables because the Vanilla baseline is already within error bars of the best performing methods in T As such, this dataset would not provide much useful signal for assessing which version of particular axis is best.
IMD6
Yelp
AGNews
TREC
Spouse
Chemprol
0.905
0.961
0.883
0.748
0.332
0.600
No SSL
(0.011)
(0.002)
(0.005)
(0.023)
(0.105)
(0,006)
0.895
0.958
0.883
0.700
0,286
0.586
EM
(0.005)
(0.003)
(0.002)
(0.018)
(0.153)
(0.007)
0.907
0.961
0.887
0.765
0.392
0.601
ST
(0.004)
(0.003)
(0.005)
(0.012)
(0.098)
(0.005)
0.889
0.958
0.885
0.725
0.531
01.01
(DM I.F)
(0.004)
(0.003)
(0.003)
(0.018)
(0.039)
(0.008)
0.900
0.958
0.883
0.743
0.437
0.605
VAT
(0.004)
(0.001)
(0.002)
(0.029)
(0.004)
(0.002)
UDA
0.965
0.885
0.753
(0.002)
(0.003)
(0.021)
Table 10: Closer look at for DM = RoBERTa, We report the best performance of any method
that incorporates given SSL technique.
IMDb
Yelp
AGNews
Spouse
Chemprot
Re-labeting
0.885
0.959
0,887
0.732
0,529
0.597
(0.018)
(0.002)
(0.005)
(0.014)
(0.057)
(0.009)
Re-labeling
0.907
0.965
0.885
0.765
0.531
0.605
(0.004)
(0.002)
(0.003)
(0.012)
(0.039)
(0.002)
Table 11: Closer look at re- labeling for DM = RoBERTa. We report the best performance of any
method that either uses or does not use re-labeling.
17

---

C.2.2 MLPs
Method
IMDb
Yelp
Youtube
AGNews
TREC
Spouse
Chemprot
Census
0.841
0,925
0,935
0.877
0.662
0.375
0.357
0.542
Coverage-based
(0.004)
(0.006)
(0.005)
(0.001)
(0.011)
(0.029)
(0.00)
(0.025)
0.684
0.606
Cui-based
(0.002)
(0.001)
(0.002)
(0.000)
(0.016)
(0.015)
(0.003)
(0.001)
Confidenee-bsed
0.850
0.929
0.878
0.691
0.560
0,592
(0,002)
(0.002)
(0.005)
(0.002)
(0.012)
(0,009)
(0,011)
(0.008)
Table 12: Closer look at thresholding for DM = MLP. We report the best performance of any method that incorporates given thresholding method.
Method
IMDb
Yelp
Youtube
AGNews
TREC
Sponse
Chemprot
Census
0.602
SSL
(0.002)
(0.001)
(0,002)
(0.001)
(0.016)
(0.007)
(0.011)
(0.002)
VAT
0.850
0.926
0.691
0.563
0.605
(0.002)
(0.002)
(0,005)
(0,000)
(0.012)
(0.015)
(0,007)
(0.006)
0.835
0.922
0.925
0.869
0,664
0.268
0.560
0.595
EM
(0.001)
(0.001)
(0.002)
(0.002)
(0.007)
(0.032)
(0.009)
(0.004)
0.838
0.928
0.929
0.866
0.674
0.379
0.566
0.592
ST
(0.001)
(0.002)
(0.018)
(0.003)
(0.002)
0.834
0.930
0.935
0.80 8.869
0.668
0.417
0.561
0.606
ST as
(0.001)
(0,001)
(0.002)
(0.000)
(0.038)
(0.009)
(0.005)
(0.001)
Table 13: Closer look at SSL for DM = MLP. We report the best performance of any method that incorporates given SSL technique.
Method
IMDb
Yelp
Youtube
AGNews
TREC
Sponse
Chemprot
Census
No Re-labeling
0.841
0.926
0.932
0,878
0.691
0.417
0.360
0.605
(0.009)
Re Re-labeling
0,850
0.930
0.935
0,880
0.677
0.347
0.566 (0.003)
0.606
(0.002)
(0.005)
(0.000)
(0.016)
(0.043)
(0.001)
Table 14: Closer look at SSL axis for DM = MLP. We report the best performance of any method that either uses does not re-labeling.
18

---

D Beyond standard WS benchmarks: datasets for coverage ablations
D.I LF subsampling and procedural generation on existing WRENCH datasets
Overall, WS benchmarks lack agreed-upon ways to explore different LF coverage/precision trade-offs. In this work, we choose to do so by sub-sampling LFs from an overall base set. However, the default LF sets associated with WS benchmarks are not equally as suitable to subsample from. In particular, we are wary about subsampling default LF sets when they:
Contain very few LFs, which reduces the possible granularities of subsampling
Contain LFs which are heterogeneous have untracked lineages (e.g., A closely relates to or was only provided in response to another LF 22).
For datsets where we subsample directly, we choose ones with both larger LF counts and where all LFs are of the same form. This includes:
rules selected by humans from an automated candidate generation procedure (34]
Semeval: 164
Chemprot: 26 individual keyword- -oooddbbsedd rules (30
These contrast with LF sets that are far smaller, more heterogeneous, or have unclear dependencies.
For these datasets, we instead procedurally generate LFss using the tools from WRENCH 33
IMDb: 4 aggregate keyword based rules (i.e., each LF checks for the presence of any of multiple keywords), expression- rule 26
Yelp:
heuristic rules on keywords, thiry-party model on polarity of sentiment 26
Agnews: 9 aggregate keyword based rules split amongst four respective classes 26
Subsampling. On some datasets, directly subsample the LFs coming from WRENCH, selecting the most accurate LFs in a class-stratified fashion in order to roughly preserve similar ratios of LFs between classes. We also take care minimum count to per class. We explore subsampling per-class ratios within (0.1, 0.9), avoiding LF sets that near-duplicates of each other (i.e., having coverage levels within 1% of each other). We subsample based on accuracy to (optimistically) simulate an LF writer who is both careful and has considerable domain expertise; we assume that if they were end up at lower coverage LF set (either by writing fewer LFs or by pruning LFs written during a "brainstorming" phase), they would prioritize rules that they are most confident about. Assuming the LF writer has sufficient domain knowledge, these LFs are also the ones more likely to be accurate on the examples they fire on.
completely remove all the LFs for given class, setting the
WRENCH's LF generator to construct LF sets with (2, 5, 10. 15, 20) n-gram based LFs per class, choosing the most accurate LFs from the candidate pool that have at least 2% coverage over the training set.
Procedural Generation. For the datasets where subsampling is less appropriate,
D.2 New WS benchmarks: Massivel8 and Banking77
We also create two new WS benchmarks by writing LFs for the publicly available intent classification datasets MASSIVE [8] and Banking77 (5). These tasks were chosen high-level capture some practical challenges that are not as well-represented by current WRENCH tasks; most notably, they contain significantly higher class counts, which makes them more challenging to write LFs for.
Roughly speaking, assuming one writes uni-polar LFs (i.e., each LF either votes for a single class abstains), the form of the vast majority of LFs in WRENCH, the number of LFs needed to reach a certain coverage level will likely need to scale with the number classes Ø Further, because tasks with larger label spaces require writing more LFs, also view these tasks being especially relevant for studying lower coverage LF sets. We provide more details about the new tasks as follows:
Massivel8 18 is derived from the MASSIVE dataset, collection of single-shot interactions between human users and general intelligent voice assistants across 52 languages. We exclusively use the
This, of course, also assumes that the coverage of LFs within each class does not change dramatically
19

---

English-US US portion of this dataset and treat the broad scenarios (i.e., general domains) as labels instead of the finer-graine gre--gr intents. For instance, "alarm" a scenario whereas "alarm_set" and "alarm_ "errmreeeee""_" are two intents within that scenario. We call the resulting task Massive 18 since there are 18 different scenario classes. Notably, even with this simplification, Massive contains at least nearly double the class count of all previous WRENCH classification tasks (see Table 13 helow).
Banking77 is a collection of online banking queries along with the corresponding user intents. We use this task as defined with the full set of 77 intents, which can be both quite domain specific and especially fine-graine fine-grained. If we were to write the minimum number of LFs for Banking77, i.e. one for each class, this would already require 77 LFs, more than the number of LFs for most previous WRENCH datasets.
Datasets
31
[Trii]]
TTst
LFs
Coverage 87 77
Snork. Precision
Metric
IMD6
2
20000
2500
74.4%
Acc.
Yelp
30400
3800
82.8%
75.4%
Acc.
Youtuhe
1586
250
87.7%
87.0%
Acc.
AgNows
96000
12000
69. 1%
82.5%
Acc.
Tree
500
Acc.
Massive18 (ours)
18
11514
2974
59
61.3%
83.8%
Acc.
Ranking77 (ours)
9003
3080
54.5%
76.5%
Acc.
Spouse
22254
2701
25.8%
65.6% (on Val)
Fl (binury)
Chomprot
10
12861
1607
26
85.6%
58.0%
Aoc.
Census
10083
16281
99.1%
58.0%
Fl (binary)
Table 15:
Compuring our two new datasets (blue) to previous benchmarks. Coverage is the
percentage of training examples on which at least one LF does not abstain. "Snork. precision" refers to the accuracy of the Snorkel LM on the covered set of inputs.
Labeling Functions. For both datasets, we manually write a set of LFs ourselves based upon a randomly sampled development ser of 250 cleanly-labeled examples. Each LF checks whether specific keyword (or set of multiple keywords) is substring of the given example and then votes for a specific class if the substring(s) are present (and otherwise abstaining). We provide some examples of the LFs we wrote in Tables 16 and 17 while we share the full LF sets in our supplied codebase. Importantly, while we believe these LF sets are reasonable, they should not be treated as the be-all-end-all and encourage others to iterate or develop their own for these tasks in the future.
Finally, in our experiments, we also try the same subsampling procedure from Appendix D.I to further explore different coverage levels. For Massivel8 we use the ratios (0.1, 0.5, 0.7, 0.8, 1] and for Banking77 we use the ratios (0.1, 0.5, 0.7, 1.0).
Label
Keyword LFs
"alarm'
['alarm",, "wake+up'] ["takeaway", "delivery", "order"]
"takeaway"
"social"
["tweet", "wwitter", "facebook", ''copllain'] ("vhat+song",, "saveetong"', "sbufffee]] ("calendar"' "schedule", "remind"'
"music"
"calendar"
Table 16: Example LFs for Massivel8. Each row shows one of the possible labels and the associated keyword-based LFs. Note that some LFs contain multiple keysords/substringg that are joined by the "+" character. This signifies that the LF checks whether all the key keywords supplied are in a given example (though they do not necessarily have to appear in the provided order).
Label
Keyword LF's
"ago_limit"
['age limit", "child" "my son',, "my daughter"] ["takeavay"' "delivery", "order"]
"pin_blocked" blocked"
"lost_or_ate phost_or_sttoen_cam
["tueet", "twitter", "facebook", "complain"] ("idtcheck"' "what+ id"]]
"disposable_can caaeee__erd_____ limits" Limitt
['disposablerlimitt" "disposabletnsx"1
Table 17: Example LFs for Banking77. Note the usage of "4", as explained in the caption of Table 16
20

---

D.3 Tabular Datasets
Labeling Functions. We use three tabular tasks and the LF generation procedure from [32. At a high-level, this procedure uses the sklearn implementation of random forests (i.e., LandomForestClassifiie
train multiple decision trees on a small cleanly-labeled development set (i.e., formed by uniformly sampling 5% of the training set in our experiments). Uni-polar LFFs are then derived from individual trees. In our experiments, we generate a set of 100 candidate LFss for each task and then select the top, 10%, 15%, 20%) most accurate LFs (per-class) based upon the full training set (i.e., similar in spirit WRENCH's procedural LF generator). Again, while this selection step isn't possible without access to all the labels, we consider it as loose (optimistic) approximation integrating a domain expert's knowledge and judgment.
Robustness to seeding, One caveat of this approach for genearting LFs is that the behavior may vary depending on the initial random seed given RandomFo www.rrrrsssssiiiercma this, we try multiple seeds and find that the overall conclusions do not change across different runs of the LF generator. As seen Figure 7 while the absolute performance levels may vary across seeds, SSL is consistently more useful at lower coverage levels.
To try to account for
Mushroom (Seed 1)
Mushroom (Seed 2)
Mushroam (Soed
0.9
0.9
0.5
0.7
0.7
0.7
0.0
1.0
0.0
0.2
0.4
0.0
08
10
Spambase (Seed
Spambase (Seed 2)
Spambase (Seed
0.95
0.95
0.95
0.85
0.85
0.85
0.80
0.80
0.80
6.0
1.0
0.70
0.2
0.4
1.0%%
5.0
0.6
1.0
Phishingwcbsttes
Phishingwecbikess (Seed 2)
Phishingwwoaites
0.8
0.90
0.81
0.0
0.4
0.6
0.8
1.0
0.5
0.0
0.2
0.4
1.0
0.85
0.0
0.2
0.4
0.6
0.8
1.0
Coverage
Coverage
Coverage
GT (Cov)
Vanill
-
Thresh-SSL - Thresh+Re: abel
-e.. GT (Full)
Figure 7: Tabular experiments across different random seeds
21

---

E Coverage bias explains the value of unlabeled data
Semeval
Chemprot
Massive18 (Ours)
Banking77 (Ours)
0.95
0.50
0.90
0.85
0.85
0.80
05
0.75
0.7
0.75
0.70
0.70
0.65
0.6
0.65
0.65
0.60
0.60
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Coverage
Coverage
Coverage
Coverage
Imdb
Yelp
Agnews
0.98
0.92
0.96
0.92
0.86
0.88
0,92
0.84
0.90
0.80
0.86
0.0
0.2
0.4
0.2
0.4
0.6
0.8
1.0
1.0
Coverage
Coverage
Coverage
GT (Full)
GT (Cov)
(Sub)
Vanilla Ws
Figure 8: Limited Size versus Coverage Bias. We may assess the relative impacts of the limited size of WS training sets (comparing GT (Full) and GT (Sub)) versus their coverage bias (comparing GT (Sub) and GT (Cov)).
In Section 4.5 we observed that some datasets, as coverage decreases, so does the performance of GT (Cov). Indeed, for any LF set, we can view the gap in performance between GT (Cov) and GT (Full) reflection of the aggregate impact of two data gaps that WS datasets suffer from: (1) limited size (i.e., not all training examples are labeled) and (2) coverage bias (i.e., the examples covered by LFs come from biased subpopulation of the test distribution). Because GT (Cov) simply ignores all unlabeled examples, this gap also then reflects the value of said examples.
A natural follow up question is whether limited size or coverage bias more greatly hinders learning under WS. Here, we further decompose the performance drops we observed between GT (Cov) and consider
GT (Full) by considering third model that interpolates between the two. Specifically, GT (Sub), a model that suffers only from limited size but not coverage bias. This model is trained on a training set that shares the same size the dataset for GT (Cov), but consists of examples re-sampled uniformly from the full training set (i.e., eliminating coverage bias).
Once we have trained all three models, we may then simply compare the following performance gaps:
Gest (Full)) TestPerf( GT (Sub)): to ablate the impact limited size
TestPerf(GTT (Sub)) TestPerf(GT (Cov)): ablate the impact of coverage bias
As we see in Figure 8 when large gaps exist between GT (Full) and GT (Cov), most of this gap is explained by the performance drop between GT (Sub) and GT (Cov) instead of the drop between GT (Full) and GT (Sub) (e-g., especially on Semeval, Massivel8, Banking77). This suggests that coverage bias far more responsible than limited size, which perhaps makes sense the context of WS, as LFs can label an arbitrary amount of data but only from the covered set. Also, this means that SSL techniques, initially developed settings where labeled and unlabeled sets are i.i.d. (i.c., where limited size is the only relevant data gap), actually remain effective in low coverage WS settings despite the significant impacts coverage bias
22


Reviews:
[{"review_id": "R6bTuHT1pT", "reviewer": "Reviewer_ohhg", "summary": "The paper presents a systematic study of how useful SSL is in  weak supervision. Specifically, the authors analyze SSL and weak supervision (WS) along three axes and explore various approaches along each axis. First,  the paper analyzes what to consider as `unlabeled` data for SSL training. The second axis refers to what SSL method to utilize. Finally the paper analyzes various ways of refining the WS weak labels through relabeling. In short, this paper is a large-scale analysis into SSL and WS which tries to understand where the improvements of SSL and WS reported in prior work stem from. All methods are thoroughly tested on 8 classification WS benchmarks.", "questions": "- Given the LLM craze and their outstanding few-shot capabilities, I would be curious to see how this analysis would look like when having a strong LLM weak labeler. What do you think?\n\n- I think reducing coverage by downsampling is not adequate. The downsampled set will still have the same distribution whereas a real-world limited coverage would indicate different distributions. Can you comment a bit more on how you did the downsampling?", "limitations": "The paper discussed the limitations at length. There is no negative societal impact of this work.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The presentation of the paper is great. The study in thoroughly detailed and easy to understand.\n- The motivation is also excellent. Sheding light on how to achieve an effective WS training setup and analyzing its interaction with SSL is much needed. \n- The paper performs the analysis on as many as 8 datasets and performs a thorough ablation study and analyses.\n- There are some very interesting findings, such as the fact that SSL does not help too much in various setups. In a way it makes sense that a high coverage for weak labeling means that the unlabeled data may not contain enough information for the model to learn.", "weaknesses": "- I believe the search space considered in the paper could be more comprehensive. I appreciate that the authors acknowledge this in the paper (Limitations), however, given that the paper is effectively an in-depth analysis of various ways of combining WS with SSL, I believe that these analyses are needed. Specifically, besides soft label renormalization and contrastive learning, the paper mentions that augmentations are less straightforward for NLP tasks. However, prior work such as UDA [1], MixText [2] or AUM-ST [3] have effectively used augmentations in SSL setups. I am just a bit concerned that the strength of the SSL methods is not adequate. Given the aim of the paper, these methods should have been taken into account.\n\n- Along the same lines, the novelty of the paper seems a bit limited. Once again, these types of in-depth analyses this study tries to address are much needed, but I believe the scale of the study could be a bit larger.\n\n[1] -  Unsupervised Data Augmentation for Consistency Training\n[2] - MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification\n[3] - Leveraging Training Dynamics and Self-Training for Text Classification", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and helpful feedback! To respond to your questions and comments:\n\n**Scale of study / Stronger SSL methods.** We agree that it is possible that more sophisticated SSL techniques could lead to a conclusion that SSL is more useful. Based on your feedback, we implemented and ran UDA (using En-De backtranslations) on three of the text classification benchmarks (i.e, Youtube, Trec, AGNews). Overall, the results shown in Table 1 in the pdf (attached to the global response) do not change the conclusions from our paper: using UDA does not improve upon the best methods we previously found, even when combining it with Thresholding + Re-labeling. We’re happy to expand these results to other tasks and will incorporate them into future revisions!\n\nWhile UDA can outperform other SSL methods on traditional SSL benchmarks, we suspect that our results here are because the SSL settings induced by WS are quite different in that for labeled set L and unlabeled set U: (i) the |L| : |U| ratio is much higher; (ii) L and U are not i.i.d.; (iii) L contains (feature-dependent) noisy labels. Our analysis in Figure 3 (from the submission) perhaps adds another perspective here, showing that the unlabeled data in these tasks may simply not be necessary for learning a strong model: assuming access to ground-truth labels, the final performance changes little whether one trains on the full training set or just the covered examples.\n\n**LLMs in WS.** We note first that there are many settings in which few-shot learning with LLMs may not be as effective. For example, when the task involves private data and/or narrow domain knowledge, LLMs are less likely to have been trained on sufficiently similar data or to be able to capture all the relevant nuances of the problem (e.g. a company's specific policies/ontology for classifying customer intents). Hence, we believe our focus on weak supervision settings where the supervision comes from primarily human-written labeling functions, remains relevant to real world practice.\n\nNonetheless, we see using LLMs in WS as an interesting setting. One nuance for our work would be that using an LLM directly as a few-shot labeler (by default) does not leave any examples unlabeled. Thus, applying SSL would require some additional thresholding/abstaining mechanism. Meanwhile, LLMs may also allow for several new techniques to perform thresholding (e.g. via prompting an LLM to provide reasoning/confidence along with individual labels) as well as SSL (e.g. generating augmentations for consistency regularization). These would all be interesting to look into in future work!\n\n**Clarification of downsampling.** In our programmatic supervision setups, the distribution is indeed biased according to where the labeling functions label. We totally agree that downsampling a high-coverage WS training set at a *per-data point level* would not adequately capture the biases of actual low-coverage settings. That is why we instead downsample at a *per-LF level*, which reflects the real world process that more bias can be incurred when labeling fewer points with smaller LF sets. We provided more details about the specific subsampling procedures in Appendix D, but are happy to clarify any points in further back-and-forths."}, {"review_id": "9wljG31Nid", "reviewer": "Reviewer_vRqg", "summary": "Getting labeled training data is a bottleneck in the development of machine learning pipelines. Two families of methods that address this are weak supervision (WS), which aggregates multiple sources to produce weak labels, and semi-supervised learning (SSL), which combines a (weakly) labeled dataset and an unlabeled dataset. While there have been works that study how to utilize ideas from WS and SSL to best produce high quality models from unlabeled data, the ways they combine these ideas has been fairly ad-hoc and unstructured. This paper conducts a systematic analysis of how to merge ideas from WS and SSL, breaking it down into three design choices: thresholding (what weakly labeled data to revert to unlabeled), the SSL technique for using unlabeled data, and re-labeling, which creates a self-loop of updating weak labels. The authors find that many existing methods that merge both WS and SSL can be expressed in this design framework, and their design space search finds combinations of strategies (thresholding/ssl/re-labeling) that match existing methods' performance. They study if simpler techniques, such as thresholding based on model confidence, can match more complex techniques. They also uncover that in many cases, the SSL step does not help that much unless there is a significant performance gap due to poor and biased labeling function coverage.", "questions": "- Can this framework be applied outside of weak supervision for generic noisy soft labels?\n- Seems strange that Massive18, Banking77 and the tabular datasets have this gap from (1) and (2) but Wrench datasets don't. Any idea why? \n- In figure 4, why are you comparing thresh+ssl and thresh+re-label? Shouldn't we be studying the marginal benefit of adding SSL? \n- See above for suggestions around clarity.\n\nRebuttal acknowledgement: I have read and acknowledged the author rebuttal. It has addressed my concerns above.", "limitations": "Limitations are addressed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Originality:\n- This structured survey of ways to use SSL for WS is very novel.\n\nQuality:\n- Paper is sound with extensive empirical evaluation. The authors seek to thoroughly understand why they observe that SSL isn't helpful, and design compelling experiments to validate their hypothesis. \n\nSignificance:\n- I think such a paper is very valuable to those who work on WS/SSL. Personally, I have also thought about how to best use SSL for WS and have not been convinced why one way is better than the other - even trying to prove this theoretically has been technically challenging, as these methods all ultimately use the same amount of information (some labeling functions and an unlabeled dataset; maybe some validation data, etc.) And so perhaps the solution here is not to drill into finding better method but to systematically view these methods as combinations of simpler design choices. This perspective will also be highly useful for practitioners.\n- Random thought: These findings on unlabeled data being unnecessary are potentially connected to data pruning literature, where one can discard low confidence points and still have a good model; however, there are cases when these unlabeled points are especially hard and thus worth training on --- maybe this is one perspective to understand why some datasets have a gap between GT-cov and GT-full?", "weaknesses": "Clarity:\n- Description of methods/baselines were a bit unclear and hard to follow at times. The paper describes these methods only via text and fig 1. It would be helpful to put in the appendix some formalization of these if appropriate; for instance, the two paragraphs on re-labeling L at line 149 took a bit of time to understand (stuff around decoupling SSL and re-labeling, dynamic vs one-time scheduling).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and helpful feedback, we’re glad you appreciated the paper! To respond to your questions and comments:\n\n**Clarity.** We really appreciate your feedback here. In future revisions, we will take care to make these descriptions more explicit in the Appendix.\n\n**Connections to example difficulty.** Interesting thought! Intuitively, using (human-written) labeling functions could risk bias towards labeling “easy points” instead of  “hard points” given that they often take the form of simple rules. Indeed, our results in Appendix E more generally suggest that the gaps between GT (Cov) and GT (Full) are due to coverage bias, i.e. distributional differences between what labeling functions can label and the full data distribution. My guess is that in some cases where the GT (Cov)-GT (Full) gap exists, coverage bias could be closely tied to difficulty, e.g., LFs fail to cover examples close to the ideal decision boundary, leading to a sub-optimal model. There may be other cases though where this link does not hold; perhaps the uncovered examples are not difficult but simply poorly represented by covered ones (i.e., they are “far away” from both the ideal decision boundary and all covered examples). In future work, it would be interesting to investigate/disentangle these scenarios further!\n\n**Questions**\n\n**(1)** Correct, many of the techniques in our design space could be ported over to the setting where there are generic noisy labels. The exception would be anything that involves the label model, such as in the version of self-training where the end model is used as an LF. In the settings typically studied for learning with generic noisy labels, one also isn’t usually given estimated soft-labels to start with, so some other technique would have to be applied to get confidence estimates (i.e., if one wants to use confidence-based thresholding). \n\n**(2)** For Massive18/Banking77, one perspective on why a gap exists is that these are more complex tasks (at least in terms of the label cardinality). The datasets with relatively small label spaces in WRENCH (e.g., Yelp, Imdb, AGNews) are the ones that show smaller gaps between (1) and (2), whereas datasets like Chemprot (|Y|=10) and Semeval (|Y|=9) do show larger gaps. As for the tabular datasets, we suspect that not having a pre-trained model might be an explanation. For text datasets, it may be that pre-trained models/representations can help “smooth” over the gaps in LF-coverage. \n\n**(3)** The main conclusion we wanted to show with this set of plots is that using SSL can add value in different scenarios (e.g., as coverage levels decrease), contrary to what happens on the original WS benchmarks. For this, we saw using simpler Thresh+SSL methods as sufficient though it is certainly possible that Thresh+SSL+Re-label may further improve upon Thresh+SSL. While we don’t expect the high-level conclusions to change, the coverage range in which SSL is helpful could expand as a result. We plan to include such experiments in future revisions.\n"}, {"review_id": "NB3SmZCaaF", "reviewer": "Reviewer_62vf", "summary": "This paper empirically studies the interface between (programmatic) weak supervision (WS) and semi-supervised learning (SSL). Several existing works have tried to leverage SSL techniques and other tricks to improve the performance of weakly supervised learning, since the two settings are fairly similar on the surface. \n\nThis paper provides a taxonomy of the key techniques from this line of work, arguing that most methods differ along three main axes: thresholding, SSL technique, and re-labeling. The paper empirically explores combinations across these three axes to study if and when the SSL techniques themselves really improve WS. The best single method of the design space matches or exceeds existing methods from the literature on 5/7 benchmark datasets, indicating that design space is large enough to be representative of the current literature.\n\nSurprisingly, though, the ablation results indicate that the SSL techniques themselves are not really the main cause of good performance in the regime of the most common benchmark datasets (relatively high coverage of weak rules). Instead, thresholding and re-labeling are shown to be responsible for most of the gain of these methods. The SSL techniques are shown to help more when the coverage is (artificially) reduced.", "questions": " ", "limitations": "The authors appropriately discuss the limitations of their work.\n", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Well-written empirical study that addresses an important question for the WS field, and could influence which directions are explored by future research.\n\n- The key empirical takeaway, that existing SSL methods are mainly helpful in the low-coverage regime, can drive more research in both weak supervision and semi-supervised learning. Many WS papers have been focused on using SSL techniques to improve performance, but the empirical results from this careful study indicate that the main benefits usually come from other tricks like thresholding. This could spur more research into why these other techniques work well and also into new SSL methods that use unlabeled data to improve learning from structured label noise.\n\n- The design space of combinations is empirically shown to be powerful, exceeding or matching the state-of-the-art results on benchmark datasets.\n\n- The WRENCH benchmark datasets tend to have fairly high coverage rates (in my opinion this is one of the few weaknesses of WRENCH). In response, this paper details a carefully thought-out way of artificially inducing lower coverage to study how WS methods perform in this empirically relevant setting, and also provides two new benchmark datasets that have lower coverage rates.\n\n", "weaknesses": "- Fairly large validation sets are used to select best DM from each training iteration and also to select best combination from the design space. If we have access to (say) 300 labeled examples for validation, can't we use 100 of them for training as well? It's important to note that the authors are following most if not all other works in the WS field with this choice, and the authors appropriately cite works that mix the supervised and weakly-supervised settings. But it would be interesting to see how many of the results remain the same when much smaller (e.g., something like num_classes * 10) validation sets are used. I've found that more complicated methods like DM as LF are less robust in this setting, so that may affect some of the results (but probably not the key takeaway).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and helpful feedback, we’re glad you appreciated the paper! Regarding your point about validation sets: like you said, we chose to follow the standard setups established by other works, as our main questions were about assessing various trends we saw from the WS literature. However, we agree that these default validation sizes are probably large and it would be valuable to explore the importance of validation data as an additional factor for which methods from our design space succeed/fail. One natural hypothesis would be that as less validation data is used, performance would decrease more sharply for methods that involve more components, i.e., especially the SSL methods since they tend to introduce the most additional hyperparameters. As such, it may be even more difficult for SSL to be usefully applied. "}, {"review_id": "M0eUvpGtcT", "reviewer": "Reviewer_1c9G", "summary": "This paper studies the impacts of semi-supervised learning (SSL) for programmatic weak supervision (WS) in a systematic way. The authors define a modular design space with three key methodological considerations, thresholding, SSL Technique, re-labeling, to study the use of SSL for WS. Their results show that fairly simple methods from their design space can match the performance of complex state-of-the-art method. Also, SSL is not necessary to obtain the best performance on most WS benchmarks, but is more effective when the end model is small or WS can only label a small portion of training examples.", "questions": "1. This paper involves a lot of acronyms which made it occasionally hard to follow. For \"GT\" discussed in Section 4.5, does it refer to \"Ground Truth\"? I didn't find a definition of this acronym in the paper.", "limitations": "The authors have already discussed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a systematic design space to study WS. Most of the previous baselines can be placed into the design space which makes a thorough analysis of existing WS methods possible. Also, the paper conducts extensive experiments on multiple design choices and a wide range of datasets - the benchmark results can be beneficial to the community.\n2. The best method discovered by the design space can match the performance of more complex methods. This may arouse the interest of practitioners since they can use the proposed design space to find effective WS methods instead of doing a lot of huristic hand-craft.", "weaknesses": "1. The findings of the paper are not surprising. Though the authors claim that SSL is not necessary, the experimental results show that SSL can still bring benefits on many datasets - I cannot see why the paper emphasizes SSL is not necessary if it still helps. Another finding of the paper is that SSL is effective when WS can only label a small portion of training examples. This is easy to foresee as SSL can further exploit a large number of unlabeled examples.\n2. While this is an analysis paper, all the analyses are empirical and there is no theoretical analysis. Therefore, the conclusions are drawn from the experimental results on the benchmark datasets. It's unsure whether these datasets are representative and extensive enough to cover all cases. One claim made by this work is that \"SSL is not necessary to obtain the best performance on most WS benchmarks\". A possible reason for this is that the benchmark datasets are rather easy - according to Table 2, most of the datasets are binary classification; for those with more classes, the coverage of WS labeling is very high. I'm skeptical whether this is true for datasets in real application.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and helpful feedback! To respond to some of your questions and comments:\n\n**(W1a)**  *“The experimental results show that SSL can still bring benefits on many datasets - I cannot see why the paper emphasizes SSL is not necessary if it still helps.”*\n\nWe’d like to highlight that on the standard benchmarks, improvements enjoyed by additionally applying SSL mostly do not exceed the stdev error bars when compared to Thresh+Re-label. This is surprising, as we note, because previous works test on these benchmarks and suggest that SSL is a central reason for improved performance. For instance, the WRENCH paper posits (in their Sec. 7) that *“the superiority of COSINE suggests that uncovered data should also be used in training an end model; this inspires [the exploration of] new DM training strategies combined with SSL techniques”.* That being said, in Sec 4.5, our point is exactly that SSL can often be more impactful in scenarios that differ from these standard benchmarks. Though we tried to be explicit about these important nuances in this manuscript, we will work to make them even more upfront in future revisions!\n\n**(W1b)** *“Another finding of the paper is that SSL is effective when WS can only label a small portion of training examples. This is easy to foresee as SSL can further exploit a large number of unlabeled examples.”*\n\nFirst, we agree that this is not overly surprising, but we still view this as an important practical result given the increasing ubiquity of SSL approaches, e.g. we hope this can provide some guidance for questions such as: “at what coverage levels is SSL useful?” and “is it better to learn from fewer (more-accurate) LFs + SSL or from a larger overall set of LFs?” We also note that there are at least two more non-obvious conclusions when it comes to *high-coverage settings*: (i) even when we allow for thresholding which yields smaller datasets (i.e., more unlabeled data), SSL still helps minimally; and (ii) given we are using programmatic WS, the unlabeled examples, despite their small number, may come from key parts of the input space that are systematically abstained on by LFs; one might expect that SSL could help address this biased coverage of the LFs, but this also does not happen.\n\n**(W3)** *“A possible reason for this is that the benchmark datasets are rather easy - according to Table 2, most of the datasets are binary classification; for those with more classes, the coverage of WS labeling is very high.”*\n\nWe agree regarding your assessment of the commonly used datasets in Table 2. Indeed, this skepticism is precisely what motivated us to study WS settings that notably differ from previous benchmarks in Sec 4.5. In particular, we chose to look at:\n\n* LF sets that span a greater range of coverage levels (on the same dataset)\n* Tasks that have considerably larger label spaces (i.e., Massive18, Banking77)\n* Tabular tasks instead of text-based \n\nWhile we do not claim that these factors now capture all aspects of real applications, our conclusions from these experiments are already that the impactfulness of SSL can indeed change, as you allude to.\n\n**(Q1)** Apologies for any confusions here! GT indeed corresponds to ground-truth in Sec 4.5, which we definitely should take care to explicitly define. We also appreciate the general feedback about acronyms a lot and plan to more carefully reduce our reliance on them in future revisions. "}, {"review_id": "r3SEPOhM0q", "reviewer": "Reviewer_xhQz", "summary": "The authors propose a design space to analyze the effects of WS + SSL, and how to combine them, and on which regimes that SSL help WS. The paper's biggest conclusion is: training using unlabeled datain SSL is mainly unhelpful, except when the main issue is bad labeling function (LF) accuracy. The design space is based on 3 major axis: Thresholding, SSL technique, and re-labelling.", "questions": "See Weakness", "limitations": "Yes, limitations is addressed", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a clear framework to analyze the effect of using SSL on WS\n- The framework provides a design space that is clear and intuitive, and based on 3 axis that helps t disentangle the effect from WS or SSL\n- Important result presented on when SSL is not or is helpful to improve WS", "weaknesses": "- Perhaps analysis can also be conducted on snorkel's competitive alternative (e.g., FlyingSquid https://github.com/HazyResearch/flyingsquid)\n- While discussing label noise, its important to clarify that WS required label function to have better accuracy than random (ACC > 50%)\n- An important ablation and information need also to be included is LF accuracy.\n- For the NLP task that uses embedding, its interesting to also analyze how SSL interact with smoothness based methods to improve LF coverage: LIGER https://arxiv.org/pdf/2203.13270.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and helpful feedback! We see your comments as covering two main points, which we will respond to below:\n\n**(W1) Testing on more label models (i.e., FlyingSquid, LIGER)**\n\n**FlyingSquid (FS).** For our experiments, we chose label models (i.e., Snorkel/MajorityVoting) by looking at the results from the WRENCH paper and picking the ones which led to the strongest performance for Vanilla end model training. For FS, we saw its performance was similar in most cases, while considerably worse on several tasks (e.g., Spouse, Chemprot, Trec)– hence we did not prioritize it. Of course, the implicit assumptions here are that: (1) Vanilla performance is also an indicator of performance once our design space is applied on top; (2) our high-level conclusions should translate over across different LMs.   \n\nTo your point, it could be worth exploring assumptions (1)-(2) further.  As a preliminary experiment, we ran both Vanilla end model training and the method we identified as being the best across datasets (in Sec 4.2) on top of FS labels. From the results in Table 2 in the pdf (attached to global response), the results for FS and Snorkel are extremely similar on Imdb and Yelp. On most other datasets, Snorkel’s better performance for Vanilla translates over to better performance once our method is applied (which still improves upon Vanilla for both Snorkel/FS). The only exception is on Youtube, where FS overtakes Snorkel once our method is applied (though within fairly wide error bars). Of course, more comprehensive verification of (1)-(2) would require running many more methods from our design space on FS. Nonetheless, we believe these results provide some positive initial signal about our assumptions. \n\n**LIGER.** We agree that it would be interesting to apply our analyses on top of this approach given that it directly expands the coverage of LFs. Based on our results, we’d suspect that applying SSL on top of LIGER’s expanded LFs are likely to make less of an impact than on the labels produced by the raw LFs, but it would be nice to verify this!  While we have not yet been able to try such experiments, we would be happy to try including it in future revisions. \n\n**(W2) Discussion/Ablation of Label Noise** \n\n*“The paper's biggest conclusion is: training using unlabeled data in SSL is mainly unhelpful, except when the main issue is bad labeling function (LF) accuracy”*\n\nJust to be on the same page here, we actually see the takeaway as being the other way around: “when the main issue is bad LF accuracy (as opposed to incomplete coverage)”, we find that SSL is less likely to be helpful. Namely, our results in Sec. 4.5 show that on standard WS benchmarks (where SSL is not too helpful),  label noise is the main issue: removing label noise and training only on covered examples allows us to recover the performance of fully supervised learning. However, in settings with lower coverage, label noise is no longer the only significant issue and SSL is indeed more helpful. \n\n*\"“It’s important to clarify that WS requires label function to have better accuracy than random”*\n\nThanks for pointing this out. Especially for theoretical treatments of WS, assumptions about the informativeness of LFs are important and we’d be happy to discuss this more in future revisions. For Snorkel at least, the assumption is that LFs are **on average** better than random. Other approaches may have stronger assumptions though, and we will try to clarify this!\n\n*\"“An important ablation and information need also to be included is LF accuracy.”*\n\nWe agree that the precisions of LFs are important to consider. In the current manuscript, we do include the precision of the aggregated weak labels (i.e. the label model outputs) for each of the datasets and their full LF sets. In future revisions, we can add this information for each of the reduced coverage setups as well as some summary stats for the individual LFs (similar to Table 5 in the WRENCH paper). \n\nOne experiment we’ve started to run since the submission is to try subsampling the least accurate LFs from a candidate pool instead of the most accurate. Unsurprisingly, this results in worse (and arguably less realistic) training sets but does allow us to compare LF sets of similar coverage but different precision levels. From the preliminary results in Figure 1 of the attached pdf, we observe that on Semeval and Massive18, we get qualitatively similar results to the ones we saw before: Thresh+SSL meaningfully outperforms Thresh+Re-label at lower coverage levels but less so at higher ones. In future revisions, we’re happy to extend such an analysis to more datasets!"}]

Author rebuttal:
We thank all of the reviewers for their time and thoughtful feedback! We will respond to each review individually, using this "global response" space to upload our pdf containing the Tables/Figures for new results (which are referenced in our responses). 

Meta-review: Reviewers are positive about this paper. The paper is an analysis paper about semi supervised learning and weakly supervision. Reviewers considered the analysis is extensive with good coverage of benchmarks and methods. Reviewers consider that the paper shines a good light on how to use SSL methods for WS and draws some conclusions in the right level of nuance. Moreover, reviewers believe that this paper should be informative to researchers working in both SSL and WS. Hence acceptance is recommended.
Decision: Accept (poster)