SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Characterizing the Impacts of Semi-supervised Learning for Programmatic Weak Supervision
Jeffrey Li $^{1}$ , Jieyu Zhang $^{1}$ , Ludwig Schmidt $^{1}$ , Alexander Ratner $^{1,2}$
<sup>1</sup>University of Washington, <sup>2</sup>Norkel AI
{jwl2162, jieyuz2, schmidt, ajratner}@cs.washington.edu
Labeling training data is a critical and expensive step in producing high accuracy ML models, whether training from scratch or fine-tuning. To make labeling more efficient, two major approaches are programmatic weak supervision (WS) and semi-supervised learning (SSL). More recent works have either explicitly or implicitly used techniques at their intersection, but in various complex and ad hoc ways. In this work, we define a simple, modular design space to study the use of SSL techniques for WS more systematically. Surprisingly, we find that fairly simple methods from our design space match the performance of more complex state-of-the-art methods, averaging a 3 p.p. increase in accuracy/F1-score across 8 standard WS benchmarks. Further, we provide practical guidance on when different components are worth their added complexity and training costs. Contrary to current understanding, we find SSL is not necessary to obtain the best performance on most existing WS benchmarks but is more effective when: (1) end models are smaller, and (2) WS provides labels for only a small portion of training examples.
Learning with limited labels is a fundamental challenge in machine learning (ML) applications [12, 35]. To address the significant costs of hand-labeling training sets, programmatic weak supervision (WS) has emerged as a high impact research area [25, 31], where the aim is to learn from multiple cheaper sources of noisy labels. Meanwhile, semi-supervised learning (SSL) is a more classical direction with similar high-level motivations. Instead of generating larger quantities of noisy labels, SSL aims to directly leverage additional unlabeled data. It seems natural that these two fields could be applied productively with one another, yet their intersection has not been systematically studied.
In this work, we anchor on WS approaches and study whether they can be enhanced using techniques from SSL. At a high-level, most WS methods consist of two steps. First, a label model aggregates a set of weak label sources to noisily label a training set. Commonly, these sources take the form of user-written heuristics (e.g., for sentiment analysis, a user may check for the keyword "great" to provide the label "positive"). Second, an end model is learned on this training set. Crucially, weak sources can often abstain (e.g., the absence of "great" may not imply "negative"), leaving certain examples to remain unlabeled and thereby unused. This presents a natural opportunity to use SSL.
Indeed, this approach of using SSL in WS settings has motivated several recent methods [30, 26, 11, 9]. Though these works often attribute their observed improvements to their usage of unlabeled data [33, 30, 26], they also incorporate various algorithmic components in addition to SSL. Thus, we lack clarity about the precise contributions of SSL and whether simpler methods might also suffice. Also, while varying the amount of unlabeled data is crucial when evaluating SSL techniques [22], previous WS benchmarks contain little diversity along this key dimension. Here, we conduct a more systematic study of how useful SSL is in various WS settings, as well as how and when to best employ it.
Specifically, we first organize the intersection between SSL and WS by proposing an explicit design space, centered around disentangling the following key methodological considerations:
(1) Thresholding: What to treat as (un)labeled? Because WS uses heuristics for labeling, it often can only provide labels for a subset of examples, leaving the rest as unlabeled. Further, it can also be beneficial to additionally remove some (likely) incorrect labels provided by WS, as shown by [15]. Since unlabeled data can result in multiple ways when using WS, we view "what to treat as unlabeled" as a non-trivial and first-class axis in our design space.
(2) SSL Technique: How to use unlabeled examples? After deciding what data should be treated as unlabeled, one can leverage these examples by simply using any existing SSL technique. Most recent proposals do so via self-training, which uses the end model to periodically provide labels for unlabeled examples; we also try other out-of-the-box SSL methods.
(3) Re-labeling: Whether to update weak labels during end model training? Because some labels from WS are incorrect, it can be helpful to re-label a training set with the end model during training. This approach, increasingly used by WS methods [30, 11, 4], is often packaged as part of self-training-based SSL. Here, we identify that re-labeling and SSL can be independently employed, and we aim to disentangle their impacts.
With our design space, we can organize previous works and modularly generate a variety of methods. We test these methods on several standard WS benchmarks, finding that our design space is sufficient for matching the performance of more complex state-of-the-art methods. We then compare methods within our design space to ablate the importance of each axis and so provide guidance on when each is worth using (and thus more carefully exploring). We summarize our key findings as follows:
- By searching over our design space, we identify two methods that at least match all previous baselines on 6 of 8 WS benchmarks, averaging a 3 p.p. increase in accuracy/F1-score.
- While previous works emphasize utilizing data left unlabeled by WS sources, we find that on 6 of 8 benchmark tasks, SSL is actually not necessary for achieving high performance: thresholding and re-labeling can recover  $90\%$  of the gains achieved by also using SSL.
- To explain SSL's lack of impact, we find that the small amounts of unlabeled data in these 6 benchmarks (i.e.,  $< 31\%$ ) are largely unnecessary for the underlying tasks; when using clean instead of weak labels, ignoring unlabeled examples drops test accuracy by  $< 2.5$  p.p.
- In contrast, when WS sources leave more data as unlabeled (i.e.,  $>65\%$ ), SSL is generally worth prioritizing; using it can improve upon thresholding and re-labeling by up to 16 p.p.
SSL for WS tasks. Methods in WS have increasingly turned to SSL to improve end model training. This includes DENOISE [26], which incorporates the temporal ensembling SSL algorithm [14], as well as COSINE [30] and KeyClass [9], which both use self-training [16]. Other works [11, 18, 6, 20, 19, 23, 1] apply SSL when learning from weak labels plus a small set of clean labels. However, these methods fundamentally differ in their use of SSL, i.e., they define a labeled unlabeled split based on cleanly versus weakly labeled examples. Further, integrating clean labels into WS enables a greater variety of specialized strategies, so we do not consider this setting in our study. Likewise, [2] assumes additional supervision via heuristics for which examples share the same labels, similar in spirit to consistency-based SSL methods. Finally, we defer a background on SSL in its own right to Section 3.
SSL for learning with noisy labels. SSL techniques have been regularly applied in the literature concerning learning from noisy labels [17, 7, 13]. Though this setting is similar to WS, its main difference is that label noise comes from a single "black-box" noising process instead of from multiple explicit WS sources. Thus, the resulting label noise patterns, often also artificially injected in input-independent ways [28], may differ significantly from those in WS. Furthermore, the WS setting contains some examples with no labels since WS sources may abstain.
Subset selection in WS. [15] showcases the broad utility of more carefully selecting subsets of weak labels before end model training. In our work, we consider subset selection in the greater context of two other trends in the WS literature, applying SSL and re-labeling. Compared to the core method of [15], we also try a simpler baseline, similar to [9], based on thresholding the existing confidences produced for weak labels. We find that this method offers a competitive alternative on most datasets.
Figure 1: Overview of our design space. In WS, a label model produces a weakly labeled training set  $\tilde{D}$  which is used to train a discriminative end model. Our design space overlays three decision points on this pipeline: (1) thresholding, which filters out some weak labels in  $\tilde{D}$  to return labeled and unlabeled sets  $L$  and  $U$ ; (2) SSL technique, which defines how the end model can still use  $U$  during training; (3) re-labeling, which uses the end model to update previously used weak labels.
3 Design Space
In this section, we first formalize WS and SSL. Then, we describe how our design space overlays on WS, detailing its three key axes along with the specific instantiations of each that we explore in our experiments. Finally, we contextualize which parts of existing work fit within our framework.
3.1 Problem Formalization
Weak supervision (WS) starts with an unlabeled training set  $D = \{x_{i}\}_{i=1}^{n} \in \mathcal{X}^{n}$  drawn from an underlying distribution  $(x, y) \sim P$ . Labels are provided by a set of labeling functions (LFs)  $\{\lambda_{j}\}_{j=1}^{m}$ , where each  $\lambda_{j}: \mathcal{X} \to \mathcal{Y} \cup \{\emptyset\}$  either labels or abstains (denoted as  $\lambda_{j}(x_{i}) = \emptyset$ ) on each  $x_{i}$ . The goal is to train a discriminative end model (DM)  $f: \mathcal{X} \to \mathcal{Y}$  that performs well on  $P$ . Canonically, WS methods contain two components. First, a label model (LM) observes  $\left(D, \{\lambda_{j}\}_{j=1}^{m}\right)$  and outputs a weakly labeled training set  $\tilde{D} = \{(x_{i}, \tilde{y}_{i})\}_{i=1}^{n}$ . Second,  $f$  is trained using  $\tilde{D}$ . Often, the LM models  $\tilde{y}_{i}$  probabilistically, allowing  $\tilde{y}_{i}$  to be a soft-label, a vector of probabilities over  $\mathcal{Y}$ .
Problematically,  $\tilde{D}$  may contain several uncovered examples, where  $\lambda_{j}(x_{i}) = \emptyset$  for all  $\lambda_{j}$ . Thus, the default practice is then to train  $f$  only on covered examples that received at least one non-abstaining LF vote. Concurrently,  $\tilde{D}$  may also contain several incorrect labels, where  $\tilde{y}_i\neq y_i$  (or in the soft-label case,  $\arg \max_{c\in \mathcal{V}}\tilde{y}_i[c]\neq y_i$  where  $\tilde{y}_i[c]$  is the probability assigned to class  $c$ ). Both issues may lead to sub-optimal performance compared to standard supervised learning. Here, we focus on whether SSL lets us more effectively learn from  $\tilde{D}$  in light of these challenges.
Semi-supervised learning (SSL) assumes access to a labeled dataset  $L = \{(x_i^\ell, y_i)\}_{i=1}^{|L|}$  as well as an unlabeled dataset  $U = \{x_i^u\}_{i=1}^{|U|}$ . The goal is to obtain a model that performs well on  $L$ 's underlying distribution despite the limited size of  $L$ . Though not strictly required, it is often assumed that  $|L| << |U|$  and both are drawn from the same test distribution. Generally, each SSL method makes a core assumption about how  $P(x)$  relates to  $P(y|x)$ . Popular categories of methods include entropy-minimization [10, 16], which assumes  $P(y|x)$  is uncertain only when  $P(x)$  is small, and consistency regularization [21, 27, 14], which assumes local smoothness of  $P(y|x)$  when  $x \sim P(x)$ .
3.2 A Simple Design Space
By considering the standard WS pipeline, we anchor our design space on three natural decision points as shown in Figure 1: (1) thresholding strategy, (2) SSL technique, and (3) whether to re-label previously used weak labels. Importantly, our design space is agnostic to the specifications of label model and end model, though particular choices for each could affect which methods work best.
Method	Thresholding	SSL Technique	Re-labeling L	Novel LM	Add. Reg.	Add. Labels
Vanilla WS	Coverage	-	-	-	-	-
Cutstat [1]	Cut-based (one-time)	-	-	-	-	-
Denoise [2]	Coverage	Temp. Ensembling	Self-Train LM	✓	-	-
Weasel [3]	-	-	Agreement-based	✓	-	-
Cosine [4]	Confidence (dyn.)	Self-Train	DM directly	-	✓	-
KeyClass [5]	Confidence (dyn.)	Self-Train	DM directly	-	✓	-
ASTRA [11]	Coverage	Self-Train (DM as LF)	DM as LF	✓	-	✓
LPA+WL [23]	-	Label Propagation	-	-	-	✓
SPEAR [18]	-	Entropy Min	Agreement-based	✓	✓	✓
Table 1: Contextualizing methods within (middle columns) and outside (rightmost) our design space. "Novel LM" refers to a method introducing its own label model. "Add. Reg." refers to using additional regularizations, such as contrastive losses and soft-label re-normalization. "Add. Labels" refers to assuming an additional set of clean labels. For methods using clean labels, we do not compare to their results directly, but we can still place elements of their approaches in our design space.
Thresholding. To apply SSL, we must first define which examples in  $\tilde{D}$  should be considered as part of  $L$  and  $U$ , respectively. Though this is baked into the problem definition in traditional SSL settings, it is a non-trivial choice in WS settings. As a default, standard WS pipelines select  $L$  based on labeling function coverage, ignoring uncovered examples on which all  $\lambda_{j}$  abstain. However, as shown by [15], removing additional examples from  $\tilde{D}$  can help if they are more likely to be incorrectly labeled. In our work, we consider the following strategies for partitioning  $\tilde{D}$  into  $L$  and  $U$ :
- Coverage-based (default): This removes uncovered points  $\{(x_i, \tilde{y}_i) : \lambda_j(x_i) = \emptyset, \forall j\}$ .
- Confidence-based: Since most label models can output probabilistic labels, a basic yet under-explored approach tried by [9] is to remove examples that have low estimated confidence, here defined as the highest probability assigned to a class. Formally, this removes the examples  $\{(x_i,\tilde{y}_i):\max_{c\in \mathcal{Y}}\tilde{y}_i[c] < \xi \}$  for some threshold  $|\mathcal{Y}|^{-1} < \xi < 1$
- Cut-based: This is the method of [15], which at a high-level removes examples whose labels differ most from those of their nearest neighbors in some pre-trained embedding space.
SSL Techniques. After partitioning  $\tilde{D}$  into  $L$  and  $U$ , we next consider how to still make use of the examples in  $U$  to train better end models. By default, WS methods ignore  $U$  altogether, but we may instead apply a variety of strategies, such as plugging in any existing SSL technique. Like [22], we limit ourselves to SSL techniques that add unsupervised loss terms during training since these methods tend to achieve the best performance on traditional SSL benchmarks. We also pick representative methods from their taxonomy of approaches:
- Entropy minimization (EM): A classical approach, EM [10] penalizes uncertain predictions on unlabeled examples, aiming to place a decision boundary in low-density regions of  $P(x)$ .
- Self-training (ST): Self-training, along with the closely related pseudo-labeling [16, 3], are traditional SSL methods that iteratively use current predictions on unlabeled examples as true labels. Uniquely in the WS setting, we also consider self-training the label model and end model together. Inspired by [11], we try one way to do so by feeding the end model as an additional labeling function  $\lambda_{m+1}$  used to re-fit the LM. We call this ST (DM as LF).
- Consistency regularization: These methods encourage similar predictions on (realistic) perturbations of unlabeled examples. We use VAT [21] as it does not rely on data augmentations, which are less straightforward for text datasets, the majority of WS tasks. On some tasks, we are also able to use UDA [29], generating perturbations via En-De backtranslations.
Re-labeling  $L$ . Traditional SSL assumes that all labels in  $L$  are correct. However, in WS, we may also consider using the end model to correct labels in  $L$  as it trains. This increasingly popular technique in WS methods [4, 30, 11, 18] is often packaged with traditional self-training as an overall "SSL method" [30, 11]: instead of using the current model to label just  $U$ , these methods also do so for  $L$ . However, this makes it difficult to discern whether these methods improve performance because they leverage  $U$  (i.e., apply SSL) or simply because they clean up labels in  $L$ . Therefore, in our study, we explicitly decouple the use of SSL and re-labeling  $L$  as two independent decisions; i.e., we can re-label  $L$  regardless of whether we self-train on  $U$ , or indeed use any specific SSL technique.
Specifically, we consider two forms of re-labeling analogous to the two types self-training for SSL: (i) using the end model's predictions directly; (ii) using an LM re-fit with the end model as an LF. For
Datasets	|Y|	|Train|	|Test|	#LFs	Coverage	Snork. Precision	Metric
IMDb	2	20000	2500	5	87.6%	74.4%	Acc.
Yelp	2	30400	3800	8	82.8%	75.4%	Acc.
Youtube	2	1586	250	10	87.7%	87.0%	Acc.
AgNews	4	96000	12000	9	69.1%	82.5%	Acc.
Trec	6	4965	500	68	95.1%	60.0%	Acc.
Spouse	2	22254	2701	9	25.8%	65.6% (on Val)	F1 (binary)
Chemprot	10	12861	1607	26	85.6%	58.0%	Acc.
Census	2	10083	16281	83	99.1%	58.0%	F1 (binary)
Table 2: Statistics for the WS benchmarks that we use. Groupings are by task type: text, relation, and tabular classification. Coverage is the percentage of training examples on which at least one LF does not abstain. "Snork. precision" refers to the accuracy of the Snorkel LM on the covered set of inputs.
tractability, however, we only consider re-labeling  $L$  in manner (i) when using SSL, except when the SSL technique is already ST (DM as LF). Also, since re-labeling effectively loops the two-stage WS pipeline back onto itself,  $N$  rounds of re-labeling could in principle be paired with  $N + 1$  separate choices for thresholding and SSL. However, to reduce this search space, we only ever apply the same SSL method across all rounds. For thresholding, we try two possible schedules: dynamic, which applies the same thresholding after each re-labeling round and one-time, which fixes the  $L / U$  split for all rounds of end model training. Finally, some recent methods re-label by using a specialized LM that can be jointly learned with the end model [4, 18]. We do not include this type of agreement-based re-labeling in our study, instead focusing on methods agnostic to the form of LM.
3.3 Contextualizing previous works
With our design space, we can contextualize several recent WS methods, as shown in Table 1. Overall, this table demonstrates the lack of systematic exploration. For instance, few works perform any thresholding beyond coverage-based, and only [15] and [9] threshold initial LM outputs. Furthermore, assessing the impact of SSL in WS settings is muddled because methods often incorporate several different techniques. A salient example is that COSINE [30] was found by WRENCH [33] to obtain state-of-the-art performance, with both works championing the usage of unlabeled data as a key driver of improvements; however, what COSINE refers to as self-training actually involves pseudolabeling the whole dataset, thereby performing both SSL and re-labeling. Further, many methods use techniques outside of our design space entirely, such as novel LMs [26, 4, 11, 18], contrastive learning [30], and soft-label re-normalization [30, 9]. As a result, it remains unclear whether our three axes are necessary or even sufficient to achieve optimal performance. Though our design space is by no means exhaustive, we believe it offers a useful starting point to answer such questions.
We begin by describing our experiment setup and various baselines in Section 4.1. Then, Section 4.2 explores how our design space yields methods that perform at least as well as the aforementioned baselines. In Section 4.3 and Section 4.4, we conduct extensive ablations on our three axes, finding that SSL is surprisingly unnecessary on most existing WS benchmarks. Finally, Section 4.5 explains this phenomenon and explores settings in which SSL is more helpful.
Datasets and models. We use 8 classification datasets (see Table 2) and largely follow the end model configurations from WRENCH [33] with a few changes to the hyperparameter grid (Appendix A). For NLP tasks, we both fine-tune RoBERTa pre-trained models and train MLP classification heads on (frozen) RoBERTa embeddings, deferring all results for the latter to the appendix. For tabular tasks, we just train MLPs on raw features. We tune all methods on a shared hyperparameter budget of 50 trials for full RoBERTa fine-tuning and 300 trials for MLPs. All reported test performances are then averages over over three additional test runs, while all error bars are the standard deviations over these runs. Finally, though our design space is compatible with any LM, we use the soft-labels produced by the Snorkel LM from [24]. However, we test robustness to this choice by also trying Majority Voting when comparing with existing methods. From the conclusions of [33], these were most consistently the best LMs across many benchmark tasks.
Figure 2: Test performance for  $LM =$  Snorkel. We compare the best single method found (for each end model) from our design space (green) with the best methods found per-dataset (turquoise, narrower error caps) as well as four recent methods from the literature. All performances are plotted relative to that of vanilla training (deferring absolute metrics to Table 6 in Appendix B). Note that for Census, a tabular task, the best single method is different since we report the one we found for MLPs.
Baselines. We consider five baselines, corresponding to the WRENCH implementations of existing state-of-the-art methods. As Table 1 shows, each incorporates some aspects of our design space, while not necessarily being contained completely within it. We list these methods as follows:
Vanilla: applies the default WS pipeline, using coverage thresholding and no SSL or re-labeling.
Cutstat + {Snorkel, MV}: uses the thresholding method from [15], with no alterations. Notably, this method is contained within our design space as a pure thresholding method.
COSINE + {Snorkel, MV}: modifies [30] in two ways when appropriate for fair comparison; the LM used is Snorkel instead of MV except as shown in Table 7, and the DM is an MLP in Table 8.
Denoise + {Snorkel, MV}: applies [26] except using Snorkel instead of MV to initialize their label aggregator as shown in Table 7. Also, we use RoBERTa-based end models instead of BERT.
Weasel: applies the method from [4] except to train our RoBERTa-based end models for text datasets.
4.2 Does our design space yield competitive methods?
We first demonstrate that simple methods from our design space can at least match the performance of all previous methods. When the end model is RoBERTa, our best single method, shown in Figure 2, is the combination of (dynamic) confidence thresholding, ST (DM as LF), and re-labeling. This method at least matches the other baselines on 5 of the 7 tasks. On Chemprot, only Cutstat results in a better point estimate. However, we may also swap in cut-based thresholding into our method, which results a state-of-the-art accuracy of 0.601 (0.008) (as shown in Appendix B, this method is similarly strong when the LM is instead Majority Voting). Finally, Figure 2 illustrates that searching our design space exhaustively per dataset—while not necessarily practical—can be even more powerful, improving RoBERTa fine-tuning performance by 1.6 p.p. on average and closing  $16\%$  of the remaining gap between our baselines and fully supervised training. Significantly, our methods are strong despite not using additional techniques employed by recent works (as shown in Table 1). This provides empirical justification for focusing only on methods from within our design space in subsequent analyses.
4.3 Which axes are most important on current benchmarks?