SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Though our design space is sufficient for strong performance, we now show that SSL is, by and large, not necessary on current benchmarks. Specifically, we conduct an extensive ablation comparing all eight possible subsets of including (or ignoring) each axis, presenting the results in Tables 3 and 9. As an example, for the row "Thresh + SSL," we run every combination within the cross-product of
Method	IMDb	Yelp	Youtube	AGNews	TREC	Spouse	Chem.	Mean	w/o Spouse
Vanilla	0.874	0.925	0.948	0.867	0.646	0.215	0.572	0.721	0.805
(0.002)	(0.017)	(0.009)	(0.008)	(0.011)	(0.012)	(0.002)	(0.004)	(0.004)
Thresh Alone	0.885	0.946	0.951	0.882	0.706	0.263	0.587	0.746	0.826
(0.002)	(0.006)	(0.002)	(0.001)	(0.015)	(0.071)	(0.005)	(0.010)	(0.003)
SSL Alone	0.885	0.941	0.955	0.885	0.660	0.343	0.593	0.752	0.820
(0.002)	(0.008)	(0.005)	(0.002)	(0.074)	(0.075)	(0.010)	(0.015)	(0.013)
Re-label Alone	0.901	0.944	0.952	0.877	0.700	0.332	0.577	0.755	0.825
(0.013)	(0.004)	(0.006)	(0.002)	(0.014)	(0.105)	(0.010)	(0.015)	(0.004)
Thresh + SSL	0.885	0.959	0.943	0.887	0.732	0.529	0.597	0.790	0.834
(0.018)	(0.002)	(0.008)	(0.005)	(0.014)	(0.057)	(0.009)	(0.009)	(0.004)
Thresh + Re-label	0.905	0.961	0.952	0.883	0.748	0.236	0.600	0.755	0.841
(0.011)	(0.002)	(0.006)	(0.005)	(0.023)	(0.085)	(0.006)	(0.013)	(0.005)
SSL + Re-label	0.894	0.953	0.951	0.882	0.702	0.365	0.583	0.761	0.828
(0.008)	(0.008)	(0.002)	(0.001)	(0.035)	(0.073)	(0.003)	(0.012)	(0.006)
Thresh + SSL + Re-label	0.907	0.965	0.949	0.885	0.765	0.531	0.601	0.800	0.845
(0.004)	(0.002)	(0.010)	(0.003)	(0.012)	(0.039)	(0.008)	(0.006)	(0.003)
Entire Design Space	0.907	0.965	0.952	0.885	0.765	0.531	0.601	0.801	0.846
(0.004)	(0.002)	(0.006)	(0.002)	(0.012)	(0.039)	(0.008)	(0.006)	(0.003)
Fully supervised	0.932	0.976	0.967	0.918	0.966	-	0.894	-	0.943
(0.005)	(0.001)	(0.013)	(0.006)	(0.004)	-	(0.012)	-	(0.003)
all non-default thresholding and SSL techniques (i.e., {Conf-based, Cut-based}  $\times$  {EM, VAT, UDA, ST, ST w/ DM}) and then select the best of these combinations using the validation set. For "Entire Design Space," we perform this method selection procedure over all methods from the design space. $^1$
From this analysis, we observe that SSL helps only in limited scenarios. Including SSL on Spouse yields a 0.2 increase in F1-score compared to not using any form of SSL. However, on all the other six tasks, "Thresh + Re-label" performs at least within a standard deviation of the best method, making up  $90\%$  of the gap between "Vanilla" and "Thresh + SSL + Re-label." One explanation for this result is the distinctly lower coverage LF set for Spouse, a factor we explore in depth in Sec 4.5. Further, model size may also play a role. In the corresponding Table 9 (Appendix B) for MLPs, "Thresh + Re-label" can make up only  $68\%$  of the gap between "Vanilla" and "Thresh + SSL + Re-label."
4.4 What are the best instantiations of each axis?
Having compared the three axes at a macro-level, we now zoom in on each. Specifically, we compare all implementations of a given axis when paired with the best possible setting of the other two. For thresholding and SSL, we find that simple or previously underexplored approaches are worth using.
Thresholding. Examining Table 4, we see that thresholding is significantly helpful in most cases. This extends the conclusions of [15], showing that removing labels still helps even when also using SSL and re-labeling. Interestingly, the simpler confidence-based threshold matches the cut-based method (within error bars) on all datasets except Chemprot.
SSL and Re-labeling. For these two axes, we observe largely similar trends in Tables 10 and 11 in Appendix C.2. SSL and re-labeling are each only strictly nec
essary for a minority of datasets, outperforming "no SSL" and "no re-labeling" (beyond error bars) on just one and two datasets, respectively. As such, all SSL methods also tend to perform similarly. For MLP models, though, where SSL is more useful, we see from Table 14 that VAT and ST (DM as LF) are the most consistent. No other method comes within error bars on four datasets.
Table 3: Axis ablation for  $LM =$  Snorkel,  $DM =$  RoBERTa. Each entry corresponds to picking a specific method within our design space using validation tuning. Blue numbers are the highest for a given dataset, while bold numbers are those within error bars of the best result.
IMDb	Yelp	AGNews	TREC	Spouse	Chem.
Cov	0.901(0.013)	0.953(0.008)	0.885(0.002)	0.702(0.035)	0.365(0.073)	0.593(0.010)
Cut	0.907(0.004)	0.965(0.000)	0.887(0.005)	0.765(0.012)	0.531(0.007)	0.605(0.002)
Conf	0.906(0.005)	0.965(0.002)	0.884(0.002)	0.741(0.029)	0.531(0.039)	0.593(0.010)
Table 4: Closer look at thresholding for DM = RoBERTa. We report the best performance of any method that uses each type of thresholding. Blue indicates the best result while bold indicates being within error bars of the best.
<sup>1</sup>This differs from "Thresh + SSL + Re-label" because methods for that row must employ non-default choices.
4.5 When is SSL more useful?
To explain why SSL is largely redundant on most existing WS benchmarks, we show that the unlabeled data in each task is generally unnecessary for learning strong models. However, when using lower coverage LF sets, a setting not captured by these benchmarks, ignoring unlabeled data can significantly compromise performance, allowing SSL more room to be impactful.
Data gaps in WS. We can think of any WS training set  $L = \{(x_i^\ell, \tilde{y}_i)\}_{i=1}^{|L|} \subseteq \tilde{D}$  as suffering from:
(1) Limited size: Since not all examples are labeled, the quantity of labels  $|L|$  may be insufficient.
(2) Coverage bias: Since LFs abstain based on feature-dependent rules, the inputs in  $L$  represent a biased subpopulation of the true distribution that the test set is drawn from.
(3) Label noise: Since LFs are just heuristics, labels  $\tilde{y}_i$  can be biased towards incorrect classes.
We hypothesize that SSL adds more unique value within our design space when gaps (1) and (2) are more significant. When gap (3) is the only dominating factor, there is less reason to expect SSL to outperform thresholding and re-labeling since the latter two more directly attempt to address label noise. However, these two axes still do not use the unlabeled examples, which cause gaps (1) and (2).
Label noise (not unlabeled data) is the main gap on existing benchmarks.
To measure the relative importance of the three data gaps, we compare the following models:
$GT(Cov)$ : the model trained on the ground-truth labels for only LF-covered inputs, removing gap (3) but retaining gaps (1) and (2).
$GT$  (Full): the model trained with a clean and fully labeled version of  $D$ , removing all gaps.
Given our hypothesis, we would expect these models to perform similarly on WRENCH benchmarks. Indeed, as shown in Figure 3, the largest gap between them is  $< 2.5$  p.p.
SSL helps more when coverage is lower. While SSL's ineffectiveness on existing benchmarks corresponds to label noise being the predominant data gap, we would ideally also show that SSL is more useful when the other two gaps are significant, i.e., when  $GT(Cov)$  performs markedly worse than  $GT(\text{Full})$ . One natural way to explore this would be to test on lower coverage LF sets, which result in more unla
Figure 3: Measuring the importance of unlabeled data on WS benchmarks. Assuming no label noise, removing unlabeled data for  $GT(Cov)$  decreases performance minimally compared to  $GT(Full)$ .
beled data. However, most existing benchmarks have high coverage, often above  $80\%$  as shown in Table 2. To overcome this limitation, we first explore a wider range of coverage levels by subsampling or generating LFs on existing datasets. We also create two new WS text classification tasks based on publicly available datasets, Massive18 [8] and Banking77 [5]; these tasks have larger label spaces than all WRENCH tasks and therefore require more effort (i.e., LFs) to obtain high coverage. Finally, we explore the less frequently studied tabular setting, using Mushroom, Spambase, and PhishingWebsites in a setup similar to that of [32]. We defer details on all these data/LF sets to Appendix D.
From this analysis, we first see that smaller coverage levels (within the ranges we test) do not always cause  $GT(Cov)$  to perform poorly. On some tasks (top two rows of Figure 4), including the three tabular and our two new ones,  $GT(Cov)$  performs significantly worse as coverage decreases. $^3$  In contrast, on IMDb, Yelp, and AGNews (bottom row of Figure 4),  $GT(Cov)$  surprisingly comes within  $2\%$  of  $GT(Full)$  even when coverage drops to  $10 - 20\%$ .
Importantly, the partitioning of datasets based on the performance of  $GT(Cov)$  also corresponds to the effectiveness of using SSL over not using it. For each LF set used, we run the methods within
2 Unfortunately, Spouse does not have ground-truth training labels, preventing us from adding it to Figure 3.
3In Appendix E, we show that coverage bias (not limited size) is the main cause of these performance drops.
Figure 4: Impacts of coverage level on data gaps and the effectiveness of SSL. Plotting  $GT$  (Full) (gray-dashed) and  $GT$  (Cov) (blue), we measure the impacts of removing label noise across nested LF subsets. We also compare the effectiveness of Thresh + SSL (green) to Thresh + Re-label (red). For datasets more impacted by coverage bias and limited size (larger gaps between dashed and blue lines), SSL adds more unique value. "(Ours)" refers to datasets we introduce. "(Tab.)" refers to tabular tasks.
a reduced version of our design space: we allow for confidence thresholding and try both versions of self-training to perform SSL (i.e., by labeling points in  $U$ ) or to re-label (i.e., by labeling points in  $L$ ). For the tabular tasks, we also try VAT for the SSL technique because of its effectiveness for MLPs. We plot the best performing methods for both "Thresh + SSL" and "Thresh + Re-label" in Figure 4. As shown in the top two rows, "Thresh + SSL" is enough to consistently outperform "Thresh + Re-label" at lower coverage levels. In contrast, in the bottom row, where  $GT(Cov)$  drops in performance by  $< 2$  p.p., "Thresh + SSL" continues to perform within errors of "Thresh + Re-label."
Overall, these results suggest that SSL can indeed be useful in WS settings that differ from those captured by standard benchmarks. At lower coverage levels (i.e.,  $< 35\%$ ), SSL is consistently worth trying; in practice, SSL may allow users to reduce the number of LFs they need to write in order to achieve a particular target performance. On our tabular tasks, these savings seem especially impactful; SSL allows one to achieve performance equivalent to having all 40 LFs (highest coverage plotted) with at most 20 LFs (second lowest coverage plotted).
We constructed a design space for combining SSL and WS, using it to both contextualize and match the performance of state-of-the-art WS methods. We show that on existing WS benchmarks, using the unlabeled data is surprisingly not essential. However, SSL can be more useful when training MLPs and when the LFs cover fewer examples. Some limitations of our work include: (1) a lack of finer-grained heuristics for more efficiently navigating our design space when given a new task; (2) relying on ground-truth labels to measure the relative importance of different data gaps; (3) using automated ways to generate low coverage LF sets which may not fully represent those written by real users. We see all of these limitations as seeding important directions for future work.
4Unlike in Section 4.3, here we also allow "Thresh" in "Thresh + {SSL, Re-label}" to be coverage-based.
[31] Jieyu Zhang, Cheng-Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner. A survey on programmatic weak supervision. arXiv preprint arXiv:2202.05433, 2022.
[32] Jieyu Zhang, Haonan Wang, Cheng-Yu Hsieh, and Alexander J Ratner. Understanding programmatic weak supervision via source-aware influence function. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 2862-2875. Curran Associates, Inc., 2022.
[33] Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner. WRENCH: A comprehensive benchmark for weak supervision. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
[34] Wenxuan Zhou, Hongtao Lin, Bill Yuchen Lin, Ziqi Wang, Junyi Du, Leonardo Neves, and Xiang Ren. Nero: A neural rule grounding framework for label-efficient relation extraction. In Proceedings of The Web Conference 2020, WWW '20, page 2166-2176. Association for Computing Machinery, 2020.
[35] Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. Can chatgpt reproduce human-generated labels? a study of social computing tasks, 2023.
A.1 Search Spaces
Method	Hyperparameters	Description	Range
Snorkel LM	lr	learning rate	1e-5,1e-4,1e-3,1e-2,1e-1
weight Decay	weight decay	1e-5,1e-4,1e-3,1e-2,1e-1
num_epoch	the number of training epochs	5,10,50,100,200
MLP	batch_size	batch size	32,128,512
lr	learning rate	1e-4,1e-3,1e-2
dropout	dropout probability	0.2, 0.0
weight Decay	weight decay	0.0
num_layer	the number of hidden layers	2
hidden_size	the hidden size of MLP layers	128, 256, 512
BERT	batch_size	batch size	32
lr	learning rate	1e-5,3e-5,5e-5
weight Decay	weight decay	1e-4
COSINE	T	period for updating pseudo-labels	50,100,200
ξ	confidence threshold	0.1, 0.3, 0.5, 0.7, 0.9
λ	weight for confidence regularization	0.01,0.05,0.1
μ	weight for contrastive regularization	1
γ	margin for contrastive regularization	1
Denoise	α	momentum term for temporal ensembling	0.6
d	size of hidden layer	2^6 - 2^9
c1	coefficient of denoiser loss	0.1,0.3,0.5,0.7,0.9
c2	coefficient of classifier loss	0.1,0.3,0.5,0.7,0.9
c3	coefficient of unsupervised self-training loss	1-c2-c1
WeaSEL	γ	temperature	1.0, 0.33
d	size of hidden layer	2^6 - 2^9
dropout	dropout prob	0.3
Confidence Threshold	threshold	minimum confidence to keep	np.inspace(1/|V| + 0.1, 0.9, 9)
Cutstat Thresh	percentile	what percentile as cut-off	np.inspace(0.1, 0.9, 9)
Self-Training	T	period for updating pseudo-labels	50, 100
Self-Train (DM as LF)	T	period for updating pseudo-labels	50, 100
EM	λ	the coefficient for EM loss	1, 0.5, 1e-1, 1e-2, 1e-3
VAT	λ	the coefficient for VAT unsupervised loss	1, 0.5, 1e-1, 1e-2, 1e-3
VAT ip	Iterations of the power method for VAT	1
ξ	Finite difference for approximation in VAT	0.05, 0.1, 0.5, 1, 5
ε	VAT Perturbation distance	1e-6, 1e-3, 1, 2.5, 5
UDA	λ	the coefficient for UDA unsupervised loss	4, 2, 1 0.5, 1e-1
tsa_sched	schedule for training signal annealing	none, log
τ	temperature for prediction sharpening	1, 0.85, 0.4
Table 5: Search spaces organized by type of method: label models, end models, baselines, thresholding, and SSL techniques.
A.2 Implementation Details
Here we provide high-level details on the various components of our design space. For further details, please see our codebase at https://github.com/jeffreywpli/SSL4WS.
Label Models. Of the two label models that we use, we must tune parameters only for Snorkel. We tune over the search space in Table 5 once per dataset; we then fix the hyperparameters and seed for fitting Snorkel to ensure the same exact weak labels are given to all methods. This seed is chosen from a pool of three, also based on the validation set.
End Models. For MLPs, we increase the range of hidden sizes compared to WRENCH and tune for dropout instead of weight decay. For RoBERTa, we do not tune the batch size in order to reduce the search space, observing minimal differences compared to using an alternative batch size of 16. Following WRENCH, we perform early stopping based on validation performance for all methods. Specifically, we use a patience of 1000 steps for MLPs and 100 steps for RoBERTa.
Thresholding. Whenever applying dynamic versions of thresholding, we use the same threshold value across different rounds of end model training for simplicity. Setting separate thresholds for different stages of learning (or adaptively) could be an interesting direction for future work.
SSL Techniques. We make note of some important details relevant to specific SSL techniques:
- For the two self-training methods, we use a training schedule similar to that of COSINE, whereby we divide training into two stages: In the first stage, the end model is trained on weak labels as is done in vanilla WS end model training. In the second stage, new pseudo-labels are generated upon reaching a pre-specified update period of \{50,100\} steps.
- For self-training, we also allow for applying thresholds to the pseudo-labels given to unlabeled data. When combining this approach with thresholding the initial weak labels, we again use the same threshold value.
- When using ST (DM as LF), we do not re-tune the label model-specific hyperparameters for Snorkel in subsequent self-training rounds. We default to instead using the same hyperparameters found from the initial search.
- For EM, VAT, and UDA, we sample batches with equal numbers of labeled and unlabeled examples when taking each step but tune a weighting parameter to balance their losses.
- When applying VAT to RoBERTa models, we calculate perturbations with respect to the RoBERTa-based features (which are continuous) instead of the raw text inputs.
- We try UDA on the text classification datasets where translations are straightforward to apply (i.e., do not require extra annotation effort). This includes Youtube, Trec, AGNews, and Yelp. For Yelp, where some examples are too long, we split examples at the sentence-level and backtranslated each sentence before concatenating following [29].<sup>5</sup> For the other text datasets, which are relation classification, we did not use UDA as it requires non-trivially annotating the entity spans in the backtranslations.
Re-labeling. For re-labeling, we again use a two-stage training schedule where the second stage contains a label update period (as explained when discussing self-training). When combining relabeling with thresholding, we consider the set of examples with candidate labels (i.e., which are possibly removed when performing dynamic thresholding) to be fixed after the initial round of thresholding (of LM outputs). Thus, re-labeling provides new pseudo-labels only for examples that survived the initial thresholding. This is so as to cleanly differentiate it from SSL.
Compute. We ran all MLP experiments on AWS, using up to four g4dn. 4xlarge EC2 instances at one time. Each instance allowed for running up to 10 different experiments (i.e., here considered as a hyperparameter sweep for any specific method from our design space) in parallel. For all RoBERTa training runs, we ran our experiments on a fleet of up to 15 NVIDIA-A40 GPUs hosted on the University of Washington Hyak cluster. Each experiment fits on a single GPU, which is large enough to use our batch sizes without needing gradient accumulation.
${}^{5}$  ImdB had a similar length issue, but unfortunately the version used in WRENCH had been stripped of punctuation, rendering sentence-splitting non-trivial.
B Additional Results for Baseline Comparisons
B.1 Additional Plots
Figure 5: Test performance for  $LM =$  Snorkel,  $DM = MLP$ . We compare a selected method from our design space (green) with four recent proposals from the literature. We plot all performances relative to that of vanilla training (see Table 8 for the absolute metrics).
Figure 6: Test performance for  $LM = MV$ ,  $DM = RoBERTa$ . We compare a selected method from our design space (green) with four recent proposals from the literature. We plot all performances relative to that of vanilla training (see Table 7 for the absolute metrics). For this setting, we did not run an exhaustive search over the whole design space so we do not report "Best of Design Space."
MLP Results. While fully fine-tuning RoBERTa end models provides better overall performance across text-based tasks, we also provide results for MLP end models for completeness. Here, the best single-method across datasets was (one-time) confidence thresholding + VAT + re-labeling. This method achieves the best point estimate on all 8 datasets compared to previous baselines.
Majority Voting Results. We also ablate the specific choice of label model to be Majority Voting for the RoBERTa experiments. Here, we mostly use the same method as for when the label model is Snorkel. However, we found that swapping in cut-based thresholding tended to perform better than sticking with confidence-based. This behavior can perhaps be explained by the relatively cruder confidence estimates for Majority Voting soft-labels; instead of learning a probabilistic model, the soft-labels for Majority Voting simply use the ratios of LF votes (e.g. for a binary task, if an example received 2 negative and 3 positive votes, the soft-label would be [0.4, 0.6]). Overall, we find that this method at least comes within error bars of previous baselines on all datasets though the point estimate is 8.4 p.p. lower on Spouse. Notably though, the best overall performance on Spouse is still obtained by applying our design space on top of the Snorkel label model (see Tables 6 and 7).
B.2 Tables with Absolute Performance
Method	IMDb	Yelp	Youtube	AGNews	TREC	Spouse	Chemprot
Vanilla	0.874	0.925	0.948	0.867	0.646	0.215	0.572
(0.002)	(0.017)	(0.009)	(0.008)	(0.011)	(0.012)	(0.002)
Cutstat + Snorkel	0.871	0.939	0.951	0.881	0.707	0.242	0.587
(0.006)	(0.006)	(0.002)	(0.001)	(0.015)	(0.023)	(0.005)
Conf + ST (DM as LF) + Re-label	0.888	0.954	0.947	0.882	0.711	0.531	0.581
(0.013)	(0.006)	(0.012)	(0.002)	(0.030)	(0.039)	(0.012)
Best of Design Space	0.907	0.965	0.952	0.885	0.765	0.531	0.601
(0.004)	(0.002)	(0.006)	(0.002)	(0.012)	(0.039)	(0.008)
COSINE (RoBERTa) + Snorkel	0.884	0.938	0.952	0.875	0.675	0.269	0.573
(0.004)	(0.009)	(0.003)	(0.004)	(0.002)	(0.106)	(0.005)
Denoise (RoBERTa) + Snorkel	0.867	0.922	0.949	0.867	0.633	0.287	0.544
(0.007)	(0.018)	(0.006)	(0.008)	(0.014)	(0.202)	(0.005)
Weasel (RoBERTa)	0.866	0.928	0.958	0.871	0.629	0.293	0.586
(0.009)	(0.017)	(0.008)	(0.001)	(0.044)	(0.117)	(0.007)
Table 6: Test performance for LM = Snorkel, DM = RoBERTa. Blue refers to the best performance of a single method (i.e., excluding "Best of Design Space", which reports the best method per dataset) while bold refers to being within standard deviation error bars of the best method. Red indicates where "Best of Design Space" outperforms the best single method outside of error bars.
Method	IMDb	Yelp	Youtube	AGNews	TREC	Spouse	Chemprot
Vanilla	0.859(0.004)	0.904(0.017)	0.960(0.014)	0.869(0.004)	0.669(0.012)	0.248(0.030)	0.574(0.003)
Cutstat + MV	0.861(0.013)	0.953(0.005)	0.968(0.006)	0.881(0.005)	0.731(0.027)	0.279(0.013)	0.592(0.010)
Cutstat + ST (DM as LF) + Re-label	0.882(0.007)	0.954(0.004)	0.969(0.004)	0.882(0.004)	0.743(0.049)	0.347(0.052)	0.606(0.004)
COSINE (RoBERTa) + MV	0.884(0.004)	0.945(0.004)	0.959(0.006)	0.876(0.002)	0.716(0.067)	0.431(0.034)	0.589(0.010)
Denoise + MV	0.866(0.013)	0.905(0.029)	0.956(0.010)	0.873(0.006)	0.678(0.025)	0.172(0.205)	0.576(0.010)
Weasel (RoBERTa)	0.866(0.009)	0.928(0.017)	0.958(0.008)	0.871(0.001)	0.629(0.044)	0.293(0.117)	0.586(0.007)
Table 7: Test performance for LM = Majority Voting, DM = RoBERTa. Blue refers to the best performance of a single method while bold refers to being within standard deviation error bars of the best method.