SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Method	IMDb	Yelp	Youtube	AGNews	TREC	Spouse	Chemprot	Census
Vanilla	0.828(0.001)	0.917(0.001)	0.925(0.004)	0.854(0.002)	0.623(0.004)	0.228(0.013)	0.544(0.002)	0.511(0.012)
Cutstat + Snorkel	0.827(0.002)	0.913(0.001)	0.919(0.005)	0.865(0.001)	0.662(0.017)	0.228(0.003)	0.553(0.004)	0.587(0.001)
Conf + VAT + Re-label	0.850(0.002)	0.924(0.004)	0.929(0.005)	0.870(0.000)	0.678(0.023)	0.330(0.005)	0.558(0.005)	0.593(0.005)
Best of Design Space	0.850(0.002)	0.926(0.004)	0.929(0.005)	0.880(0.000)	0.677(0.016)	0.394(0.006)	0.561(0.005)	0.605(0.006)
COSINE (MLP) + Snorkel	0.833(0.001)	0.915(0.002)	0.921(0.002)	0.853(0.002)	0.641(0.012)	0.248(0.028)	0.536(0.004)	0.520(0.027)
Denoise (MLP) + Snorkel	0.806(0.011)	0.895(0.006)	0.528(0.000)	0.856(0.001)	0.568(0.003)	0.169(0.207)	0.535(0.004)	0.569(0.007)
Weasel (MLP)	0.826(0.003)	0.905(0.005)	0.921(0.016)	0.856(0.005)	0.446(0.117)	0.261(0.017)	0.551(0.007)	0.552(0.051)
Table 8: Test performance for LM = Snorkel, DM = MLP. Color-coding is done in the same way that is described in Table 6.
C Additional Ablation Results
C.1 Axis Ablation for MLP
Method	IMDb	Yelp	Youtube	AGNews	TREC	Spouse	Chemprot	Census	Mean
Vanilla	0.828(0.001)	0.917(0.001)	0.925(0.004)	0.854(0.002)	0.623(0.004)	0.228(0.013)	0.544(0.002)	0.511(0.012)	0.679(0.002)
Thresh Alone	0.833(0.002)	0.924(0.002)	0.915(0.002)	0.865(0.001)	0.678(0.027)	0.228(0.003)	0.547(0.004)	0.587(0.001)	0.697(0.003)
SSL Alone	0.841(0.004)	0.918(0.002)	0.932(0.000)	0.877(0.001)	0.605(0.008)	0.375(0.029)	0.533(0.003)	0.518(0.015)	0.700(0.004)
Re-label Alone	0.833(0.001)	0.924(0.002)	0.917(0.014)	0.862(0.001)	0.631(0.024)	0.243(0.003)	0.549(0.007)	0.526(0.027)	0.686(0.005)
Thresh + SSL	0.838(0.004)	0.926(0.004)	0.929(0.002)	0.878(0.001)	0.691(0.012)	0.394(0.006)	0.560(0.001)	0.605(0.006)	0.728(0.002)
Thresh + Re-label	0.837(0.002)	0.925(0.003)	0.917(0.007)	0.870(0.001)	0.677(0.016)	0.270(0.007)	0.558(0.002)	0.602(0.002)	0.707(0.002)
SSL + Re-label	0.835(0.001)	0.925(0.006)	0.935(0.005)	0.875(0.002)	0.662(0.011)	0.347(0.043)	0.557(0.004)	0.542(0.025)	0.710(0.006)
Thresh + SSL + Re-label	0.850(0.002)	0.930(0.001)	0.929(0.005)	0.880(0.000)	0.671(0.034)	0.347(0.043)	0.561(0.005)	0.606(0.001)	0.722(0.007)
Entire Design Space	0.850(0.002)	0.926(0.004)	0.929(0.005)	0.880(0.000)	0.677(0.016)	0.394(0.006)	0.561(0.005)	0.605(0.006)	0.728(0.002)
Table 9: Axis ablation for  $LM =$  Snorkel,  $DM = MLP$ . Each entry corresponds to picking a specific method within our design space using validation tuning. Blue numbers are the highest for a given dataset, while bold numbers are those within error bars of the best result.
C.2 Axis Instantiations
C.2.1 RoBERTa
Note that we exclude Youtube from these tables because the Vanilla baseline is already within error bars of the best performing methods in Table 6. As such, this dataset would not provide much useful signal for assessing which version of a particular axis is best.
IMDb	Yelp	AGNews	TREC	Spouse	Chemprot
No SSL	0.905	0.961	0.883	0.748	0.332	0.600
(0.011)	(0.002)	(0.005)	(0.023)	(0.105)	(0.006)
EM	0.895	0.958	0.883	0.700	0.286	0.586
(0.005)	(0.003)	(0.002)	(0.018)	(0.153)	(0.007)
ST	0.907	0.961	0.887	0.765	0.392	0.601
(0.004)	(0.003)	(0.005)	(0.012)	(0.098)	(0.005)
ST (DM as LF)	0.889	0.958	0.885	0.725	0.531	0.601
(0.004)	(0.003)	(0.003)	(0.018)	(0.039)	(0.008)
VAT	0.900	0.958	0.883	0.743	0.437	0.605
(0.004)	(0.001)	(0.002)	(0.029)	(0.004)	(0.002)
UDA	-	0.965	0.885	0.753	-	-
(0.002)	(0.003)	(0.021)
Table 10: Closer look at SSL for DM = RoBERTa. We report the best performance of any method that incorporates a given SSL technique.
IMDb	Yelp	AGNews	TREC	Spouse	Chemprot
No Re-labeling	0.885	0.959	0.887	0.732	0.529	0.597
(0.018)	(0.002)	(0.005)	(0.014)	(0.057)	(0.009)
Re-labeling	0.907	0.965	0.885	0.765	0.531	0.605
(0.004)	(0.002)	(0.003)	(0.012)	(0.039)	(0.002)
Table 11: Closer look at re-labeling for DM = RoBERTa. We report the best performance of any method that either uses or does not use re-labeling.
C.2.2 MLPs
Method	IMDb	Yelp	Youtube	AGNews	TREC	Spouse	Chemprot	Census
Coverage-based	0.841	0.925	0.935	0.877	0.662	0.375	0.557	0.542
(0.004)	(0.006)	(0.005)	(0.001)	(0.011)	(0.029)	(0.004)	(0.025)
Cut-based	0.832	0.930	0.935	0.880	0.684	0.394	0.566	0.606
(0.002)	(0.001)	(0.002)	(0.000)	(0.016)	(0.015)	(0.003)	(0.001)
Confidence-based	0.850	0.928	0.929	0.878	0.691	0.417	0.560	0.592
(0.002)	(0.002)	(0.005)	(0.002)	(0.012)	(0.009)	(0.011)	(0.008)
Table 12: Closer look at thresholding for  $DM = MLP$ . We report the best performance of any method that incorporates a given thresholding method.
Method	IMDb	Yelp	Youtube	AGNews	TREC	Spouse	Chemprot	Census
No SSL	0.837	0.927	0.935	0.870	0.684	0.270	0.560	0.602
(0.002)	(0.001)	(0.002)	(0.001)	(0.016)	(0.007)	(0.011)	(0.002)
VAT	0.850	0.926	0.935	0.880	0.691	0.355	0.563	0.605
(0.002)	(0.002)	(0.005)	(0.000)	(0.012)	(0.015)	(0.007)	(0.006)
EM	0.835	0.922	0.925	0.869	0.664	0.268	0.560	0.595
(0.001)	(0.001)	(0.002)	(0.002)	(0.007)	(0.032)	(0.009)	(0.004)
ST	0.838	0.928	0.929	0.866	0.674	0.379	0.566	0.592
(0.001)	(0.002)	(0.005)	(0.001)	(0.018)	(0.008)	(0.003)	(0.002)
ST (DM as LF)	0.834	0.930	0.935	0.869	0.668	0.417	0.561	0.606
(0.001)	(0.001)	(0.002)	(0.000)	(0.038)	(0.009)	(0.005)	(0.001)
Table 13: Closer look at SSL for  $DM = MLP$ . We report the best performance of any method that incorporates a given SSL technique.
Method	IMDb	Yelp	Youtube	AGNews	TREC	Spouse	Chemprot	Census
No Re-labeling	0.841	0.926	0.932	0.878	0.691	0.417	0.560	0.605
(0.004)	(0.004)	(0.000)	(0.001)	(0.012)	(0.009)	(0.001)	(0.006)
Re-labeling	0.850	0.930	0.935	0.880	0.677	0.347	0.566	0.606
(0.002)	(0.001)	(0.005)	(0.000)	(0.016)	(0.043)	(0.003)	(0.001)
Table 14: Closer look at SSL axis for DM = MLP. We report the best performance of any method that either uses or does not use re-labeling.
D Beyond standard WS benchmarks: datasets for coverage ablations
D.1 LF subsampling and procedural generation on existing WRENCH datasets
Overall, WS benchmarks lack agreed-upon ways to explore different LF coverage/precision trade-offs. In this work, we choose to do so by sub-sampling LFs from an overall base set. However, the default LF sets associated with WS benchmarks are not equally as suitable to subsample from. In particular, we are wary about subsampling default LF sets when they:
(1) Contain very few LFs, which reduces the possible granularities of subsampling
(2) Contain LFs which are heterogeneous or have untracked lineages (e.g.,  $\lambda_{1}$  closely relates to or was only provided in response to another LF  $\lambda_{2}$ ).
For datasets where we subsample directly, we choose ones with both larger LF counts and where all LFs are of the same form. This includes:
- Semeval: 164 string-matching rules selected by humans from an automated candidate generation procedure [34]
- Chemprot: 26 individual keyword-based rules [30]
These contrast with LF sets that are far smaller, more heterogeneous, or have unclear dependencies. For these datasets, we instead procedurally generate LFs using the tools from WRENCH [33]
-  $IMDb$ : 4 aggregate keyword based rules (i.e., each LF checks for the presence of any of multiple keywords), 1 expression-based rule [26]
- Yelp: 7 heuristic rules on keywords, 1 third-party model on polarity of sentiment [26]
- Agnews: 9 aggregate keyword based rules split amongst four respective classes [26]
Subsampling. On some datasets, we directly subsample the LFs coming from WRENCH, selecting the most accurate LFs in a class-stratified fashion in order to roughly preserve similar ratios of LFs between classes. We also take care not to completely remove all the LFs for a given class, setting the minimum count to 1 per class. We explore subsampling per-class ratios within [0.1, 0.9], avoiding LF sets that are near-duplicates of each other (i.e., having coverage levels within  $1\%$  of each other). We subsample based on accuracy to (optimistically) simulate an LF writer who is both careful and has considerable domain expertise; we assume that if they were to end up at a lower coverage LF set (either by writing fewer LFs or by pruning LFs written during a "brainstorming" phase), they would prioritize rules that they are most confident about. Assuming the LF writer has sufficient domain knowledge, these LFs are also the ones more likely to be accurate on the examples they fire on.
Procedural Generation. For the datasets where subsampling is less appropriate, we use WRENCH's LF generator to construct LF sets with  $\{2, 5, 10, 15, 20\}$  n-gram based LFs per class, choosing the most accurate LFs from the candidate pool that have at least  $2\%$  coverage over the training set.
D.2 New WS benchmarks: Massive18 and Banking77
We also create two new WS benchmarks by writing LFs for the publicly available intent classification datasets MASSIVE [8] and Banking77 [5]. These tasks were chosen a high-level to capture some practical challenges that are not as well-represented by current WRENCH tasks; most notably, they contain significantly higher class counts, which makes them more challenging to write LFs for.
Roughly speaking, assuming one writes uni-polar LFs (i.e., each LF either votes for a single class or abstains), the form of the vast majority of LFs in WRENCH, the number of LFs needed to reach a certain coverage level will likely need to scale with the number of classes. Further, because tasks with larger label spaces require writing more LFs, we also view these tasks as being especially relevant for studying lower coverage LF sets. We provide more details about the new tasks as follows:
Massive18 is derived from the MASSIVE dataset, a collection of single-shot interactions between human users and general intelligent voice assistants across 52 languages. We exclusively use the
This, of course, also assumes that the coverage of LFs within each class does not change dramatically
English-US portion of this dataset and treat the broad scenarios (i.e., general domains) as labels instead of the finer-grained intents. For instance, "alarm" is a scenario whereas "alarm_set" and "alarm_remove" are two intents within that scenario. We call the resulting task Massive18 since there are 18 different scenario classes. Notably, even with this simplification, Massive18 contains at least nearly double the class count of all previous WRENCH classification tasks (see Table 15 below).
Banking77 is a collection of online banking queries along with the corresponding user intents. We use this task as defined with the full set of 77 intents, which can be both quite domain specific and especially fine-grained. If we were to write the minimum number of LFs for Banking77, i.e. one for each class, this would already require 77 LFs, more than the number of LFs for most previous WRENCH datasets.
Datasets	|y|	|Train|	|Test|	#LFs	Coverage	Snork. Precision	Metric
IMDb	2	20000	2500	5	87.6%	74.4%	Acc.
Yelp	2	30400	3800	8	82.8%	75.4%	Acc.
Youtube	2	1586	250	10	87.7%	87.0%	Acc.
AgNews	4	96000	12000	9	69.1%	82.5%	Acc.
Trec	6	4965	500	68	95.1%	60.0%	Acc.
Massive18 (ours)	18	11514	2974	59	61.3%	83.8%	Acc.
Banking77 (ours)	77	9003	3080	218	54.5%	76.5%	Acc.
Spouse	2	22254	2701	9	25.8%	65.6% (on Val)	F1 (binary)
Chemprot	10	12861	1607	26	85.6%	58.0%	Acc.
Census	2	10083	16281	83	99.1%	58.0%	F1 (binary)
Labeling Functions. For both datasets, we manually write a set of LFs ourselves based upon a randomly sampled development set of 250 cleanly-labeled examples. Each LF checks whether a specific keyword (or set of multiple keywords) is a substring of the given example and then votes for a specific class if the substring(s) are present (and otherwise abstaining). We provide some examples of the LFs we wrote in Tables 16 and 17, while we share the full LF sets in our supplied codebase. Importantly, while we believe these LF sets are reasonable, they should not be treated as the be-all-end-all and encourage others to iterate or develop their own for these tasks in the future.
Finally, in our experiments, we also try the same subsampling procedure from Appendix D.1 to further explore different coverage levels. For Massive18 we use the ratios  $\{0.1, 0.5, 0.7, 0.8, 1\}$  and for Banking77 we use the ratios  $\{0.1, 0.5, 0.7, 1.0\}$ .
Table 15: Comparing our two new datasets (blue) to previous benchmarks. Coverage is the percentage of training examples on which at least one LF does not abstain. "Snork. precision" refers to the accuracy of the Snorkel LM on the covered set of inputs.
Label	Keyword LFs
“alarm”	[“alarm”, “wake+up”]
“takeaway”	[“takeaway”, “delivery”, “order”]
“social”	[“tweet”, “twitter”, “facebook”, “complain”]
“music”	[“what+song”, “save+song”, “shuffle”]
“calendar”	[“calendar”, “schedule”, “remind”]
Table 16: Example LFs for Massive18. Each row shows one of the possible labels and the associated keyword-based LFs. Note that some LFs contain multiple keywords/substrings that are joined by the "+" character. This signifies that the LF checks whether all the keywords supplied are in a given example (though they do not necessarily have to appear in the provided order).
Label	Keyword LFs
“age_limit”	[“age limit”, “child”, “my son”, “my daughter”]
“pinBlocked”	[“takeaway”, “delivery”, “order”]
“lost_or_stolen_card”	[“tweet”, “twitter”, “facebook”, “complain”]
“verify_my_identity”	[“id+check”, “what+ id”]
“disposable_cardlimits”	[“disposable+limit”, “disposable+max”]
Table 17: Example LFs for Banking77. Note the usage of “+”, as explained in the caption of Table 16.
D.3 Tabular Datasets
Labeling Functions. We use three tabular tasks and the LF generation procedure from [32]. At a high-level, this procedure uses the sklearn implementation of random forests (i.e., RandomForestClassifier) to train multiple decision trees on a small cleanly-labeled development set (i.e., formed by uniformly sampling  $5\%$  of the training set in our experiments). Uni-polar LFs are then derived from individual trees. In our experiments, we generate a set of 100 candidate LFs for each task and then select the top- $\{5\%, 10\%, 15\%, 20\}$  most accurate LFs (per-class) based upon the full training set (i.e., similar in spirit to WRENCH's procedural LF generator). Again, while this selection step isn't possible without access to all the labels, we consider it as a loose (optimistic) approximation of integrating a domain expert's knowledge and judgment.
Robustness to seeding. One caveat of this approach for genearting LFs is that the behavior may vary depending on the initial random seed given to RandomForestClassifier. To try to account for this, we try multiple seeds and find that the overall conclusions do not change across different runs of the LF generator. As seen in Figure 7, while the absolute performance levels may vary across seeds, SSL is consistently more useful at lower coverage levels.
Figure 7: Tabular experiments across different random seeds
E Coverage bias explains the value of unlabeled data
Figure 8: Limited Size versus Coverage Bias. We may assess the relative impacts of the limited size of WS training sets (comparing  $GT$  (Full) and  $GT$  (Sub)) versus their coverage bias (comparing  $GT$  (Sub) and  $GT$  (Cov)).
In Section 4.5, we observed that on some datasets, as coverage decreases, so does the performance of  $GT(Cov)$ . Indeed, for any LF set, we can view the gap in performance between  $GT(Cov)$  and  $GT(Full)$  as a reflection of the aggregate impact of two data gaps that WS datasets suffer from: (1) limited size (i.e., not all training examples are labeled) and (2) coverage bias (i.e., the examples covered by LFs come from a biased subpopulation of the test distribution). Because  $GT(Cov)$  simply ignores all unlabeled examples, this gap also then reflects the value of said examples.
A natural follow up question is whether limited size or coverage bias more greatly hinders learning under WS. Here, we further decompose the performance drops we observed between  $GT(Cov)$  and  $GT(\text{Full})$  by considering a third model that interpolates between the two. Specifically, we consider  $GT(\text{Sub})$ , a model that suffers only from limited size but not coverage bias. This model is trained on a training set that shares the same size as the dataset for  $GT(Cov)$ , but consists of examples re-sampled uniformly from the full training set (i.e., eliminating coverage bias).
Once we have trained all three models, we may then simply compare the following performance gaps:
- TestPerf(GT(Full)) - TestPerf(GT(Sub)): to ablate the impact limited size
- TestPerf(GT(Sub)) - TestPerf(GT(Cov)): to ablate the impact of coverage bias
As we see in Figure 8, when large gaps exist between  $GT$  (Full) and  $GT$  (Cov), most of this gap is explained by the performance drop between  $GT$  (Sub) and  $GT$  (Cov) instead of the drop between  $GT$  (Full) and  $GT$  (Sub) (e.g., especially on Semeval, Massive18, Banking77). This suggests that coverage bias is far more responsible than limited size, which perhaps makes sense in the context of WS, as LFs can label an arbitrary amount of data but only from the covered set. Also, this means that SSL techniques, initially developed in settings where labeled and unlabeled sets are i.i.d. (i.e., where limited size is the only relevant data gap), actually remain effective in low coverage WS settings despite the significant impacts of coverage bias.