Training Subset Selection for Weak Supervision
Authors: Hunter Lang
Aravindan Vijayaraghavan
David Sontag
Venue: NeurIPS 2022 Accept (2022)
Keywords: weak supervision
data editing
subset selection
weakly-supervised learning

Abstract: Existing weak supervision approaches use all the data covered by weak signals to train a classifier.  We show both theoretically and empirically that this is not always optimal.  Intuitively, there is a tradeoff between the amount of weakly-labeled data and the precision of the weak labels. We explore this tradeoff by combining pretrained data representations with the cut statistic to select (hopefully) high-quality subsets of the weakly-labeled training data. Subset selection applies to any label model and classifier and is very simple to plug in to existing weak supervision pipelines, requiring just a few lines of code. We show our subset selection method improves the performance of weak supervision for a wide range of label models, classifiers, and datasets.  Using less weakly-labeled data improves the accuracy of weak supervision pipelines by up to 19% (absolute) on benchmark tasks.

Training Subset Selection for Weak Supervision
Hunter Lang MIT CSAIL hjl@mite 1.mmt.eeu
Aravindan Vijayaraghavan Northwestern University aravindt@@notthwsster.. edu
David Sontag MIT CSAIL dsontagomit e.eeai
Abstract
Existing weak supervision approaches use all the data covered by weak signals train classifier. We show both theoretically and empirically that this is not always optimal. Intuitively, there is tradeoff between the amount of weakly- labeled data and the precision of the weak labels. We explore this tradeoff by combining pretrained data representations with the cut statistic [23] select (hopefully) high- quality subsets of the weakly-labeled training data. Subset selection applies to any label model and classifier and is very simple to plug in to existing weak supervision pipelines, requiring just few lines of code. We show our subset selection method improves the performance weak supervision for wide range label models, classifiers, and datasets. Using less weakly-labeled data improves the accuracy of weak supervision pipelines by up to 19% (absolute) on benchmark tasks.
Introduction
Due to the difficulty of hand-labeling large amounts training data, an increasing share of models are trained with weak supervision [30, 28). Weak supervision uses expert-defined "labeling functions" to programatically label a large amount of training data with minimal human effort. This pseudo-labeled training data is used train classifier (e.g., deep neural network) if it were hand-labeled data.
Labeling functions are often simple, coarse rules, so the pseudolabels derived from them are not always correct. There is an intuitive tradeoff between the coverage of the pseudolabels (how much pseudolabeled data do we use for training?) and the precision on the covered set (how accurate are the pseudolabels that do use?). Using all the pseudolabeled training data ensures the best possible generalization the population pseudolabeling function Y(X). On the other hand, if we can select high-quality subset of the pseudolabeled data, then our training labels (XX) are closer the true label Y. but the smaller training set may hurt generalization. However, existing weak supervision approaches such Snorkel [28]. MeTaL [29]. FlyingSquid [10], and Adversarial Label Learning [3] use all of the pseudolabeled data to train the classifier, and do not explore this tradeoff.
We present numerical experiments demonstrating that the status quo of using all the pseudolabeled data is nearly always suboptimal. Combining good pretrained representations with the cut statistic [23] for subset selection, we obtain subsets of the weakly-labeled training data where the weak labels are very accurate. By choosing examples with the same pseudolabel as many of their nearest neighbors in the representation, the statistic uses the representation's 's geometry identify these accurate subsets without using any ground-trutte truth labels. Using the smaller but sugher--uality training sets selected by the cut statistic improves the accuracy of weak supervision pipelines by up to 19% accuracy (absolute). Subset selection applies to any "label model" (Snorkel, FlyingSquid, majority vote, etc.) and any classifier, since is modular, intermediate step between creation of the pseudolabeled training set and training. We conclude with theoretical analysis of a special case of weak supervision where the precision/coverage tradeoff can be made precise.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

2 Background
The three components of weak supervision pipeline are the labeling functions, the label model, and the end model. The labeling functions are maps A : X' - > o {2], where Ø represents abstention. For example, for sentiment analysis, simple token-basee labeling functions are effective, such as: "bad" 2
(a) [ "good" E I
A2(x)
otherwise
otherwise
If the word "good" is in the input text x, labeling function A₁ outputs 1; likewise when "bad" e æ, A2 outputs -. Of course, an input text could contain both "good" and "bad", so A and A2 may conflict. Resolving these conflicts is the role of the label model.
Formally, the label model is a map y : ( - y That is, if we let ((x) refer to the vector (A (x(,,.,((((())) then (((())) is a single pseudolabel (or "weak label") derived from the vector of K labeling function outputs. This resolves conflicts between the labeling functions. Note that we can also consider Y as deterministic function of X. The simplest label model majority vote, which outputs the most common label from the set of non-abstaining labeling functions:
YMV (x) = mode(((Ax()) (x( * Ø))
all the labeling functions abstain (i.e., (x) = 2 for all k), then YMV(2) - 2. More sophisticated label models such as Snorkel [30] and FlyingSquid [10] parameterize Y to learn better aggregation rules, e.g. by accounting for the accuracy of different A₂'s 's or accounting for correlations between pairs (Aj, Ag). These parameters are learned using unlabeled data only: the methods for doing so have rich history dating back at least to Dawid and Skene [8]. Many label models (including Snorkel and its derivatives) output a "soft" pseudolabel, i.e., a distribution P ....., set the hard pseudolabel as ((X) = argmax, y P|y = y (
(X))).,,AA()))) Ax()), and Ax())]
Given an unlabeled sample [- the label model produces pseudolabeled training set 'T {(x,, (xx)): (a,) Ø). The final step in the weak supervision pipeline is to use like regular training data to train an end model (such as deep neural network), minimizing the zero-one loss:
I7
= argmin JEF IT
Σ Uf(x;)) # (z:))
(1)
or a convex surrogate like cross-entropy. For many applications, we fine-tune a pretrained repre- sentation instead of training from scratch. For example, on text data, we can fine-tune a pretrained BERT model. We refer to the pretrained representation used by the end model as the end model representation, where applicable.
Notably, all existing methods use the full pseudolabeled training set T to train the end model. T consists of all points where Ý # Ø. In this work, we experiment with methods for choosing qugler-quality subsets T' T and use T' in (1) instead of T.
Related Work. The idea of selecting a subset of high-quality training data for use in fully- supervised or semi-supervised learning algorithms has a long history. It is also referred to as data pruming [1], and significant amount of work has focused on removing mislabeled examples to improve the training process [e.g., 26, 18, 7, 25]. These works do not consider the case where the pseudolabels come from deterministic labeling functions, and most try estimate parameters of specific noise process that is assumed to generate the pseudolabels. Many of these approaches require iterative learning or changes to the loss function, whereas typical weak supervision pipelines do one learning step and little or no loss correction. Maheshwari et al. [21] study active subset selection for weak supervision, obtaining a small number of human labels to boost performance.
In self-training [c.g.. 34], an initial labeled training set iteratively supplemented with the pseudola- beled examples where a trained model is most confident (according to the model's probability scores). The model is retrained on the new training set in each step. Yarowsky [40] used this approach starting from a weakfy-labele self-training with an initial -eakly-labeled training set, and both have deep-model-bert for selecting confident data in each round. We view these weakly-supervised self eeff-training methods as orthogonal to our approach, since their main focus is on making better use of the data that not covered by weak rules, not selecting good pseudolabeled subsets. Indeed, we show in Appendix
training set; Yu et al. [41], Karamanolakis et al. [15] also combine
procedures

---

G
Figure 1: Cut statistic procedure. A representation e is used to compute the nearest-neighhor graph G. Nodes that have the same pseudolabel as most of their neighbors are chosen for the subset T'.
B.7 that combining our method with these approaches improves their performance. Other selection schemes, not based model confidence, have also been investigated for self-training [e.g.. 44, 24).
Muhlenbach al. [23] introduced the cut statistic as a heuristic for identifying mislabeled examples in training dataset. Li and Zhou [19] applied the cut statistic to self-training, using it to select high- quality pseudolabeled training data for each round. Zhang and Zhou [43] applied the cut statistic to co-training and used samiing-itt--oise results from Angluin and Laird [2] to optimize the amount of selected data each round. Lang et al. [16] also used co-training and the cut statistic to co-train large language models such GPT-3 [5] and TO [31] with smaller models such as BERT [9] and RoBERTa [20]. These previous works showed that the cut statistic performs well in iterative algorithms such as self-training and co- co-training; we show that it works well in one-step weak supervision settings, and that it performs especially well when combined with modern pre-trained representations. Our empirical study shows that this combination is very effective at selecting good pseudolabeled training data across a wide variety of label models, end models, and datasets.
As detailed in Section 3, the performance of the cut statistic relies on good representation of the input examples x; to find good subsets. Zhu al. [45] also used representations to identify subsets of mislabeled labels and found that methods based on representations outperform methods based on model predictions alone. They use different ranking method and do not evaluate in weakly- supervised settings. Chen et al. [6] also use pretrained representations to improve the performance of weak supervision. They created a new representation-avve aware label model that uses nearest neirest-neighhors in the representation to label more data and also learns iner-graine, label model parameters. In contrast, our approach applies to any label model, can be implemented in a few lines of code, and does not require representations from very large models like GPT-3 or CLIP. Combining the two approaches is an interesting direction for future work.
3 Subset Selection Methods for Weak Supervision
In this work, study techniques for selecting high-quality subsets of the pseudolabeled training set T. We consider two simple approaches to subset selection in this work: entropy scoring and the cut statistic. In both cases, we construct subset T' by first ranking all the examples in T,, then selecting the top 33 fraction according to the ranking. In our applications, 0 < 3 & 1 is a hyperparameter tuned using a validation set. Hence, instead of [T] covered examples for training the end model in (1), we use | examples. Instead of a single, global ranking, subset selection can easily be stratified to use multiple rankings. For example, if the true label balance F[Y] is known, we can use separate rankings for each set T, = (a; : Yxx) = y) and select the top 3P = y] points from each Ty- This matches the pseudolabel distribution
T' to the true marginal IP[Y]. For simplicity, we use a global ranking in this work, and our subset selection does not use [P[]] or any other information about the true labels. Below give details for the entropy and cut statistic rankings.
Entropy score. Entropy scoring only applies to label models that output a "soft" pseudo-label ([Y]A(X)] For this selection method, we rank examples by the Shannon entropy of the soft label, H (P[{{AAzz;]] and set T' to the T] examples with the lowest entropy. Intuitively, the label model is the "most confident" the examples with the lowest entropy. If the label model is well-calibrated, the weak labels should be more accurate on these examples.
Cut statistic [23]. Unlike the entropy score, which only relies on the soft label distribution P[Y]A], the cut statistic relies on good representation of the input examples 24. Let be representation for
3

---

Accuracy vs Coverage, Yelp
Majority Vole
Snurkel (DP)
Dawid-Slene
Sqying
McTal.
Accurcy
09
0.9
09
method
88
esenopy
08
0.8
0.8
CUBILEE
0.6
0.2
0.4
0.6 Coverage
0,3
1.0
0.2
0.4
066 Coverage
0.8
1.0
0.2
0.4 066
08
1.5
0.2
0.4
066
08
1.0
0.2
0.4 06 0.8 LG Covesage
Coverage
Coverage
Figure 2: Accuracy of the pseudolabeled training set versus the selection fraction / for five different label models. A pretrained BERT model is used as ó for the cut statistic. The accuracy of the weak training labels is better for 3 < L, indicating that sub-selection can select higher-quality training sets.
examples in X. For example, for text data, Φ could be the hidden state of the [CLS] token in the last layer of a pretrained large language model.
Recall that T {(mr, (1:)) (a,,) * Ø). To compute the statistic using d. we first form a graph G = (V, E) with one vertex for each covered zi and edges connecting vertices who are K-nearest neighbors in Φ. That is, for each example x; with ((x)) * 2, let
NN
-xz (z) are K ne-arest-neighhon in Φ).
Then we set V (i Y(,)) + Ø). E ((i.j) 2, E NNo())) I2 E NNo(2.)). For each node i, let (1) = (j: (i.5) E E} denote its neighbors in G. We assign a weight wiy to each edge so that nodes closer together & have a higher edge weight: Waj (1 + ||ó(,,) - ((xx))12)-1 We say an edge (i,,j) is cul if (xx) (zz)) and capture this with the indicator variable lij ==
[x:) * (xz)]. As suggested in Figure I, if ø is good representation, nodes with few incident cut edges should have high-quality pseudolabels-
examples have the same label as most of their neighbors. On the other hand, nodes with large number of cut edges likely correspond to mislabeled examples. The cut statistic heuristically quantifies this idea to produce a ranking.
Suppose (as a null hypothesis) that the labels Y were sampled i.i.d. from the marginal distribution PIý- = w]. Large deviations from the null should represent the most noise-free vertices. For each vertex i, consider the test statistic: J: L))N()) wijlij. The mean of J, under the null hypothesis is:
H4; =1 PPY E))r(s) Wij, and the variance is: of Pf (z; (z;) ](1 (I;))) EjEN()) wiz- Then for each i we can compute the Z-score Z; = Jーي and rank examples by Zg. Lower is better, ot
since nodes with the smallest Z, have the least noisy Y assignments in Φ. As with entropy scoring, we set 7' to be the the 3|7") points with the smallest values of Z, We provide code for simple (< 30 lines) function to compute the Z; values given the representations (ω(x;;) : 22, in Appendix C. Calling this function makes it very straightforward to incorporate the statistic in existing weak
supervision pipelines. Since the cut statistic does not require soft pseudolabels, it can also be used for
label models that only produce hard labels, and for label models such Majority Vote, where the soft label tends to be badly miscalibrated.
3.1 Cut Statistic Selects Better Subsets
To explore the two scoring methods, we visualize how T' changes with 3 for entropy scoring and the cut statistic. We used label models such as majority vote and Snorkel [30] to obtain soft labels P|Y A (2:)). and set (xx,) ) to be the argmax of the soft label. We test using the Yelp dataset from the WRENCH weak supervision benchmark [42]. The task is sentiment analysis, and the eight labeling functions ( ,A,,...,AA As) consist of seven keyword-based rules and one third-party sentiment polarity model. For & in the cut statistic, we used the [CLS] token representation of a pretrained BERT model. Section 4 contains more details on the datasets and the cut statistic setup.
For each 3 (0.1, 1.0). Figure 2 plots the accuracy of the pseudolabels on the training subset T'(.). This shows how training subset quality varies with the selection fraction B. We can compute this accuracy because most of the WRENCH benchmark datasets also come with ground-truth labels Y (even on the training set) for evaluation. Appendix B contains the same plot for several other WRENCH datasets and figures showing the histograms of the entropy scores and the Z, values.
Figure shows that combining the eut statistic with BERT representation selects better subsets than the entropy score for all five label models tested, especially for majority vote, where the entropy

---

scoring is badly miscalibrated. For a well-calibrated score, the subset accuracy should decrease as 5 increases. These results suggest that the cut statistic is able use the geometric information encoded in & to select more accurate subset of the weakly-labeled training data. However, it does not indicate whether that better subset actually leads to a more accurate end model. Since we could also use @ for the end model-e.g. by fine-tuning the full neural network or training a linear model on top of ø-it's possible that the training step (1) will already perform the same corrections as the cut statistic, and the end model trained on the selected subset will perform no differently from the end model trained with 3 = 1.0. In the following section, we focus on the cut statistic and conduct large-scale empirical evaluation on the WRENCH benchmark to measure whether subset selection improves end model performance. Our empirical results suggest that subset selection and the end model training step are complementary: even when we use powerful representations for the end model, subset selection further improves performance, sometimes by a large margin.
4 Experiments
Having established that the cut statistic can effectively select weakly-labeled training subsets that are higher-quality than the original training set, we now turn to a wider empirical study to see whether this approach actually improves the performance of end models in practice.
Datasets and Models. We evaluate our approach on the WRENCH benchmark [42] for weak supervi-
sion. We compare the status-quo of full coverage (3 1.0) to 3 chosen from {0.1, 0.2, .... 1.0). We evaluate our approach with five different label models: Majority Vote (MV), the original Snorkel/Data Programming (DP). [30], Dawid-Skene (DS) [8], FlyingSquid (FS) [10], and McTaL [29]. Follow- ing Zhang et al. [42], we use pretrained roberta- base and bert- base--aeecssed as the end model representation for text data, and hand-specified representations for tabular data. We performed all model training on NVIDIA A100 GPUs. We primarily evaluate on seven textual datasets from the WRENCH benchmark: IMDb (sentiment analysis), Yelp (sentiment analysis), Youtube (spam classification), TREC (question classification), SemEval (relation extraction). ChemProt (relation extraction), and AGNews (text classification). Full details for the datasets and the weak label sources are available in [42] Table 5 and reproduced here in Appendix B.I. We explore other several other datasets and data modalities in Sections 4 4.2-4.3.
Cut statistic. For the representation Φ for text datasets we used the [CLS] token representation of a large pretrained model such as BERT or RoBERTa. For relation extraction tasks, we followed [42] and used the concatenation of the [CLS] token and the average contextual representation of the tokens in each entity span. In Section 4.3, for the tabular Census dataset
the raw data features as the end model, used
for Φ. Unless otherwise specified, we used the same representation for é and for the initial end model. For example, when training pert-base-casse for the cut statistic. We explore several alternatives to this choice in Section 4.2.
sert-base-casee
Hyperparameter tuning. Our subset selection approach introduces a new hyperparameter, 3 the fraction of covered data to retain for training the classifier. To keep the hyperparameter tuning burden low, we first tune all other hyperparameters identically to Zhang et al. [42] holding 3 fixed at 1.0. We then use the optimal hyperparameters (learning rate, batch size, weight decay, etc.) from 3 1.0 for a grid search over values of | E {0.1, 0.2,.. ..... 1.0). choosing the value with the best (ground-truth) val- idation performance. Better results could be achieved by tuning all the hyperparameters together, but this approach limits the number of possible combinations, and it matches the setting where an existing, tuned weak supervision pipeline (with B 1.0) is adapted to use subset selection. In all of our exper- iments, used K 20 nearest neighbors to compute the cut statistic and performed no tuning on this value. Appendix B contains an ablation showing that performance is not sensitive to this choice.
4.1 WRENCH Benchmark Performance
Table compares the test performance of full coverage (8 = 1.0) to the performance of the cut statistic with B chosen according to validation performance. Standard deviations across five random initializations are shown in parentheses.
We refer to pretrained models by their names on the HuggingFace Datasets Hub, All model weights were downloaded from the hub: http::/huuginnffie..olaasstt
5

---

Table I: End model test accuracy (stddev) for weak supervision with 3 versus weak supervision with 3 selected from {0.1, 0.2, 0. 1.0] using a validation set ("+ cutstat"), shown for BERT (B) and RoBERTa (RB) end models. For these results, the cut statistic uses the same representation the end model for Φ. The cut statistic broadly improves the performance of weak supervision for many (label model, dataset, end model) combinations.
Label model
imdb
yelp
youtube
trec
semeval
chemprot
agnews
Majority Vote
78.32262
86.851.42
95, 25.22
66.761.46
85.17
57. 57.442.1
86.59
+ cutstal
81.861.36
89.499.78
95.600.72
71.843,00
92.470.19
$7.471.00
86.26p43
Data Programnming
75.90
76.43
92.481.30
71.20
71.97
51.89
86.01
+ cuistat
79.072.5
88.131.46
93.920.93
76.761.2
91.07as0
55.10
85.89
B
Dawid-Skent
78.861.44
88,451.42
88.45 1.42
51.041.71
72.40| .53
44.081.37
86.260.56
culstat
80.22 .69
89.04
90.72
57.282.91
89.07 1.62
49.07
86.93,.22
FlyingSquid
77.461,88
84.98
91.522,90
31.1222.99
31,830000
46.72 46.72a.96
86,10
+ cutstat
80,851.50
88.75
91.04
33.843.17
31083000
48.65p.9
85.90p.30
MeTaL
78.972.5
83,05
93.361.15
58.88
58.17
55.61
86.06g.22
cuistar
81.49
88.41
92.640.41
63.80
65.230.91
58.33
86.160.4
Majority Vote
86,99
88.51
95.84 95.84
67.602.38
85,83 22
57.061.12
87.460.53
+ cutstat
86.69u.75
95.19
96.00,.10
72.92
92.07 80
59.05p.56
88.01g.47
Data Programming
1.53
88.733.07
94.081.48
71.403,30
71.07
52.52p.69
86.75p.24
+ cutstat
86,461.82
93.950.93
93.04,,30
76.84..09
86.07
56.43
87.760.17
Dawid-Ske Skene
85.50
92.42
92.481,44
51.244.50
70.83
45.612.60
87.296.40
RB
+ cutstal
86.14
93.81g.69
93.84
58.482,75
81.671.33
52.93
88,35
Sqyin
85,25
92. 142.7
93,522.11
35.40
31.830.00
47.23
86,56
+ cutstat
87.71
94.50a.44
95.840.54
38.16
31.830.00
50.55
87.490.1
MeTaL
16,16
88.41
92.40
55,44
59.53
56. 36.74p.s
86.74
cuistat
87.460.65
94.03
93.84, 1,3
69.722.39
66.700.9
57.40p.98
88.40
The cut statistic improves the mean performance (across runs) compared 8 1.0 in 61/70 cases, sometimes by 10-20 accuracy points (c.g.. BERT SemEval DP). Since 1.0 included in the hyperparameter search over B, the only cases where the cut statistic performs worse than 1.0 are due to differences the performance on the validation and test sets. The mean accuracy gain from setting < 1.0 across all 70 trials is 3.65 points, indicating that the cut statistic is complementary to the end model training. no validation data is available to select 8, we found that 3 ==00 0.6 had the best median performance gain over all label
5 10 15
20
Figure 3: Test accuracy gain from set- model, dataset, and end model combinations: + 1.7 accuracy ting 3 < across all WRENCH trials.
points compared to 3 1.0. However, we show in Section 4.4 that very small validation sets good enough to select 8. The end model trains using 6, but using á to first select good training set further improves performance. Figure 3 displays box plot of the accuracy gain from using sub-selection. Appendix B.2 contains plots of the end model performance versus the coverage fraction B. In some cases, the cut statistic is competitive with COSINE [41]. , which does multiple rounds of self-training on the unlabeled data. Table 7 compares the two methods, and we show in Appendix B.7 how to combine them to improve performance.
For Table 1, we used the same representation for & and for the initial end model. These results indicate that representations from very large models such as GPT-3 CLIP are not needed to improve end model performance. However, there is no a priori reason to use the same representation for ó and for the end model initialization. Using much larger model for & may improve the cut statistic performance without drastically slowing down training, since we only need to perform inference on the larger model. We examine the role of the representation choice more thoroughly in Section 4.2.
4.2 Choice of Representation for Cut Statistic
How important the quality of & (the representation used for the cut statistic) for the performance of subset selection? In this section experiment with different choices of ø. To isolate the effect of

---

Accuracy vs Coverage, CDR
Majority Voto
Snorkel
Dawid Skmne
FlyingSqui
MeTal
Accuracy
0.5
9.70
uithaed
0.775
cutstst-puhmed
0.750
0.6
9.65
0.75
cutstar-bobbrt
0.75
cuistat
0.725
0.2 0.4 0.6 Coverage
0.3
1.0
0.2
0.5 Caverage
0.4
03
1.0
0.2 0.4
0.5 0.8 Coverage
1.0
0.2 0.4
0.6 0.8 1.0 Coverage
0.2 0.4 0.6 0.8
1.9
Coverage
Figure 4: Domain-specific pretraining versus domain pretraining for Φ. Stock BERT (cutstat) compared to BioBERT ccutstat-biobert) and PubMedBERT ccutstat-uubment two models pretrained on biomedical text. The domain- models select more accurate subsets than the generic model.
Accuracy Coverage,
IMDh
Majority
Snorkel
Dawid
FlyingSquid
1.0
1.0
1.0
1.0
1.0
Acammey
method
6.8
0.9
catropy
cuishe
0.6
0.8
0.8
cuistal-sst
0.4 Coverage
1.0
0.2 Coverage
0.6
0.4 Coverage
0.2 0.6 1.0
0.4 088
Covenag
Covenge
Figure 5: Comparison of IMDb training subset accuracy for the cut statistic with generic BERT (cutstat) and BERT fine-tuned for sentiment analysis the SST-2 dataset (cutstat-sst). The fine-tuned representation gives very high-quality training subsets when used with the cut statistic.
on performance, we use the generic BERT as the end model. Performance can improve further when using more powerful representations for the end model well, but use fixed end model here to explore how the choice of & affects final performance. Our results indicate that (i) for weak supervision tasks in a specific domain (e.g., biomedical text), models pretrained on that domain perform better than general-domain models as & and (ii) for a specific task (e.g.. sentiment analysis), models pretrained on that task, but different data, perform very well Φ. We also show in Appendix B that using a larger generic model for & can improve performance when the end model is held fixed.
Domain-specific pretraining can help. The CDR dataset in WRENCH has biomedical abstracts as input data. Instead of using general-domain 6 such as BERT and RoBERTa, does using a domain- specific version improve performance? Figure 4 shows that domain specific models do improve over general-domain models when used in the cut statistic. We compare base--aas-ccse to PubMedBERT- watrrr--bbeeuunnaaeeeee www--uuuaaaddaaaaaaeeeeee www.aaatttttuultttleeeeeeee---ee
[11] and biobert- cased-v1.2 2. [17]. The latter two models were pretrained on biomedical text. The domain-specific models lead to higher quality training datasets for all label models except Dawid-Skene. These gains in training dataset accuracy translate gains in end- model performance. Trained with Y from majority vote and using BioBERT as o, a seneral-domatt BERT end model obtains test Fl score 61.14 (0.64), compared to 59.63 (0.84) using BERT for both & and the end model. Both methods improve over 58.20 (0.55) obtained from training generic BERT with / = 1.0 (no sub-selection).
Representations can transfer. If model is trained for a particular task (e..g, sentiment analysis) on one dataset, can we use it as á to perform weakly-supervised learning on a different dataset? We compare two choices for á on IMDb: regular with fully-supervised learning on the Stanford Sentiment Treehank (SST) dataset [37]. As indicated in Figure 5, the fine-tuned BERT representation selects far higher-quality subset for training. This translates to better end model performance as well. Using majority vote with the fine-tuned BERT as leads to test performance of 87.22 (0.57), compared 81.86 in Table 1. These results suggest that if we have a representation 6 that's already useful for a task, we can effectively combine it with the cut statistic to improve performance on different dataset.
prrt-baae-caseee and bert-base-car cased fine-tuned
4.3 Other Data Modalities
Our experiments so far have used text data, where large pretrained models like BERT and RoBERTa are natural choices for o. Here we briefly study the cut statistic on tabular and image data. The Census dataset in WRENCH consists of tabular data where the goal is to classify whether a person's income is greater than S50k from set of 13 features. We also use these hand-crafted features for & and train a linear model on top of the features for the end model. The Basketball dataset set of still images obtained from videos. and the goal is to classify whether basketball is being
7

---

Table 2: Test F1 of a weakly-supervised linear model (LR) on the Census dataset, which consists of 13 hand-created features. Even though the representation does not come from a large, pretrained neural network, the cut statistic improves the performance of weak supervision for every label model. Test Fl of l-hidden-layer network (MLP) on CLIP representations of the Basketball dataset, where the results noisy, but the cut statistic improves the mean performance of every label model.
Dataset
Majority Vote
Data Programming
Dawid-Skene
FlyingSquid
MeTaaL
Census
50.712.18
21. 17.332
49.9 49.90u.61
38.233.6
11.4
LR
cutstat
57.98g.68
28.04
28.49
20.53
54.99
Basketball
52.29%.62
52. 14 4.80
22.59
54.04
32.99
MLP
+ cutstat
55,823.98
54.59
43.77
56.606.13
47.97 8.89
Table 3: Comparison between using the full validation set to choose and the model checkpoint versus using a randomly selected validation subset of 100 examples. These results use the majority vote (MV) label model. Standard deviation is reported over five random seeds used to select the validation set (not be confused with Table 1, where standard deviation is reported over random seeds controlling the deep model initialization). Most of the drop in performance is due to the noisier checkpoint selection when using the small validation set. Le., the difference between best and 3 1.0 is similar for the full validation and random validation cases.
Val.
imdb
yelp
youtube
semeval chemprot
agnews
full
1.0
78.32
86.85
95.12
66.76
85.17
57.44
86.59
full
best
81.86
89.49
95.60
71.84
92.47
57.47
86.26
BERT
100
1.0
79.17280
84.88
94.50aa3
65.33
85.62a.64
54.99
84.77
100
best
79.752.1
87.96 .00
94.40a6a
74.50
92.40
54.4023s
84,96,
full
1.0
86.99
88.51
95,84
67.60
85,83
$7.06
87.46
full
best
86.69
95.19
96,00
72.92
92.07
59.05
88.01
RoBERTa
100
1.0
85.74
89.32
95. 95.24
66.40
84.38
56.71o.5s
85.79
100
best
85,240.78
93.82a.4s
96.06u.7
75.152s9
91.240.76
56.52,.44
87.20
played in the image using the output of an off-the-shelf object detector in the Ag's. We used CLIP [27] representations of the video frames and trained 1-hidden-layer neural network using the hyperparameter tuning space from [42]. Table 2 shows the results for these datasets. The cut statistic improves the end model performance for every label model even with the small, hand-crafted representation, and also improves for the Basketball data.
4.4 Using a Smaller Validation Set
Many datasets used to evaluate weak supervision methods actually come with large labeled validation sets. For example, the average validation set size of the WRENCH datasets from Table 1 is over 2,500 examples. However, assuming access large amount of labeled validation data partially defeats the purpose of weak supervision. In this section, we show that the coverage parameter / for the cut statistic can be selected using a much smaller validation set without compromising the performance gain over 3 = 1.0. We compare choosing the best model checkpoint and picking the best coverage fraction 9 using (i) the full validation set and (ii) a randomly-sampled validation set of 100 examples. Table 3 shows the results for the majority vote label model. The full validation numbers come from Table 1. The difference between selecting data with the validation-optimal and using 3 1.0 is broadly similar between the full validation and small validation cases. This suggests that most of the drop performance from full validation to small validation due to the noisier choice of the best model checkpoint, not due to choosing a suboptimal B.
4.5 Discussion
Why not correct pseudolabels with nearest neighbor? Consider an example v; whose weak label (a:)) disagrees with the weak label (xx;) of most neighbors j E N(i). This example would get thrown out by the cut statistic selection. Instead of throwing such data points out, we could try to re-label them with the majority weak lahel from the neighbors. However, throwing data out is
more conservative (and hence possibly more robust) approach. For example, if the weak labels are

---

mostly wrong on hard examples close to the true unknown decision boundary, relabeling makes the training set worse, whereas the cut statistic ignores these points. Appendix B.3 contains an empirical comparison between subset selection and relabeling. For the representations studied in this work, relabeling largely fails to improve training set quality and end model performance.
Why does sub-selection work? As suggested above, subset selection can change the distribution of data points in the training set shown to the end model. For example, it may only select "easy" examples. However, this is already problem in today's weak supervision methods: the full weakly- labeled training set 7 is already biased. For example, many labeling functions are sewood-bbsedd such as those in Section 2 ("good" -positive sentiment, "bad" -negative),), In these examples, T itself is biased subset of the input distribution (only sentences that contain "good" or "bad", versus all sentences). Theoretical understanding for why weak supervision methods perform well on the uncovered set (² (xx) & Ø3) is currently lacking, and existing generalization bounds for the end model do not capture this phenomenon. In the following section we present special (but mmatically-motivann case where this bias can be avoided. In this case, we prove a closed form for the coverage- prenngeepreciiionn tradeoff of selection methods, giving subset selection some theoretical motivation.
5 Theoretical Results: Why Does Subset Selection Work?
We begin by presenting theoretical setup motivated by the CheXPert [13] and MIMIC-CRR [14] datasets, where the weak labels are derived from radiology notes and the goal to learn an end model for classifying X-ray images. Suppose for this section that we have two (possibly related) views ( ). 1 of the data X. i... v/ : X' - Vo. 2/ X - I, We use & here distinguish from o, the representation used to compute nearest neighbors for the cut statistic. For example, if the
input space X is multi-modal, and each v; (a." .r ) )r.")) then we can set tha and t/ to project onto the individual modes (e.g., (X) the clinical note and Φ1 (X) the X-ray). We will assume that the labeling functions Ag (zz) only depend on (u (z;,). and that the end model f only depends on 4/1 (x:)). In the modal example, this means the labeling functions are defined on one view, and the end model trained on the other view. To prove a closed form for the precision/coverage tradeoff, we make the following strong assumption relating the two views tho and 4/1:
Assumption 1 (Conditional independence). The random variables vo vo)X), v/ (X) are conditionally independent given the true (unobserved) label Y. That is. for any sets Vo. B c Vn,
P x (xoo E A, (X() E B|Y] = Px,yL(o(X) E
E BJY].
Note since every As only depends on v ( X), the pseudolabel Ý only depends on (X( Hence F(X)= ((o and likewise for end model f. fx)) = f(v Assumption implies: [X)(X) e BJY]
Px,r (T) * Y1, (X) B|Y] = PX) * Y|
for every B 11 In this special case, the end model training reduces to learning with class- conditional noise (CCN), since the errors (X) * Y] are conditionally independent of the repre- sentation ạ (X) being used for the end model. This assumption is most natural for the case of multi- modal data and voo, 1/1 that project onto each mode, but it may also roughly apply when the representa- tion being used for the end model (such as a BERT representation) is "suitably orthogonal" to the input X. While very restrictive, this assumption allows to make the coverage coverage-precision tradeoff precise.
Theorem 1. Suppose Assumption 1 holds, and that y {0. 1). Define the balanced error of classifier f on labels Y as: erTbal(f,Y) = (P] = O = 1] + P[ = 1|Y = O)). 10,1) be an arbitrary label model. Define a =Pp[Y- = ojř = 1] and PP = 1/Ý = 0] and suppose a a+* + < 1, PLý = yI > 0 for y E {0, 13. These parameters measure the amount of noise in Y. Define f+==inm fer
We write f(X) instead of f(1/1/12//) (X)) for convenience. Let Ý XX
Let I be the classifier obtained by minimizing the empirical balanced accuracy on ((xi, Y (x:)) Then the following holds w.p. 1 - å over the sampling of the data:
1
VC(F) + log !
-
- errbai (rr Y) &
1 1-a-2
nPIl # min, PT y//² # 00
where Ô hides log factors in m and VC(F).
Proof. For space, we defer the proof and and bibliographic commentary to Appendix A.
9

---

This bound formalizes the tradeoff between the precision of the weak labels, measured by a and and the coverage. measured by nPIYY * 2). which for large enough samples is very close to the size of the covered training set 7 = {(2,, Y (z;)) (x;) * Ø). Suppose we have a label model Y and an alternative label model Y that abstains more often than Y (so P[Y' * Ø] < PI * Φ)) but also has smaller values of a and y. Then according to the bound, an end model trained with Y" can have better performance, and the empirical results in Section 4 confirm this trend.
This bound is useful for comparing two fixed label models Y, Y with different abstention rates and (a, 7) values. However, we have been concerned in this paper with selecting a subset T' of 7 based on single label model Y, and training using T', We can represent this subset selection with new set of pseudolabels (xx) I, T that abstains more than (z+) points not chosen for T get (x;) 2. However, selection for T' depends on sample-level statistics, so the Y values are not i..dd, which complicates the generalization bound. We show in Appendix A that this can be remedied by a sample-spliting procedure: we use half T to define a refined label model Y Vo - y {Ø), effectively reduce to
and then use the other half of 'T as the initial training set. This allows the case of two fixed label models Y, Y and apply Theorem 1. We include the simpler Y versus Y bound here because it captures the essential tradeoff' without the technical difficulties.
6 Limitations, Societal Impact, and Conclusion
Surprisingly, using less data can greatly improve the performance of weak supervision pipelines when that data is carefully selected. By exploring the tradeoff between weak label precision and coverage, subset selection allows us to select a higher-quality training set without compromising generalization performance the population pseudolabeling function. This improves the accuracy of the end model on the true labels. In Section 5, we showed that this tradeoff can be formalized in the special setting of conditional independence. By combining the cut statistic with good data representations, we developed technique that improves performance for five different label models, over ten datasets, and three data modalities. Additionally, the hyperparameter tuning burden low. We introduced one new hyperparameter 3 (the coverage fraction) and showed that all other hyperparameters can be re-used from the full-coverage 3 1.0 case, so existing tuned weak supervision pipelines can be easily adapted to use this technique.
However, this approach is not without limitations. The cut statistic requires a good representation of the input data to work well. Such a representation may not be available. However, for image or text data, pretrained representations provide natural choices for Φb Our results on the Census dataset in Section 4.3 indicate that using hand-crafted features as ø can also work well. Finally, as discussed at the end of Section 4.5, subset selection can further bias the input distribution (except in special cases like the one in Section 5). However, this is already an issue with current weak supervision methods. Most methods only train on the covered data T. Labeling functions are typically deterministic functions of the input example, (such as functions based on the presence of certain tokens) and so the support of the full training set 'T is a strict subset of the support of the true input distribution, and T may additionally have a skewed distribution over its support. This underscores the need for (i) the use of ground-truth validation set to ensure that the end model is an accurate predictor on the full distribution (ii) in high stakes settings, sub-group analyses such as those performed by [35], to ensure that the pseudolabels have not introduced bias against protected subgroups and (iii) the need for further theoretical understanding on why weakly supervised end models are able to perform well on the uncovered set (æ A(x)= = Ø).
Acknowledgments and Disclosure of Funding
This work was supported by NSF AitF awards ICF-5663585 and C72-1723344. Thanks to Hussein
Mozannar for helpful conversations on Section 5 and the pointer to Woodworth et al. [39]. Thanks to Dr. Steven Horng for generously donating GPU-time on the BIDMC computing cluster [12] and to NVIDIA for their donation of GPUs also used in this work.
10

---

References
[1] Anelia Angelova, Yaser Abu-Mostafam, and Pictro Perona. Pruning training sets for learning of object categories. In 2005 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR'OS)) volume pages 494-501. IEEE, 2005.
[2] Dana Angluin and Philip Laird. Learning from noisy examples. Machine Learning. 2(4): 343-370, 1988.
[3] Chidubem Arachie and Bert Huang. Adversarial label learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3183-3190, 2019.
[4] Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi. Introduction to statistical learning theory. In Summer school on machine learning, pages 169-207. Springer, 2003.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] Mayee F Chen, Daniel Yang Fu, Dyah Adila, Michael Zhang, Frederic Sala, Kayvon Fata- halian, and Christopher Re. Shoring up the foundations: Fusing model embeddings and weak supervision. The 38th Conference on Uncertainty in Artificial Intelligence, 2022.
[7] Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependere label noise: A sample sieve approach. In International Conference on Learning Representations, 2021.
[8] Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28( 2:20-28, 1979.
[9] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171 1-66, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18853/11/119-11 1423. URL battts::/aclanttaiie
org/ N19 1423.
[10] Daniel Fu, Mayee Chen, Frederic Sala, Sarah Hooper, Kayvon Fatahalian, and Christopher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods. In International Conference Machine Learning, pages 3280 3291. PMLR, 2020.
[11] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions Computing for Healthcare (HEALTH)) 3(1):1-23, 2021.
[12] Steven Horng Machine Learning Core. 2 2022. doi: (1.600///1.iiihhhhe..99900111121 URL wwt./oooaiiaooo
https://figet
[13] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-lleus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590-597, 2019.
[14] Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum. Matthew P Lungren, Chih-ying Deng, Roger Mark. and Steven Horng. Mimic-cxI, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):1-8, 2019.
[15] Giannis Karamanolakis, Subhabrata Mukherjee, Guoqing Zheng, and Ahmed Hassan. Self- training with weak supervision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 845-863, 2021.
11

---

[16] Hunter Lang, Monica N Agrawal, Yoon Kim, and David Sontag, Co Co-training improves prompt- based learning for large language models. In International Conference on Machine Learning. pages 11885-12033. PMLR, 2022.
[17] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4): 233--2400, 2020.
[18] Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Leamingg with noisy labels as semi-super learning. In International Conference on Learning Representations, 2019.
[19] Ming Li and Zhi-Hua Zhou. Setred: Self-training with editing. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 61 1-621. Springer, 2005.
[20] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint 190i 1669, 2019.
[21] Ayush Maheshwari, Oishik Chatterjee, Krishnateja Killamsetty, Ganesh Ramakrishnan, and Rishabh lyer. Semi-supervised data programming with subset selection. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4640-4651, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/200 1t2.:///.2....nngssaaae
408. URL https: : a/acaanhoo/y org/2021 findings-acl... acl.
[22] Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from comupted binary labels via class-probability estimation. In International conference machine learning, pages 125-134. PMLR, 2015.
[23] Fabrice Muhlenbach, Stéphane Lallich, and Djamel A Zighed. Identifying and handling mislabelled instances. Journal of Intelligent Information Systems, 22(1):89-109, 2004.
[24] Subhabrata Mukherjee and Ahmed Awadallah. Uncertainty-awarr self-training for few-shot text classification. Advances in Neural Information Processing Systems, 33:21199-21212, 2020.
[25] Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411 2021.
[26] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1944-1952, 2017.
[27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pages 7768-8763. PMLR, 2021.
[28] Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the VLDB Endowment. International Conference Very Large Data Bases, volume 11, page 269. NIH Public Access, 2017.
[29] Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christopher Ré. Training complex models with multi-task weak supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4763 2019.
[30] Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. Advances in neural information processing systems, 29:3567-3575 2016.
[31] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representations, 2022.
12

---

[32] Norbert Sauer. On the density of families of sets. Journal of Combinatorial Theory, Series A, 13(1):145-147, 1972.
[33] Clayton Scott, Gilles Blanchard, and Gregory Handy. Classification with asymmetric label noise: Consistency and maximal denoising. In Shai Shalev-Shward and Ingo Steinwart, editors, Proceedings of the 26th Annual Conference on Learning Theory, volume 30 of Pruceedings of Machine Learning Research, pages 489-511, Princeton, NJ, USA, 12-14 Jun 2013. PMLR. URL. https::/ pttt::://proceedingg.. mlr. press/v300 v30/Scott13..... html.
[34] Henry Scudder. Probability of error of some adaptive pattern-recogniitoon machines. JEEE Transactions Information Theory, 11())363-337, 1965.
[35] Laleh Kalantari, Guanxiong Liu, Matthew McDermott, Irene Y Chen, and Marzyeh Ghassemi. Chexclusion: Fairness gaps in deep chest x-ray classifiers. In BIOCOMPUTING 2021: Proceedings of the Pacific Symposium, pages 232- 243.. World Scientific, 2020.
[36] Saharon Shelah. A combinatorial problem; stability and order for models and theories in infinitary languages. Pacific Journal of Mathematics, 41(1)2247-261, 1972.
[37] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[38] VN Vapnik. On the uniform convergence of relative frequencies of events their probabilities. Theory of Probability and its Applications, 16(2):264-281, 1971.
[39] Blake Woodworth, Suriya Gunasekar, Mesrob Ohannessian, and Nathan Srebro. Learning non-discriminatory predictors. In Conference on Learning Theory, pages 1920-1953. PMLR, 2017.
[40] David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Annual Meeting of the Association for Computational Linguistics, pages 189-196, Cambridge, Massachusetts, USA, June 1995. Association for Computational Linguistics. doi: 10.3115/ 981658.981684 URL https://aclanthoot/ aclaaahollnooo....oc www.hhoooooooogogr.g.eg wtt/ 1026.
[41] Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang. Fine-tuning pre- trained language model with weak supervision: A contrastive-regulaizee self-training approach. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1063-1077, 2021.
[42] Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner. Wrench: A comprehensive benchmark for weak supervision. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[43] Min-Ling Zhang and Zui Zhou. Cotrade: Confident co- co-training with data editing. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 41(6):1612-1625 2011.
[44] Yan Zhou, Murat Kantarcioglu, and Bhavani Thuraisingham. Self-training with selection-by rejection. In 2012 IEEE 12th international conference on data mining, pages 795-803. IEEE, 2012.
[45] Zhaowei Zhu, Zihao Dong, Hao Cheng, and Yang Liu. A good representation detects noisy labels. arXiv preprint 10.00221 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] We show both empirical and theoretical benefits of using less weakly-labeled data to improve performance.
(b) Did you describe the limitations of your work? [Yes] In Sections 4.5 and 6.
13

---

(c) Did you discuss any potential negative societal impacts of your work? [Yes] In Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes] In Assumption
I and in the statement of Theorem
(b) Did you include complete proofs of all theoretical results? [Yes) Appendix
3. If you experiments..
(a) Did include the code, data, and instructions needed to reproduce the main experi-
mental results (either the supplemental material or as a URL)? [Yes] In Appendix and in ZIP file in the supplementary material.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] In Section 4.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] Our tables Section 4 include standard deviations across
5 runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Described in Section 4.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We primarily used the WRENCH benchmark [42].
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL?
[Yes] We include the code for reproducing our empirical results in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "VLlfp1a1Bjk", "reviewer": "Reviewer_ocdh", "summary": "This paper studies the subset selection problem of weakly supervised learning. By dividing a weakly supervised learning pipeline into labeling functions, label model and end model, the paper specifically studies on label model which motivates the proposed subset selection method based on cut-statistics. Different from entropy-based subset selection, cut statistic based selection relies on identifying highly noisy examples based on their proximity in a constructed feature graph. The proposed selection method serves as a plugin for any label model and empirical evaluations are presented to show that it improves many existing label model and end models.", "questions": "A minor question regarding connection to semi-weakly supervised learning: to get best performance on a particular task, a validation set is required to set the selection threshold \\beta, then this essentially leverages information about true labels, I wonder how does the proposed method compare to weak label re-weighting / correction method. e.g., [1][2] (some of these require a small set of clean labels for learning, and for the studied setting in this paper weak labels + validation data, for a stretched setting we can split the validation data to use partially for training and the rest for validation and thus apply the above mentioned methods.)\n\n[1] Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting. NeurIPS '19\n[2] Meta Label Correction for Noisy Label Learning. AAAI '21", "limitations": "Potential limitations are well addressed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Using the cut statistics to identify highly noisy labels is an elegant and novel idea, which doesn't require any knowledge about true labels. Justifying using less data may actually lead to better performance in weakly supervised learning carries the most contribution to me.\n2. The proposed selection approach is simple but yet very effective which improves many existing label model and end model. \n3. The paper is extremely well presented ranging from motivation, formulation, method coverage, empirical and theoretical justifications.\n\nWeakness:\nI find no obvious weakness of the paper and it's a very enjoyable read.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7AbQvMN04h", "reviewer": "Reviewer_Y4et", "summary": "This paper proposes a simple dataset selection strategy for weakly-supervised\nlearning. The strategy is based on the cut statistics, which constructs a\nsimilarity graph among the instances and select the nodes that have consistent\nlabels with their neighbors. The authors have shown such a simple data\nselection strategy enhances previous labeling models for weakly-supervised\nlearning. They also provide a theoretical analysis on the generalization\nbound, which formulates a tradeoff between precision and coverage for weak\nlabels.\n", "questions": "\nPlease see the weaknesses above.\n\nBesides the above two concerns, an additional question is that there are\nrecent work [1] showing relabeling is better than discarding data in the\ncontext of supervised learning. It'll be great if the authors can comment on\nthese two seemingly contradictory findings.\n\n[1] Resolving Training Biases via Influence-based Data Relabeling\n\n", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n\nS1. The proposed selection strategy based on cut statistics is simple but effective empirically.\n\nS2. The authors have conducted experiments on datasets with different modalities, including text, image, and tabular data.\n\nS3. The paper is well written and easy to follow.\n\n\nWeaknesses:\n\nW1. There is a gap between the theorem and the experiments. In the theorem,\nthe author assumes two different views that are conditionally independent. The\nlabeling functions are defined on one view, and the classifier is trained on\nthe other. Unless I'm missing anything, there are no such complementary views\non the datasets used in experiments. Also, even if we have multi-view data\nsets in practice, the assumption that the two views are conditionally\nindependent, and that we use the two view separately for labeling function and\nmodel training are impractical.\n\nW2. The authors mainly evaluated the selection strategy for labeling model.\nHowever, as the authors mentioned, there are works (e.g., cosine) that perform\ndata selection during the end model training. I do not see any major obstacles\nthat prevent the proposed data selection strategy from being applied during\nthe end model training. It'll be more interesting to see how this measure compared with the previous methods that uses data selection during the end model training.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6GZZ7dhCp-", "reviewer": "Reviewer_TRjo", "summary": "This paper proposes a method to improve the precision of existing weak supervision approaches by selecting a subset of the weakly-labeled data for training the end model. Specifically, the paper proposes to use the cut statistic that considers vector representations of the input examples, and keeps examples with the same pseudolabel as their nearest neighbors in this vector space. These examples are identified by constructing a graph where each instance is a node, computing cuts using weak labels, and identifying the number of cut edges for each node. \n\nThe proposed approach is evaluated across several benchmarks for weak supervision and compared with an entropy-based measure for subset selection. ", "questions": "Are the authors planning to share their code for replicating the experiments reported in the paper (in addition to the code snippet in the appendix)?  ", "limitations": "Yes. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n* The paper is very clearly written and easy to follow. \n* The paper proposes a simple method that effectively improves the precision of existing weak supervision methods across several datasets. It is a great benefit that the proposed method can be applied in general across label models, end models, and data modalities, and as a result, can serve as a useful tool for the weak supervision community. \n* Additional experiments, such as evaluating the choice of the representation for the cut statistic are very interesting. \n* The paper provides a theoretical analysis on a special case of weak supervision that motivates subset selection and can help better understand the precision-coverage tradeoff. \n\nWeaknesses: \n* The paper experimentally compares the cut statistic approach to just a very simple entropy-based approach and does not consider existing work that attempts to improve the precision of the existing sources, e.g., via instance-specific weights (Awasthi et al., 2020; Karamanolakis et al., 2021; Yu et al., 2021). A more thorough experimental comparison would better place the paper in the context of state-of-the-art approaches and would make the paper's contribution stronger. \n\nMinor comments: \n* Section 3.1 could be a better fit under the \"Experiments\" section.\n\nReferences: \n* Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi, \"Learning from rules generalizing labeled exemplars\", (ICLR 2020)\n* Giannis Karamanolakis, Subhabrata Mukherjee, Guoqing Zheng, Ahmed Hassan Awadallah, \"Self-training with weak supervision\", (NAACL 2021)\n* Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, Chao Zhang, \"Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach\", (NAACL 2021)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3AJyEIPMmPc", "reviewer": "Reviewer_APNJ", "summary": "The authors propose a simple method for combining pretrained data representations with the cut statistic to select better quality subsets of the weakly-labeled training data. By selecting subsets of data rather than using the full covered data, a weak supervision label model is able to achieve better precision. The authors also show that their method also improves performance of end models.\n", "questions": "I like the direction of this work and the potential gain to weak supervision however I think a major flaw in the paper lies in selecting a good value for \\beta. In the paper, the authors use a validation set to select the best \\beta for the results in the table. This will not be practical when a validation set is not available and if the cost of labeling the data to obtain a validation set is high. Looking at the plots, there is no obvious alternative strategy for selecting \\beta as it depends on the classification task. I think it will be very helpful if the authors can suggest an approach for selecting \\beta when there is no access to validation data and maybe show experiments for it.", "limitations": "The authors addressed potential limitations of their paper. The work has no significant negative societal impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The method is simple and easy to integrate with any weak supervision approach.\n\n2. The authors show strong empirical results and provide theoretical guarantees for their approach.\n\nWeakness:\n - Selecting good value of \\beta may not be practical", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept