Label Augmentation with Reinforced Labeling for Weak Supervision
Authors: Gürkan Solmaz
Flavio Cirillo
Fabio Maresca
Anagha GodeAnilKumar
Venue: ICLR 2022 Submitted (2022)
Keywords: weak supervision
data programming

Abstract: Weak supervision (WS) is an alternative to the traditional supervised learning to address the need for ground truth. Data programming is a practical WS approach that allows programmatic labeling data samples using labeling functions (LFs) instead of hand-labeling each data point. However, the existing approach fails to fully exploit the domain knowledge encoded into LFs, especially when the LFs' coverage is low. This is due to the common data programming pipeline that neglects to utilize data features during the generative process.
This paper proposes a new approach called reinforced labeling (RL). Given an unlabeled dataset and a set of LFs, RL augments the LFs' outputs to cases not covered by LFs based on similarities among samples. Thus, RL can lead to higher labeling coverage for training an end classifier. The experiments on several domains (classification of YouTube comments, wine quality, and weather prediction) result in considerable gains. The new approach produces significant performance improvement, leading up to +21 points in accuracy and +61 points in F1 scores compared to the state-of-the-art data programming approach.


Under review as a conference paper at ICLR 2022
LABEL AUGMENTATION WITH REINFORCED LABELING FOR WEAK SUPERVISION
Anonymous authors
Paper under double-blind review
ABSTRACT
Weak supervision (WS) is an alternative to the traditional supervised learning to address the need for ground truth. Data programming is a practical WS approach that allows programmatic labeling data samples using labeling functions (LFs) instead of hand-labeling each data point. However, the existing approach fails to fully exploit the domain knowledge encoded into LFs, especially when the LFs' coverage low. This due to the common data programming pipeline that neglects to utilize data features during the generative process. This paper proposes a new approach called reinforced labeling (RL). Given an unlabeled dataset and a set of LFs, RL augments the LFs' outputs to cases not covered by LFs based on similarities among samples. Thus, RL can lead to higher labeling coverage for training end classifier. The experiments several domains (classification of YouTube comments, wine quality, and weather prediction) result in considerable gains. The new approach produces significant performance improvement, leading up to +21 points in accuracy and +61 points in Fl scores compared to the state-of- the-art data programming approach.
INTRODUCTION
Supervised machine learning has proven to be very powerful and effective for solving various classifi- cation problems. However, training fully-supervised models can be costly since many applications require large amounts of labeled data. Manually annotating each data point of a large dataset may take up to weeks or even months. Furthermore, only domain experts can label the data in highly specialized scenarios such as healthcare and industrial production. Thus, the costs of data labeling might become very high.
In the past few years, a new weak supervision (WS) approach, namely data programming [Ratner eet al. (2016) 2017]], has been proposed to significantly reduce the time for dataset preparation. In this approach, domain expert writes heuristic functions named labeling functions (LFs) instead of labeling each data point. Each function annotates a subset of the dataset with an accuracy expected to be better than a random prediction. Data programming has been successfully applied to various classification tasks. However, writing LFs might not always be trivial for instance, when data points are huge vectors of numbers or when they are not intuitively understandable. Developers can quickly code a few simple functions, but having heuristics to cover many corner cases is still a burden. Further, simple heuristics might cover only tiny portion of the unlabeled dataset (small coverage problem).
The existing data programming framework Snorkel [Ratner et al. (2016(2017) implements machine learning pipeline as follows. The LFs are applied to the unlabeled data points and the outcomes of LFs produce labeling matrix where each data point might be annotated by multiple, even conflicting, labels. A generative model processes the labeling matrix to make single label predictions for a subset of data points, based the agreements and disagreements a:(() using techniques such as majority voting (MV) or minimizing the marginalized log-likelihood estimates. Later, the label predictions are used to train a supervised model that serves as the end classifier (discriminative model). This approach has two major limitations: 1) Coarse information about the dataset (only LF outputs) fed to the generative model, 2) Lack of generalization due to the sparsity of labeling matrix and relying only on end classifier generalize. The current data programming does not take the data points' data features into account during the generative process, even though they available throughout the pipeline. It utilizes the data features of only the data
the LF outputs for a given data point

---

Under review as a conference paper at ICLR 2022
Labs Fardliosa
IF bbel Nagmercstionn
Generative process
Further perendiaation
Fir Drement
Reinforced labeling
033
0.02
Rainbo sar labela
0.2
0.56
0.2
Classiee
End
Cenerative mocel
prodictions
049
0.50
0.01 0.6
0.33
0.16
fuxtares
Figure 1: The label augmentation pipeline that brings data features f1, f2...... fn and weakly- supervised labels LF1, LF2..... LFm together in the generative process.
points with label predictions to train the end classifier. Various additional techniques are considered to complement the existing approach [Varma Ré (2018); Chatterjee et al. (2020): Varma al.) (2019); Nashaat et al. (2018): Varma et al. (2016]] and improve the learning process. However, these approaches do not address this major problem the design of existing data programming.
This article proposes label augmentation approach for weak supervision that takes the data features and the LF outputs into account earlier in the generative process. The proposed approach utilizes the features for augmenting labels. The augmentation algorithm, namely reinforcement labeling (RL), checks similarities between data features to project existing LF outcomes to non non-cxisting labels the matrix (i.e., to unknown cases abstains). Moreover, it uses heuristic that considers unknown cases "gravitate" towards the known cases with LF output labels. In such a way, RL enables generalization early on and creates "reinforced" label set to train an end classifier.
Label augmentation extends the data programming to new scenarios, such as when LFs have low coverage, domain experts can implement only limited number of LFs, or LFs outcome result in a sparse labeling matrix. Label augmentation can provide satisfactory performances in these cases, although data programming was previously non-applicable. The proposed approach can reduce the time spent by the domain experts to train a classifier as they need to implement fewer LFs. One advantage compared to the existing complementary approaches that RI. does not require any additional effort for labeling data, annotating data, or implementing additional LFs. In other words, the label augmentation enhances classification without any further development burden or assumption of available labeled datasets (e.g.. the so-called "gold data"). Furthermore, one can easily combine this approach with the existing solutions.
The RL method is implemented and tested compared to Snorkel (Sn) using different suly--uupervieed models as end classifiers. The experimental results span classification tasks from several domains (YouTube comments, white/red wine datasets, weather prediction). The new approach outperforms the existing model in terms of accuracy and Fl scores, having closer outcomes to the fully-supervised learning, thanks to the improved coverage that enables end classifier convergence.
2 METHOD DESCRIPTION
2.1 BACKGROUND ON DATA PROGRAMMING
In data programming Bach et al. (2019 : Ratner et al. (017)). ]. a set of LFs anmotate a portion (subset) of the original unlabeled dataset X = {a x
x (5) with total labeling coverage ny * [X], where y E [o, 1]. Given a data point x (1) an LF, takes (0)) as input and annotates the input with a label. LFs are considered weak supervisors implemented by application developers, and they can programmatically annotate many data points at once, as opposed to hand-labeling data points one by one. On the other hand, LFs may have lower accuracies than ground truth for the data points.
For a binary classification task, an LF may return two classes or abstain from making a prediction. For simplicity, let us consider T.F, returning I or 0 as the class labels and- (abstain) when it refrains
2

---

Under review as a conference paper at ICLR 2022
Iteration over LFs
Lin
espp Jano ungezai
x!9
0.23
0.59
"
Bf
x <0 (1 (o (1
x
0.01
"
x
0.32
0.16
x(0]
0.15
0.92
0.01
AL method Iteration LR. forx
Data points
LF outputs
Data features
(F5()00))
+
Abstain
Figure 2: Illustration of the reinforced labeling method (see Alg. | in Supp. material A.3
from class prediction. LFs' outputs form a labeling matrix where rows represent the data point indices x(1), ,x(2) ..... .z.2)).....c (2()), and columns represent the LF indices LFi, LF2...., ..,2....,............ LFm. A generative model takes the labeling matrix as input, filters out the data points with no label (all LFs voted for abstains), and tries to predict a label for the remaining data points. An example of a generative model might be a majority voter (MV) based on LF outputs or by minimizing the negative log marginal likelihood Ratner et al. 2017)] (likelihood over latent variables, i.e., LF outputs). Dependency structures between the LFss are learned as shown Alg. in [Bach et al. (2017)]. In both MV and marginal likelihood approaches, the generative model takes the labeling matrix as input and tries to make label decision based on the agreements or disagreements of LFs. The generative model may fail to make decision in certain cases, such when equal numbers of LFs disagree the features of the weakly-labeled data points within X and the labels from the generative model used to supervise end classifier (discriminative) model. The design of the data programming model is agnostic to the end classifier (discriminative) model, so various supervised machine learning models can be candidates the end model.
data point. Lastly,
2.2 LABEL AUGMENTATION AND REINFORCED LABELS
In the descrihed design of data programming, the two limitations mentioned above (in Sec. namely the coarse information in the labeling matrix and sparsity, lead to failure for generalizing to new and unseen data points. Therefore, these limitations may lead to reduced performances. In such scenarios, the outputs of the existing generative model may not be satisfactory to train the end classifier. Therefore, the end classifier model may not converge generalize well enough to cover different cases. Implementing many LFs that cover different cases is costly and not very straightforward in most scenarios. Although various additional techniques focus the weak supervision problem [Varma & Ré 2018)); Chatterjee et al. 2020)); Varma et al. (2019): Nashaat et al. (2018); Varma et al. (2016)), J, they rather extend the existing pipeline with additional features. On the other hand, label augmentation targets these major limitations by eliminating the sparsity using data features. Thus, it can lead to higher accuracy.
Fig. M illustrates the new pipeline for label augmentation. The new generative process brings together the outputs of (LF1, LF2, LFn), and the data features fi, f2....... ......ff fn of data points (+)) = {x)() ,I2 ...... xx, = } in the unlabeled dataset early on. Different methods can utilize data features in the generative process for augmenting the labels in the labeling matrix. This label augmentation approach differs from the existing "data augmentation" approaches [Wang et al. (2019); Tran et al. 2007):|Cubuk Cubuk et al. (2019) that create new (synthetic) data points, as the new goal is to project LF outcomes to the existing data points as opposed to creating new data points.
The outcome of the new generative process can be more representative of the data and weak supervi- sors than the outputs of the previously existing generative process due to additional coverage and accuracy gains without any additional LF implementation or data annotation. For instance, abstain values (-1s) from the LFs' outputs representing the unknown cases (left side of the matrix in Fig. 1 can be predicted by the new generative process (as outlined in Sec. te3.).3). The abstain values in the labeling matrix can be augmented with classification prediction values.
Data programming applies statistics only to data points that are already covered by LFs, resulting in a single predicted label for subset of the dataset. Similarly, by using likelihood estimation over the
3

---

Under review as a conference paper at ICLR 2022
outliers
I data points features X2.22
(F >E
Ppus = Q3 IQR
Ragi
x(i)
( 2
I.f (x(11) =0
Q3
(R(("")) -11
Aggregated
boundaries adjustment based on IQR factor
median
10R
Li (x(""
22
(f((+"))= -1
Distance adding regative effect
Q1
Distance adding positive effect
-QR Dnog 21
PIgR
Distance exceeding Ey
Aggregated effect threshold
outliers
Figure 3: Illustration of the gravitation-base RL method
Figure 4: The IQR factor adjustment to dynamically select e for gravitation.
augmented matrix, the generative process predicts "reinforced labels". The details of the generative be seen in [Bach al. (2017)] After the generative process, the reinforced labels are used to train the end classifier. The end classifier can still be the same supervised machine learning model. As the augmented matrix has more density compared the labeling matrix, it may lead to a larger training set. As a simple example, a disagreement between two LFs for a data point can be eliminated by a new prediction for the abstain case of third LF. As a result, more data points can be used to train the end model. The benefits of using the end classifier include further generalization for the data points that are still not labeled. Furthermore, the reinforced labels can fine-tune pretrained machine learning models.
model implementation
2.3 IMPLEMENTING REINFORCED LABELING
Let us describe the RL method for label augmentation in the generative process. The intuition behind this method is that a data point æ(0) that LF does not label, might have very similar features compared to another data point, such as a data point (1()) labeled by the same LF. ((i might not be labeled due to the conditions boundaries in the LF missing subset of the data features in its heuristic implementation. Reinforcing the labels means predicting label for the previously unlabeled data points by certain LFs and therefore creating a denser version of the labeling matrix
during the generative process. Furthermore, RL could lead to a higher coverage 7 [X] 7 * [X] for training the end classifier.
Fig. 2 illustrates the RL method. The method iterates over (LF,, ......., LFm) Fm and for each LF on the left part of the matrix, it finds the data points x(5)s that LF, abstains from (-Is on the matrix). The abstain points are compared with the points that are labeled by LFi for their distances based
on data features. The similarity between a point (+)) labeled by LF, point and the unlabeled x(6
represented by the "effect function" (ff(x LF((x x()). As inputs, the effect function has the data features of the labeled ( = (x(
222 ..... (6) output label (F((z()) and the features of (3)
(1)
the unlabeled (0)) = (-(2) 22 In Based on the label, the effect may take a positive value (if (z = 1) or a negative value (if LF((x()) = 0). For any other data point that the LF, also abstains, it takes the value zero. These values can be aggregated as Σ ==1 (ff((()) LF (()) for any LFj and where
|X|.
We propose heuristic algorithm to implement RL based on gravitation. Fig. illlstraats the heuristic method considering a simple case having only two data features Tı, 222 In the gravitation- based RL method, for the abstain point .ا(ر) that is not labeled by LF,, each other point that LF; labels (shown as colored particles in Fig. 3 is considered
particle that attracts z(1) towards positive (L))(I()) = 1) or negative (z)) = 0) aggregated effect. Fig. shows positive or negative attractions by different-colored lines between the data points. The attraction inversely proportional to the pairwise distance of every labeled point (+)) to (t) For optimizing the caleulations,
4

---

Under review as a conference paper at ICLR 2022
consider maximum distance threshold Eg above which attraction does not take place. Hence, the effect function is defined as follows:
Distance(2 x0)) < Ed
Eff (2)))
Distance (alt (a(0,ata) a
otherwise,
where a and B : are constants for optional adjustment of the effect and distance relationship. As a distance function Distance(z1),, x00 RL can use different metrics based on the data types (c.g.. sensor readings, text, image) such as Euclidean, Levenshtein or Mahalanobis distances. The effeets of all labeled points are aggregated based on their positive or negative attractions. The decision of updating an abstain value depends upon a parameter called the aggregated effect threshold e. x!i) labeled with that class for the given LP; if the following condition holds:
I Σ Eff (,.0), L.F(((()))) (,01))
> E.
(2)
& adjusts the degree of reinforcement and the density of the resulting augmented matrix. In the evalu- ation section, we consider various empirical parameters and heuristic for automatic configuration.
EXPERIMENTAL EVALUATION
3.1 BENCHMARK FRAMEWORK
The experimental study includes four metries: Classification accuracy (# correct/# tested), precision. recall, and F1 score (FI). Particularly, higher accuracy and Fl score are important to understand the performance for the given classification task for non-biased and biased tasks, respectively. Other than these performance metrics, the experimental evaluation includes insights on the labeling itself (labeling metrics) such as: Number of LFs: The number of LFs used as weak supervisors; Labeled samples: The number of samples labeled for training the end models; LF coverage: The ratio of the number of data points labeled by LFs to the number of samples; LF overlap: The ratio of LFs' overlapping outputs with each other to the number of samples; LF contlicts: The ratio of LFs' conflicting outputs with other each other to the number of samples. The last three metrics represent the mean values, averaged based on the number of LFs.
We evaluate the RL approach with four datasets from different domains: YouTube comments AAlerto ett al. [0015)]. red and white wine quality [Cortez et al. 20009]] and Australia Weather (rain) datasets" YouTubb comments dataset consists of texts that may be legit user comments or spam. This text-based
dataset is used for benchmarking various data programming approaches [Chen et al. (2020); Evensen
ct al. (2020); |en et al.] (2000): Karamanolakis et al. (202T); [Sedova et al. (2021 ): Awasthi et al.
(2020) and also Snorkel's 's tutorial for LFs. The YuuTube dataset largely unlabeled except for small testing dataset. Only the text comment is used as feature for the You dataset, whereas we removed the others (such as user ID and timestamp). On the other hand, the end models get a sparse matrix of token counts computed with Scikit-learn Count Vectorize To classify the wine quality, there exist 12 real number features (e-g., acidity, residual sugar) for the two wine datasets. For training and testing, a wine is considered good (labeled with 1) when the wine quality feature is more than out of 10; otherwise considered a bad wine (0). The Australia Weather dataset is widely used for the next-day rain predictions. It also consists of 62 features based on daily weather observations spanning ten years period. The wine and Australia Weather datasets are fully labeled with ground truth. For all datasets Euclidean distance metric computes distances between data point pairs. For the YouTube dataset, Euclidean distance is applied to the one-hot encoding of the tokenized texts
Snorkel (Sn) and fully-supervie learning (Sup) are the two main approaches for comparison. In addition to those, the majority voting (MV) approach is tested as the simple generative model. For
'https://www.t kaggle. hww.gg/.....iiiiooaaaaaaaaaaaaae
http:///hh....ooooooooooooonnnne htt.:///ammmmmtttttttttttttatotn text, CountCCConnnnnizeeeo
hwtp://www...piii........c...... tokenize.hmm
5

---

Under review as a conference paper at ICLR 2022
Reinforeed labeling
Snorkel
Supervised learning F1
Dataset
Acc
Prec
Rec
F1
Fl-Gain
Ace
Prec
Rec
F1
Ace
Prec
Rec
YouTube
0.75 0.98 0.47 0.64
+61
0.54 1.00 0.02 0.03
0.91 0,96 0.84 0.90
Red Wine 0.71
0,80
0.66 0.72
+7
0.61
0.66 0.77
0.65
0.75
0.81 0.74 0.76
White Wine
0.63
0.64 0.98
0.78
+34
0,50
0.82
0.32 0,44
0.54
0.71
0.48
0.57
Weather
0.59
0.29 0.78 0.42
+34
0.54
0.06 0.10 0.08 0.90
0.86
0.59 0.70
Table I: RI, Snorkel, and juuply-)supervised learning results: Accuracy, precision, recall, and FI. F1-Gain shows the Fl score advantage of RL compared to Snorkel.
Reinforced labeling
Snorkel
All End model LFss
Dataset
RL. labels
LF cov.
I. ov.
LF con.
e
Ed
Sn labels
LF cov.
I.F ov.
L. con.
YouTube
1273
0.22
0.11
0.04
75 5.0
916
0.16
0.08
0.03
svm
Red Wine
375
0.12
0.02
0.02
125
0.5
247
0.08
0.02
0.01
RF
White Wine
3269
0.39
0.15
0.07
350
0.5
1995
0.21
0.04
0.01
NaivB
Weather
3415
0.58
0.56
0.49
200
2384
0.19
0.13
0.10
RF
Table 2: Labeling metries for the approaches and dataset statistics.
each data point, MV labels the point based the majority of the LF outputs (i.c., 0 or 1) from the LFs that do not abstain. We also experiment with MV combined with RL (MVRL). For all results, the existing data programming approaches Sn and MV use the same set of LFs as RL. The LFs are listed in Supp. material A.3 We implement Sn and MV as well as the RL. approach using the Snorkel library (version 0.9.5) with the existing features in the tutorial. The framework uses absolute latent labels for training end model (0,1), RL adapts the same scheme for the generative model and training end classifiers. Supervised learning leverages ground truth data (30% of the dataset) for the training. For the experimental scenario, supervised learning is considered the optimal result of machine learning, whereas the other two models are based on only programmatic labels. For supervised learning, the small testing dataset of YouTube comments is used for both training and testing due to lacking ground truth in the training set.
For the end classifier, different machine learning models are used for testing purposes. The models include two logistic regression models: the first one, namely "logit", the model that Snorkel uses by default (inverse of regularization strength C == 1000 and liblinear solver); the second one, "LogR", uses Ibfgs solver for optimization. addition, we test the random forest (RF), naive Bayes (NaivB), decision tree (DT), k-nearest neighbor (knn), support vector machine (svm), and multi-layer perceptron (mlp) end models. In each experiment, all approaches use the same end classifier. Each experiment consists of 5 runs, and the results averaged. We do not observe any notable difference between experiment runs. Our assumption is that the tested end models learning behavior rather deterministic they use the same the data points for their trainings. There are no additional hyperparameters used other than the stated in this section.
3.2 EXPERIMENTAL RESULTS
Benefits of RL The first set of results shows the advantage of using RL compared to the existing data programming approach Sn in terms of accuracy and F1 score gains. Table[[ results from the four datasets in terms of the four performance metrics. In all datasets, we observe substantial gains in accuracy and Fl scores compared the benchmark Sn approach. Moreover, RL performance is closer to fully-supervised learning, although it does not use any ground truth labels, even when LFs are relatively few. For the YouTube dataset, we experience that although the Snorkel and RL approaches have the same set of LFs, RL provides up to 64% FI, whereas Sn provides less than 3% F1. In RI, the end model can converge thanks to the additional coverage by the reinforced labels. Table 2 reports the number of computed labels after the application of the generative model with reinforcement (RL) and without it (Snorkel). The gains for accuracy and Fl require no additional human effort or involvement. Only two parameters& e and configure the RL (see Table 2 for Table experiments settings). Later in this section, we present how these parameters could be configured automatically for any given scenario based on labeling metrics (i.e., LF coverage, LF overlaps, and LF conflicts). Furthermore, our approach consistently outperforms Sn for training any of the tested end models. An extended table in Supp. material A.T reports the experimental results varying end

---

Under review as a conference paper at ICLR 2022
models for each dataset. Lastly, observe that Ed affects the performance as it adjusts the trade-off between augmenting similar labels and the bias (noise) of the augmentation.
Auto-adjustment of the RL method We observe that is dependent the dataset and the set of L.s.. A simple heuristic method to choose e is as follows: First calculate the distribution of aggregated effects of unlabeled data point for each LF (through boxplot in Fig. D. then set to have boundaries bneg and poos far enough from the quartiles Q1 and Q3. When using 2 such boundaries are symmetric to aggregate effect equal to 0 and they are chosen by the data scientist as we did for the results shown in Tables | Another way to calculate such boundaries is to have them symmetrical to the aggregated effect distribution. We can achieve this with the formula Dney = Q1 IQR # hIQR (and similarly for bpos) where InterQuartic (angarttieennge(((( and hIQR is a parameter, namely the IQR factor. When hIQR = 1.5, the gravitation method labels only outliers of the aggregated effect distribution. At this point, there is no need to set the parameter and the boundaries automatically adapts to the aggregated effects that depends on the features range, distance metrics and the sparsity of the initial labeling matrix. However, varying fIQR R (as tested for the range [0, 2 affects the performances. Thus, the problem of finding an optimal fiQn still persists.
= Q3 Q1
Fig. 5a top row shows the comparison with Snorkel generative model, while Fig. Sa middle row shows the comparison with MV. As in Fig. Sattop, h1Q8 <- 0.5 cause a substantial Fi gain for fewer LFs (e.g.. 5 to 8 LFs), whereas it may cause detrimental bias for the case of higher number of LFs as the sum of LFs' coverage increases. In this case, the reinforcement may not need to be applied as extensively when the sum of L.Fs' coverage is low. In the latter scenario, using hurr +- 1.0 would be the most conservative approach (no reinforcement) that makes sure to add no additional noise. Similar behavior observed with majority voter as label aggregation algorithm (Fig. middle row).
Fig. ba-bottom row shows the effect of RL with different HIQR in terms of labeled samples, LFs' coverage, mean overlaps, and mean conflicts. The smaller is IQR factor, the higher the values for those metrics are because the boundaries are closer to the IQR. One can infer that for higher number of LFs, smaller IQR factor results in excessive noise confusing the generative model (Sn or MV) and degrading the end model. Thus, we define a simple heuristic to automatically configure the gravitation method (shown in Alg. Supp. material A.3) through calculating hIQR by linking to the LF statistics for the given dataset. The below formula uses these statistics and empirical constant &
hIQ8 = + Σ coverage
Σ overlaps LF
Σ conflicts
LA
LA
We set E = 0.35 through empirical experiments. By this approach, the higher density the labeling matrix has (e.g., more LFs, higher LF coverage), the fewer the abstain labels updated by RL are.
We test the auto-adjin adjjstmettt approach and show the results in Fig. 5b for both Sn (top row) and MV (middle row). We have also set the second parameter Ea virtually to infinite, relying no hyper-parameter. When fewer LFs exist (e.g.. 5, 6, or 7), RI. provides significant performance gains, especially for F1 scores. On the other hand, when more L.ss exist, RL adjusts itself by becoming more conservative on the reinforcement. Therefore, auto-adjustment preserves the good performances when having sufficient LFs, leading to larger area under the curve for both Sn and MV. Our proposed RL approach, together with this heuristic, does not require any additional parameter setting or human effort by automatically adjusting its IQR factor. However, additional studies are due for more enhanced methods of adjustmnjt
Fig. 55 shows the effect of the automatic IQR adaptation in terms of labeled samples and LF metrics. As expected, when fewer LFs exist, RL provides gains for the number of labeled data points and mean LF coverage, overlaps, and conflicts. Furthermore, it adapts itself and gradually provides lower numbers of additional labels for higher numbers of LFs, thus reducing the noise.
Varying distance metrics Our gravitation method highly rely on similarities between data points. In the experiments described above, we always use the Euclidean distance for the real number features. Therefore, we investigate how different distance metrics affect the performances. Table B shows the results on the white wine dataset with RF as the end classifier and the previous heuristics to caleulate the aggregated effect boundaries with hIQR +- 1.4. RL consistently provides improved performance even when the distance metric changes.
7

---

Under review as a conference paper at ICLR 2022
accuracy
predsion
accuracy
1.00
1.00
0.75
0.50
0.50
0.25
0.05
0.00
1.00
0.75
0.30
0.23
0.25
0.00
number b:be led semp to
number Ps Becoerage
sumber Hts Hlover upu
number Iscorficts
0.00
numbar Hr labaled samples
10 number in Incovarage
number Inovariape
10 number Pr Heconfiists
1482
CA
1009
03
62
1931
63
G2
64
kunser
Pandect
Tog: comments classificution results by increasing LPs 12) with different IQR factoes. Botom: RL, effeets on the total number of labeled samples, and mean cover- age, overlaps, and confliets.
(b) Top: YouTube commeents classification results by increasing LFs (from
12) with auto-adjusted IQR factur.
Bottom: effects of RI. with nito-adjusted FOR ficter on the soul number of
samples
coverage, overlaps, and conflics.
Figure 5: (a) Effects of the hiQR by the number of LFs. (b) RL with the auto-adjusted hIQ.-
Reinforced LFs
Snorkel Prec
Distance metric
Acc
Prec
Rec
F1
F1-Gain
Acc
Rec
F1
Chebyshev
0.63
0.65
0.94
0.77
+20
0.53
0.71
0.48
0.57
Cosine
0.96
0.79
0.70
0.44
0.54
Euclidean
0.66
0.68
0.92
0.78
+23
0.51
0.70
0.46
0.55
Hamming
0.62
0.66
0.88
0.75
+16
0.52
0.68
0.52
0.59
Jaccard
0.60
0.67
0.79
0.71
+14
0,51
0.69
0.49
0.57
Mahalanobis
0.64
0.66
0.94
0.78
+24
0.51
0.70
0.44
0.54
Minkowski
0.57
0.67
0.49
0.57
Table Testing RL with various distance metrics for WhiteWine: RF end model, hIQR 1.4.
Limitations As label augmentation relies on the data features, using a poor embedding for distance computation might lead detrimental noise in the augmented labeling matrix. In addition, more advanced augmentation adjustment would avoid performance degradation due to the tradeoff between the coverage and noise.
4 RELATED WORK
Data programming [Ranner et al. [2016)] enables programmatically labeling data and training using WS sources. In particular, the Snorkel framework [Ratner et al. (2017); Bach et al. (2019)] provides interface for data programming where
can write I.Fs and apply them generate training dataset for their end models. The generation of the training dataset relies on the generative model,
and several studies focus on this aspect [Ratner et al. 2017): Bach et al. (2017): Varma al. (2017) ].
Various recent works focus on extending the existing data programming approach. The extensions include multi-task classification [Ratner et al.] (201)), using small labeled gold data for augmenting WS [Varma & Ré (2018)) learning tasks (or sub-tasks) in slices of dataset [Chen al. [2019)].
user guidance for LF precision [Chatterjec et al. (2020)), making the training process faster Fuu
al. [220 ], generative adversarial data programming Pal & Balasubramanian (2020) 2018 user supervision for LF error estimates [Arachie & Huang (211))) learning LF dependency structures Varma et al. [,019)], user annotation of LFs [Boecking et al. (2021)), language description of LFs
Hancock et al. (2018)), active learning Nashaat et al. (2018), and so Varma et al. (2016) aims to learn common features by using a "difference model and feeding these features back to generative model. Mallinar et al. (2019) takes advantage of the natural language processing query engine to expand gold labels and generate a label matrix as input for the generative model. Zhou et al. (2020) adopts soft L.Fs matcher approach based the distances between L.Fs' conditions and data points.
8

---

Under review as a conference paper at ICLR 2022
Chen et al. 2020) uses pre-trained machine learning models to estimate distances for natural language processing. The last two studies focus on the semantic similarity of texts to improve the labeling.
Although the studies mentioned above may improve the existing generative model (e.g.. through additional human supervision), they do not focus on the problem of LF abstraction with coarse information. Solving this problem would improve the validity of data programming in various scenarios especially since the human supervision limited in its nature. In this paper, we identify this problem and propose the reinforced labeling that takes the data features into account early on in the generative process. Using this approach, one can leverage the data features and augment the matrix for further generalization and producing satisfactory performance without additional human supervision.
Weak supervision approaches outside the context of data programming consider learning from set of rules, experts, or workers as in crowdsoureing. Plataniis et al. (2017) infer accuracy from multiple classifier outputs based on the confidences. (Safranchik et al. (2020) study the usage of Hidden Markov Models for tagging data sequences. Dehghani et al. (2018) train deep NNs using weakly-labeled data. Their approach is semi-supervised, where a teacher network based on rules adjusts the predictions of student network trained iteratively by given data samples. In another study, Dehghani et al. (2017) propose WS to train neural ranking models in natural language processing and computer vision tasks.
Takeoka et al. (2020) consider leveraging unsure responses of humaa annotators as WS sources to improve the traditional supervised approach. Kulkarni al. (2018) study labeling based consensus and interactive learning based on active labeling for multi- label image classification tasks. Khetan al. (2018) propose an appectationmmaximizatinn each worker represents a WS source for image classification tasks. Qian et al. (2020) propose WS with active learning for learning structured representations of entity names. Guan et al. (2018) learn individual's 's weights for predicting a weighted sum of noisy labelers (experts). Das et al. (2020)
algorithm for learning workers' quality, where
propose a domain-agnoss samples with each other. This approach uses small gold dataset with probabilistic distributions to infer probabilistic labels.
approach to replace the needs of LFs that apply affinity functions relate
Similar approaches other than WS include domain-specific machine learning applications such as ontology matching. DDan et al. (2004) use a relaxation method to label node into graph dataset by analyzing features of the node's s neighborhood in the graph. The relaxation process is based on constraint and knowledge that leads to the final labeling. In a similar approach, Li & Srikumar (2019) describe methodology framework to augment labels guided by external knowledge. In both approaches, label augmentation is in the final phase. These approaches do not involve any generative process. Lastly, literature related supe-supee learning (e.g., (Zhu et al. (2003)] or other hybrid approaches (e.g., [Awasthi et al. (2020)) consider using a mix of clean and noisy labels, whereas this paper focuses on using only the labels from LFs and improve the validity of the existing approach.
CONCLUSION
This paper proposes novel method for label augmentation in weak supervision. In the new machine learning pipeline, the proposed RL method in the generative process leverages existing LF outputs and data features augment the superv-- labels. The experimental evaluation shows the benefits of RL for four classification tasks compared to the existing data programming approach in terms of substantial accuracy and Fl gains. Furthermore, the new method enables the convergence of the end classifier even when there exist few LFs. We consider applying RL. for matching problems (e.g., entity matching) and active learning as future work. We consider RL as an initial approach for the identified limitation of the generative process, whereas the pipeline opens up the possibility for more advanced (e.g., machine learning) models to leverage data features during the generative process.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Túlio C. Alberto, Johannes V. Lochter, and Tiago A. Almeida. Tubespam: Comment spam filtering on youtube. In 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA), pp. 138-143, 2015.
Chidubem Arachie and Bert Huang. Adversarial label learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3183-3190, 2019.
Abhiject Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita sarawagi. Learning from rules generalizing labeled exemplars. arXiv preprint arXiv:2004.06025, 2020.
Stephen Bach, Bryan He, Alexander Ratner, and Christopher Ré. Learning the structure of generative models without labeled data. In International Conference Machine Learning, pp- 273-282. PMLR, 2017.
Stephen H Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alex Ratner, Braden Hancock, Houman Alborzi, et al. Snorkel drybell: A case study in deploying weak supervision industrial scale. In Proceedings of the 2019 International Conference on Management of Data, pp. 362-375, 2019.
Benedikt Boecking, Willie Neiswanger, Eric P. Xing, and Artur Dubrawski. Interactive weak supervision: Learning useful heuristics for data labeling. In Proceedings of (ICLR) International Conference on Learning Representations, May 2021.
Oishik Chatterjee, Ganesh Ramakrishnan, and Sunita Sarawagi. Robust data programming with precision-guided labeling functions. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pp. 3397-3404, 2020.
Mayee Chen, Daniel Fu, Frederic Sala, Sen Wu, Ravi Teja Mullapudi, Fait Poms, Kayvon Fatahalian, and Christopher Ré. Train and you'll ll miss it: Interactive model iteration with weak supervision and pre-trained embeddings. arXiv preprint arXiv:2006.15168 2020.
Vincent S Chen, Sen Wu. Zhenzhen Weng. Alexander Ratner, and Christopher Ré. Slice-based learning: programming model for residual learning in critical data slices. Advances in neural information processing systems, 32:9392, 2019.
Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decision support systems, 47(4): 547-553, 2009.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Nilaksh Das, Sanya Chaba, Renzhi Wu, Sakshi Gandhi, Duen Horng Chau, and Xu Chu. Goggles: Automatic image labeling with affinity coding. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, SIGMOD '20, pp. 1717-1732, 2020.
M. Dehghami, A. Mehrjou, Gouws, Kamps, and Schölkopf. Aeeeltty-eighhee learning. In 6th International Conference Learning Representations (ICLR), May 2018.
Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and Bruce Croft. Neural ranking models with weak supervision. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 65-74, 2017.
AnHai Doan, Jayant Madhavan, Pedro Domingos, and Alon Halevy. Ontology Matching: Machine Learning Approach, pp. 385 403. Springer Berlin Heidelberg, 2004.
Sara Evensen, Chang Ge, and Cagatay Demiralp. Ruler: Data programming by demonstration for document labeling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp. 1996-2005, 2020.
10

---

Under review as a conference paper at ICLR 2022
Daniel Fu, Mayee Chen, Frederic Sala, Sarah Hooper, Kayvon Fatahalian, and Christopher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods. In International Conference on Machine Learning. pp. 3280-3291, 2020.
Melody Guan, Varun Gulshan, Andrew Dai, and Geoffrey Hinton. Who said what: Modeling individual labelers improves classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Braden Hancock, Martin Bringmann, Paroma Varma, Percy Liang, Stephanie Wang, and Christopher Ré. Training classifiers with natural language explanations. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2018, pp. 1884. NIH Public Access, 2018.
Giannis Karamanolakis, Subhabrata Mukherjee, Guoqing Zheng, and Ahmed Hassan Awadallah. Self-training with weak supervision. arXiv preprint arXiv:210.005114 2021.
Ashish Khetan, Zachary C Lipton, and Animashree Anandkumar. Learning from noisy singly- labeled data. In International Conference on Learning Representations, 2018.
Ashish Kulkarni, Narasimha Raju Uppalapati, Pankaj Singh, and Ganesh Ramakrishnan. An interac- tive multi-label consensus labeling model for multiple labeler judgments. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Tao Li and Vivek Srikumar. Augmenting neural networks with first-order logic. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 292-302. Association for Computational Linguisties. July 2019.
Neil Mallinar, Abhishek Shah, Rajendra Ugrani, Ayush Gupta, Manikandan Gurusankar, Tin Kam Ho, Q Vera Liao, Yunfeng Zhang, Rachel K.E. Bellamy, Robert Yates, Chris Desmarais, and Blake McGregor. Bootstrapping conversational agents with weak supervision. Proceedings of the AAAI Conference Artificial Intelligence, (3(01))9528-9533 Jul. 2019.
Mona Nashaat, Aindrila Ghosh, James Miller, Shaikh Quader, Chad Marston, and Jean-Francois Puget. Hybridization of active learning and data programming for labeling large industrial datasets. In 2018 IEEE International Conference on Big Data (Big Data), pp. 46-55, IEEE, 2018.
Arghya Pal and Vineeth N Balasubramanian. Adversarial data programming: Using gans to relax the bottleneck of curated labeled data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1556-1565, 2018.
Arghya Pal and Vineeth N Balasubramanian. Generative adversarial data programming, arXiv preprint -XXv:2005500334 2020.
Emmanouil A Platanios, Hoifung Poon, Tom M Mitchell, and Eric Horvitz. Estimating accuracy from unlabeled data: probabilistic logic approach. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 4364-4373, 2017.
Kun Qian, Poornima Chozhiyath Raman, Yunyao Li, and Lucian Popa. Learning structured represen- tations of entity names using active learning and weak supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6376-6383, 2020.
Alexander Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data program- ming: Creating large training sets, quickly. Advances in neural information processing systems, 29:3567-3575, 2016.
Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. volume 11. pp. 269-282, November 2017.
Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederie Sala, Shreyash Pandey, and Christopher Ré. Training complex models with multi-tast weak supervision. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 4763 2019.
11

---

Under review as a conference paper at ICLR 2022
Wendi Ren, Yinghao Li, Hanting Su, David Kartchner, Cassie Mitchell, and Chao Zhang. Denoising multi-source weak supervision for neural text classification. arXiv preprint arXiv:2010.04582, 2020.
Esteban Safranchik, Shiying Luo, and Stephen Bach. Weakly supervised sequence tagging from noisy rules. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5570-5578, 2020.
Anastasiia Sedova, Andreas Stephan, Marina Speranskaya, and Benjamin Roth. Knodle: Modular weakly supervised learning with pytorch. arXiv preprini 210X:::2200411157220011.
Snorkel. Intro labeling functions, 2021. URL UtL.tttss///wo...ooll..ooo/r///s
01 se-spammtutoria
Kunihiro Takeoka, Yuyang Dong, and Masafumi Oyamada. Learning with unsure responses. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 230-237, 2020.
Toan Tran, Trung Pham, Gustavo Carneiro, Lyle Palmer, and Ian Reid. bayesian data augmentation approach for learning deep models. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pp. 2794-2803. 2017.
Paroma Varma and Christopher Ré. Snuba: automating weak supervision to label training data. In Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases, volume 12, pp. 223. NIH Public Access, 2018.
Paroma Varma, Bryan He, Dan Iter, Peng Xu, Rose Yu, Christopher De Sa, and Christopher Ré. Socratic learning: Augmenting generative models incorporate latent subsets in training data. arXiv preprint arXiv:1610.0812 2016.
Paroma Varma, Bryan He, Payal Bajaj, Imon Banerjee, Nishith Khandwala, Daniel L Rubin, and Christopher Ré. Inferring generative model structure with static analysis. Advances neural information processing systems, 30:239, 2017.
Paroma Varma, Frederic Sala, Ann He, Alexander Ratner, and Christopher Ré. Learning dependency structures for weak supervision models. In International Conference on Machine Learning, pp. 6418-6427, 2019.
Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Gao Huang, and Cheng Wu. Implicit semantic data augmentation for deep networks. Advances in Neural Information Processing Systems, 32: 12635-12644, 2019.
Wenxuan Zhou, Hongtao Lin, Bill Yuchen Lin, Ziqi Wang. Junyi Du, Leonardo Neves, and Xiang Ren. Nero: A neural rule grounding framework for label-efficient relation extraction. In Proceedings of The Web Conference 2020, WWW'200, 20, pp. 2166-2176 2020.
Xiaojin Zhu, Zoubin Ghahramani, and John D Lafferty. Semi-supervised learning using gaussian fields and harmonic functions. In Proceedings of the 20th International conference Machine learning (ICML-03), pp. 912-919, 2003.
12

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
The results shown in the paper and the appendix can be reproduced as described below.
All of the four datasets openly available and popularly used by many machine learning researchers. The open-source Snorkel framework tutorial [Snorkel (2021)] (version 0.9.5) can be run as it is, as the existing data programming approach. In the experiments the default features in these tutorial is used without any change, whereas we implemented new LFs for the new datasets and include all of these new LFs Appendix [A.5] The algorithm implemented for reinforcement of labels and IQR calculation are included in detail in Appendix as Alg. [ and Alg. We plan to make the codes for these two algorithms open-source after our internal approval process.
A person skilled in programming (e.g.. Python) can implement the two algorithms in short time given the pseudocodes. The only two parameters and E in Table I used in the four experiments are listed in Table 2 and four distance metrics, instead of these two parameters, only one parameter value h LIQR == (for Alg. 2 is used as shown Table
13

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1
REINFORCED LABELING RESULTS ON DIFFERENT DATASETS VARYING END MODELS
Table + extends Table by reporting the results of 8 end models and 4 datasets. The hyperparameter configuratio configurations of and Ed are listed in T For all experiments and end models RL outperforms Snorkel and maintains closer performance to the sullyy-upeerised machine learning.
Reinforced LFss
Snorkel
Supervised learning
Dataset
Acc
Pree
Roc
F1
FI-Gain
Acc
Prec
Rec
Acc
Pree
Ree
F1
model
YouTube
0.75
0.64
0.54
1.00
0.02
0.03
0.91
0.90
YouTube
0.72
1.00
0.40
0.57
+55
0153
1.00
0.01
0.02
0.81
1,00
0.61
0.76
LogR
YouTube
0.68
0,98
0,34
0.50
+50
0.53
0.00
0.00
0.00
0.92
0,92
0.90
0.91
DT
YouTube
0.72
1.00
0.41
0.58
+56
0.53
1.00
0.01
0.02
0.91
1.00
0.82
0.90
logit
YouTube
0.67
0.93
0.33
0.49
+44
0.53
0.50
0.03
0.05
0.76
1.00
0.53
0.69
knn
YouTube
0.69
0.00
0.00
0.86
RF
Red Wine
0.71
0.82
0.62
0.70
0.70
0.80
0.64
0.69
0.74
0.82
0.70
0.74
Red Wine
0.68
0.85
0.53
0.64
+3
0.68
0.85
0.55
0.61
0.73
0,80
0.71
0.74
LogR
Red Wine
0.69
0.69
0.60
0.65
0.78
0.65
0.65
DT
Red Wine
0.71
0.79
0.68
0.72
+2
0.68
0.73
0.74
0,70
0.73
0.82
0.69
0.73
logit
Red Wine
0.71
0.61
0.70
0.68
0.80
0.59
0.65
0.75
0,75
0.77
knn
Red Wine
0.69
0.69
0.81
0.74
0
0.66
0.63
0.89
0.74
0.71
0.77
0.67
0.71
NaivB
Red Wine
0.71
0.80
0.66
0.72
+7
0.61
0.66
0.77
0.65
0.75
0,81
0,74
0.76
RF
Wine
0.70
0.70
0.83
0.61
0.68
0.74
mip
White Wine
0.65
0.65
1,00
0.79
+23
0.51
0.67
0.48
0.56
0.72
0.73
0.91
0.81
White Wine
0.65
0.65
1.00
0.79
+22
0,52
0.68
0.51
0.57
0.72
0,74
0.90
0.81
LogR
+26
0.69
0.41
IYT
White Wine
0.65
0.66
0.99
0.79
+23
0,50
0.67
0.49
0,56
0.67
0,86
0.62
0.69
logit
White Wine
0.64
0.99
0.78
0.59
0.67
0.73
0.70
0.65
0.75
0,70
0.72
knn
White Wine
0.63
0.64
0.78
+34
0.50
0.82
0.32
0.44
0.54
0.57
NaivB
White Wine
0.64
0.65
0.99
0.78
+27
0.50
0.69
0.41
0.51
0.71
0.81
0.73
0.76
RF
White Wine
0.66
0.54
0.68
0.56
0.61
0.71
mlp
Australia Rain
0.58
0.28
0.77
0.41
+31
01.46
0.07
0.15
0.10
0.86
0.75
0.40
0.52
Australia Rain
0.58
0,28
0.76
0.41
+32
0.49
0.07
0.13
0.09
0.86
0.73
0.40
0.52
LogR
Australia Rain
0.55
0.12
DT
Australia Rain
0.59
0.29
0.76
0.42
+32
0.44
0.07
0.16
0.10
0.87
0.74
0.53
0.61
logit
Australia Rain
0.49
0.25
0.81
0.38
+29
0.53
0.07
0.12
0.09
0.82
0.54
0.55
0.55
knn
Australia Rain
0.54
0.26
0.75
0.39
+27
0.50
0.09
0.18
0.12
0.69
0.34
0.65
0.45
NaivB
Australia Rain
0.59
0,29
0.78
0.42
+34
0.54
0.06
0.10
0.08
0.90
0.86
0.59
0.70
RF
Australia Rain
0.57
+33
0.06
0.08
0.87
mip
Table 4: RL, Snorkel, and (fully-)supervised model results: Accuracy, recall, precision and Fl scores. F1-Gain shows the Fl score advantage of RL compared to Snorkel.
14

---

Under review as a conference paper at ICLR 2022
A.2 SYMBOLS AND NOTATIONS THE PAPER
Table 5 lists and describes the frequently used symbols throughout the paper. Some listed parameters normalized (e.g., aggregated effects) adjusted for simplicity, whereas they can be easily normalized to the range [0, 1]. and so E [0, 1] based on the observed values, without causing any change the outcomes.
Symbol
Description
0
{2]
(8)
Data point compused of features
fr
Data feature
X
(o(1) ,212) .(.
Datasel
Y 10,11
Dataset coverage
== IX|
Number of data points
LR
Laheling function with index /
Output of LFj on data point g(0) Possible outcomes are classes or 1, or abstain -1
LF((()) (0,1,-1
(LF) (LF((z()). (F(())))
Output of labeling function LF) applied on the whole dataset Effect function of data points f(( and ااار and output of LF; data point (+)
Eff(2(0 2Ff(G(),LFi((())) ),
Σi Eff(x!), (z)
Aggregated effect on point . for LFj Distance (e.g.. Euclidean) between two data points
Distance( (6), x(11)
Fd a=
Cut off distance for effect to be considered Constants
Thresholds on aggregated effected to augment label as negative or positive
Dneg: Opos
E ppn = -uug
Symmetric aggregated effect threshold
Q1. 22 Q3
Quartiles. 25th, 50th and 75th percentile respectively. InterQuartile Range
JQR = Q3 Q1
hIQR
IQR factor. hIQR = 1.5 is used calculate the outliers range Statistics of the I.Fi also in relations with all the other LF,,
coverager, overlapst, flictss
Table 5: Frequently used symbols
15

---

Under review as a conference paper at ICLR 2022
A.3 REINFORCED LABELING PSEUDOCODE FOR LABEL AUGMENTATION
Algorithm 1: REINFORCED LABELING ALGORITHM
Input: LFs (LF1, IF22.... LF m) ) and unlabeled data points X (ax(1) _x(22 x (k)), where
(0) has features (ar (1) (6) In (1)), Gravity parameters a, 3. Distance threshold Ed.
.2 ....Ih!!
IQR adjusting parameter {
Output: Label for a subset 7 |X] of the data points (augmented labels)
Y [y(), y ..... (*)), where y(t) E | u (-1}
Dislances
Effects +-0 0
Labels t
for i ..2,..., [X| do
for 1........ m do
Labels [1] + LF(x Effects - 0
end end
LF's ss_atss + CalculateLFsStaaaae Labels) 10
for 1 = 1,2..... do
for 1,2, ..... X| do
12
if Labels [i]
then
14
for j = 1,2, 2, |X|
if # j & Labels -1 then
16
Distance + 00
if Distances []] is undefined then
Distance < Dew.asstttteeeeeeeeeenaaaaaaaa
x(1)
Distances [ + Distance
19
Distances j
Distance
if Distances [ < then
22
if Labels [
then
fects [4] + fffectt(( (Distances
23
else
Effects + Ef fectss (Distanccs|s][))>
8
26
end
27
end
28
end
end
30
bowndary_nmoe min boundary_max + Cadcalaee(HHoommmarssssA.. _tatss, ffffect, {)
for [
do
322
for i 1.2, .........] do
X
33
if Labels
then
if E ficts(i] < mundary_min then
Labels
0
if Effects
boundary_mas then
37
Labels 14
38
end
end
40 return Labels
Alg. shows the pseudocode of similarity- based heuristic algorithm for reinforced labeling. Given m LFs and the unlabeled dataset size X the algorithm outputs the augmented labeling matrix. The listed gravity parameters a := 1, 3 experimental study. The distance threshold e, is an optional parameter to optimize the computation or remove outliers, whereas not always used in the experimental study (Ea is set to a high number). Labels, F.f fects 2D arrays, where represent the index of the data points (x))))
constants used for all datasets in the
16

---

Under review as a conference paper at ICLR 2022
Algorithm 2: Cadeuldee[QRoomddiieeee Saaisssstttss_t fectss
Input: LFs_stats 1L.Ps_coeeroge_ L.FFsovverlaps Pers_conflcchee where LFs 1.Ps_coverage = _LF _eterrgge I.F2_ htra_oooerree..... LF coverage) and similarly LF' LF'sooverlaps and LF's LFes_aurr E fects: Array of aggregated effects,
< 0.35
Output: boundary bowndorynnmaat man
suMcoverage
oueover f-
con flicts +-
for | 1,2,.... m do
coTerree + ouveooee + _Fi____oo coverage
sumoourraps suMoverlaps + ovvelapss
cumfcffficct sumcon + LF, _Fi_conflicts
end
higr sutlleoerrage: Stioowerlaps Stifllofflictss
10 & ealculatePercentilee fects, 25)
11 Q3 & calculate!Pereentice E fects, 75)
12 igr + 93
13 boundary_min < 91 iqr * higr
boundary_no y_max t 93 + iqr hige
15 return doundary_min, boundary canndary_macs max
and columns represent the index / of the LFs (LFi). Distances is 2D array representing the distance between any two data points. For instance, Distances [i]]]]] represents the distance between r. and (5))
The 1 values that represent abstains of LFs updated based their similarity given by the FindDistanceBeture DatareeDDaPoe DataPoints function. This function can be implemented using various distance metrics such Euclidean, Cosine, Mahalanobis distances. Lastly, the updated Labels array represents the augmented labeling matrix that can be given to generative process such as Snorkel's generative model.
Alg. 2 shows the pseudocode of the C The heuristic in Alg. lleerreess three LF statistics, that are LF coverage, overlaps, and conflicts, to calculate the boundaries that are effectively the aggregated effect threshold of RI..
used for the RL implementation.
17

---

Under review as a conference paper at ICLR 2022
A.4 ADDITIONAL EXPERIMENTAL INSIGHTS
In addition to the experimental study, we implement and test hybrid approach of leveraging labels of LFs as well as strong supervision through a gold dataset. We call this approach the generative neural network for data programming (GNN). In GNN, the label outputs of LFs and data features fed to simple neural network (NN) along with the labels. The NN model contains two hidden layers (# nodes 12, 8) with ReLU activation function, and it uses Adam optimizer. The output layer has the sigmoid activation function. The NN model is used in different stages of the pipeline. First, GNN replaces the generative part using labels of Snorkel Then, the outputs of GNN are fed to an end classifier as usual. Second, the GNN itself serves as the end classifier well as the generative model (Sn+GNN RL+GNN). We applied the GNN model to both red wine and white wine datasets as these datasets have the available ground truth data to create gold datasets.
Noo+GNN++<eddoddee
or RL (RL+GNN+<<endmdell)
White Wine
appraach
Approacs
Sn=
SneNaive Naiv8
08-
RL
RL*NaIv8
RFN
Sn GIN- NaivB
RL-GNN-RF
RL+GNN- NaivB
RL+GNN
accuracy
precision
rerail
11
precision
Figure 6: Experimental results of the red wine and white wine datasets for different approaches: Sn,
RL, Sup., Sn+GNN, RL+GNN, Sn+GNN+<end _odddldmodeleeeee and RL+GNN+<<nd_mmode
Fig. [6 shows results of the red and white wine datasets in more detail, including the hybrid GNN approach using RF and NaivB end models, respectively. The results of the following 7 approaches are listed in order: Sendnnnddd,,>, RL+ RL+<eenmodel>, Sup+<endmodel>, RL+GNN+<<ndmmode Sn+GNN, RL+GNN> We observe that RL+RF outperforms the Sn+RF benchmark for the white wine dataset by +13 points accuracy and +34 points F1 and even provides better results than Sup (RF). For the red wine dataset, RL+NaivB outperforms Sn+NaivB by +10 points in accuracy and +7 points in F1 score. Moreover, although approaches such Sup GNN leverage ground truth labels their training, outcomes RL are competitive for the red wine dataset, whereas RL outperforms Sup (NaivB), Sn+GNN+NaivB, and RL+GNN+NaivB for the white wine dataset (see Fig. 6 right).
<entmmddell..
You Tube
Australia Rain
Sn+svm
RL+svm
0.0
RL+RI
Sup+
Sup
0.2
0.2
accuracy
precision
recall
f1
accuracy
precision
Figure 7: Experimental results for the YouTube comments and Australia rain datasets testing different approaches: Snorkel (Sn), Reinforced Labeling (RL), Supervised learning (Sup.).
Fig. shows the bar graph for the results of the YouTube comments and Australia Rain datasets (also listed Table in terms of the four metrics: Accuracy, precision, recall, and F1. The results of the following approaches are listed order: Sn+<<endodelz> RL+ <endmodel>> Sup+ <endmodel>. We observe that RL+svm outperforms the Sn+svm benchmark for the YouTube dataset +21 points
18

---

Under review as a conference paper at ICLR 2022
accuracy and +61 points F1. For the Australia Rain dataset, RL+RF outperforms Sn+RF by +5 points in accuracy and +34 points in Fl score.
19

---

Under review as a conference paper at ICLR 2022
A.5 LABELING FUNCTIONS
We use the Snorkel library to implement LFs and encode domain knowledge programmatically The LFs are implemented using the interactive and user-friendly features of the Snorkel framework, such as providing LF statistics and allowing high-level definitions of LFs.
As in almost all the previous data programming studies, we do not follow a certain scheme or strict guidance on implementing LFs but rather rely on the best effort based on some understanding of the datasets by visualizing the features and interactively checking the LF coverages, overlaps, and conflicts. In general, the assumption is that the developers would make the best effort to write LFs.
The Australia Weather dataset is used to build model that predicts for any given day if it will rain the day after LFs in Listing use weather data features to label data points as GOOD (1) or BAD (0) based on the features such as Humidity, Rain Today, Temperature at 9am, Humidity 9am, and pressure at 3pm.
from snorkel. labeling import labeling function inbeeling_uunction ABSTIIN -1
NO_RAIN = 0
RAIN 1
@label fnn_
def Sheck_humidityypp (x)
if x.Humidity3pm is None:
ABSTAIN
10
elif x. S.Huuidity3pm>00.33
11
return RAIN
12
elif x. H.miiditt3pmmc 15:
return WO_RAIN
else:
15
return ABSTAIN
16
def check rricckrriinttee check_rain_todayyeee (x):
19
it X.RainToday is None:
20
return ABSTAIN
21
elif x. Ra
return RAIN else:
23
24
return ABSTAIN
25
26 [
def chock_temp9an (x)
28
if x. Tempaam is None:
29
return ABSTAIN elif x. Temmp9am>00000
30
return RAIN
32
else:
33
return ABSTAIN
34
35
Blabeling function() ()
def check_rainfall (x):
37
if X.Rainfall None:
38
return ABSTAIN elif x. x.Rainfall>0.600
39
return RAIN
else: return ABSTAIN
42
43
44 Blabeling fomeiiinnnunniionii
def check_huml (x)
if Humi dimy is None:
47
return ABSTAIN
20

---

Under review as a conference paper at ICLR 2022
elif H.m.diityyamo..999 return NO_RAIN
49
50
clif x. x.Humidity9amo0.cc. 20:
51
return RAIN else:
ABSTAIN
54
55
someeling_fnnttinn
56
def pprcck_preesureppmccc
it x.Pressure3pm is None:
return ABSTAIN
59
elif s.Pressure3pm<o..D0
60
return RAIN
elif x. S.Pressure3pmo0.700 70: return NO RAIN
63
else:
64
rcturn ABSTAIN
Listing 1: Australia rain labeling functions
The wine datasets have various data features such as alcohol, sulfates, citric acid levels, etc. Listing 2 include the LFs implemented for wine quality classification for the red wine and white wine datasets, respectively. LFs labels data points as GOOD (1) or BAD (0) quality wine.
From snorkel. labeling import labe 1 functioo
ABSTAIN
BAD 0 GOOD =
coabeliin_functionn
def check_alcohel (x)
if x.alcchol is None:
return ABSTAIN
elif x.alcchol>00.1
GOOD
12
elif x.alc <0, 15:
13
return BAD
else:
ABSTAIN
16
17 @labeling_ ()
18 def chock_sulphate (x)
if x.sulphates is None:
ABSTAIN
21
elif x.sulphates>0.33
22
return GOOD
23
else:
return ABSTAIN
36
27
def check_citric (8):
if x.acidity_eciieeicc citric
-ettittyeetiiieeeiieee is None:
elif ABSTAIN
30
31
return GOOD
32
clse:
return ABSTAIN
Listing 2: Red wine labeling functions
from snorkel, labeling import label functttoooooonnnn
ABSTAIN -1
BAD 0
30OD -
21

---

Under review as a conference paper at ICLR 2022
@labeling [labeling_function
def check_alcohol (x)
if x.alcchol is None:
return ABSTAIN
elif x.alcchol>0..5.
return GOOD
12
elif x.alcohol<0. 15:
13
return BAD
14
eise:
return ABSTAIN
17 @label fabelinggfuuntton ()
18
def check_sulphate (x)
if s.sulhates is None:
ABSTAIN
21
elif s.uulpattss>0.33
22
return GOOD
23
clse:
return ABSTAIN
26
enabelingg.uunttion ()
27
def check_citric (x):
28
it w.acidity_citres is None:
return ABSTAIN
30
elit w.aciiitt_eitriiiiiee.. citric>0.77
31
return 600D
32
else:
return ABSTAIN
Listing White wine labeling functions
For the YouTube dataset, we implemented two LFs that search for the exact string "check our" or "check" in the text. Other than these two additional LFs, the LFs in the Snorkel tutorial [Snorkel (2021)] named "textblob subiiotiitteeiiitttt... "keyword subscribee" "has _eas_person porson_nlp" are used in the experiments of T The rest of LFs available from the Snorkel tutorial are used for experiments in Fig.5
From snorkel. labeling import label ing_functtin
ABSTAIN
HAM
SPAM =
Blabeling iobbbeiiigfunntiinn dof chock (x);
return SPAM if "check" in v.text.lower. else ABSTAIN
(labeling_functionn)
11 def check_out (x):
12
return SPAM is "check out" in x. texett.. Lower () else ABSTAIN Listing 4: YouTube labeling functions
22


Reviews:
[{"review_id": "yakm2zAVe82", "reviewer": "Reviewer_HUob", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper discusses an interesting method and is well-written and clear to follow. However, there appear to be major problems with the evaluation, and the method is not novel.\n\nEvaluation:\nThe authors only compare to one baseline, Snorkel, and report significantly worse performance on benchmark datasets than reported elsewhere in the literature. Consider the YouTube dataset -- this is a standard weak supervision dataset and is used as a tutorial for the open-source Snorkel library.\n\nIn the tutorial itself (https://www.snorkel.org/use-cases/01-spam-tutorial), Snorkel achieves an accuracy of 94.4%. However, the authors report an accuracy of 54%, with an F1 of 3% in Table 1. This suggests that the authors did not run Snorkel correctly. In fact, the authors' reported performance on the YouTube dataset (75% accuracy) significantly underperforms the tutorial notebook. This discrepancy is because the authors did not use all the labeling functions in the YouTube dataset. This is a non-standard evaluation procedure, and makes the results hard to compare and believe.\n\nIn future iterations, the authors should also consider comparing against the baselines in the WRENCH benchmark (https://arxiv.org/pdf/2109.11377.pdf) as well, to ensure that their evaluation is fair and standard. (This paper was released <1 month before submission, so I don't count it against them -- but it's a useful datapoint nonetheless).\n\nI am less familiar with the wine and weather datasets, since I have not seen them in other WS datasets, but performance is suspiciously low for those as well (F1 of 8% for Snorkel on Weather, which suggests that Snorkel was not tuned correctly).\n\nNovelty:\nThe method itself is almost identical to that in Chen et al 2020 (https://arxiv.org/abs/2006.15168). The authors cite this paper but do not compare against it. Chen et al also focus on improving labeling function coverage by replacing abstentions based on similarity in feature space to other points. Both methods feature a hard threshold after which they no longer transfer labeling function labels. The one new point is that this paper discusses a method for automatically adjusting the threshold (section 3.2), whereas Chen et al tuned the thresholds manually. I suggest anchoring more strongly on this contribution in the future.", "summary_of_the_paper": "This paper describes a method to improve the coverage of labeling functions in weak supervision. Labeling functions in weak supervision can abstain if they do not vote on a particular point. The method replaces abstentions with a gravitation-based method based on the other points that the labeling function has voted on. The method uses similarities between features of points for the gravitation-based method. The authors claim performance benefits over Snorkel.", "summary_of_the_review": "This paper is clearly-written, but it has significant flaws in its evaluation, and the method is only marginally novel. Therefore I recommend reject.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "yZqGU3Matli", "reviewer": "Reviewer_5fTq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- This paper focuses on a practical problem. The motivation is good. \n- The experimental results are promising. In almost all cases, we can see that the improvement on F1-Score is clear. \n\n**Weaknesses/Questions**\n- The organization of this paper is a bit confused. I understand the core idea and algorithm flow of this paper after I read it a few times. \n- The writing is a bit illogical and makes readers confused. For example, this paper tends to augment the outputs of labeling functions for better generalization. Perhaps, it is expected to see that the intuition about why the proposed method works and how it handles the issues. However, the explanation is a bit incomprehensible. \n- Figures 2 and 3 are not informative. Also, the explanations are not sufficient. \n- In Equation 1, there are some hyper-parameters. How to control them in experiments?\n- I am confused about the auto-adjustment of $\\epsilon$, which is dependent on the dataset. However, it seems that we still need to determine $\\xi$ artificially to control $\\epsilon$?\n", "summary_of_the_paper": "This paper tackles the problem of designing data programming, which is a practical approach in weak supervision. The authors state that prior effects neglect to utilize data features during the generative process, and therefore have suboptimal performance. Based on this, this paper proposes to exploit the sample similarities to augment the outputs of labeling functions. Empirical evaluations are provided to verify the effectiveness of the proposed method. ", "summary_of_the_review": "This paper tackles a practical problem. The idea is relatively novel. However, the writing of this paper needs to be improved to reach the acceptance line. The current version is hard to follow. Therefore, my score for this paper is negative. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "obEdTHFwix3", "reviewer": "Reviewer_s9uq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- (S1) The proposed approach tackles a relevant area of the space with appealing top-level motivation and intuition.\n\nWeaknesses:\n- (W1) Lack of adequate comparisons to related work: There are many weak supervision approaches that have been proposed since the original Snorkel paper referenced and compared to in the experimental results; some of which are referenced in the paper, but never actually compared to.  These more recent papers address many of the proposed issues- incorporating data features, boosting generalization with joint learning, etc.  Given the motivation of the paper, and claims in the paper that the proposed approach is better than these other approaches cited, it seems to be a major gap that no comparisons are made to these other methods in the experiments.\n- (W1a) Additionally, this work is very similar to classic label propagation techniques in semi-supervised learning, and should have been compared to these as well.\n\n- (W2) Cherry-picked experimental results: There are several aspects of the experiments that raise significant questions as to the protocol with which they were conducted, and the representativeness of the experimental configurations; for example:\n  * (W2a) In the main results table (Table 1), Snorkel is reported as having an average F1 of 0.03 on the YouTube dataset (this is a score so low as to be actively difficult to achieve in most settings!).  The YouTube dataset, as noted in the paper and appendix explicitly, is *from the Snorkel OSS intro tutorial, in which Snorkel achieves a 94+% accuracy* (see snorkel.org/use-cases/01-spam-tutorial).  Yet somehow, the results here are reported as 0.03 F1; according to the appendix, because of an arbitrary subselection of LFs.  In fact, in Fig.5(a) you can see that with more LFs, the score goes up significantly and either matches or outperforms the proposed approach (see next point for more here).  It therefore seems extremely misleading to have presented the scores as presented in Table 1 as main results.  If the objective was to show that in certain low-LF situations, the proposed approach can offer benefits, this should have been messaged clearly upfront when reporting main results.\n  * (W2b) The proposed approach is (like most/any clustering-based approaches) highly dependent on a threshold parameter, the so called \"IQR factor\".  In the main results, this is apparently \"chosen by a data scientist\".  This raises significant questions as to experimental protocol\n \n- (W3) Lack of sufficient ablation and/or formal motivation for proposed approach: The proposed approach is presented in a way that makes it seem fairly arbitrary compared to existing approaches for e.g. weak supervision, clustering, label propagation, etc.  However, minimal ablations of the approach are performed (just the threshold factor and the distance metric) to actually motivate or defend the heuristic approach taken.  This makes it hard to gain insight from the paper about why this approach should work and why it should work better than other approaches- esp. given the lack of other methods compared to.", "summary_of_the_paper": "This paper proposes an extension to the data programming / weak supervision method of Ratner et. al. 2016, in which heuristic functions called labeling functions (LF) are used to label training data.  In the proposed extension, called \"reinforced labeling\", points that are close to those labeled by a labeling function in feature space (as determined by a heuristic \"gravitation\" algorithm) are also labeled, thereby augmenting the set of programmatically labeled examples.", "summary_of_the_review": "This paper lacks sufficient comparison to other more contemporary weak supervision work, lacks sufficient motivation and/or ablation w.r.t. the heuristic design choices taken, and has potential issues with experimental protocol.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Bpli8yCLZaU", "reviewer": "Reviewer_eeud", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1.\tThe research problem of weak supervision is very important and worth studying.\n2.\tThe proposed approach for data programming can improve the performance of weak supervision.\n3.\tThe proposed approach alleviated two limitations of existing data programming methods: (1) coarse information, and (2) lack of generalization compared to existing Snorkel framework by taking data features into considerations.\nWeakness/concerns:\n1.\tThe technical novelty of this work is limited as the proposed reinforced labeling is similar to KNN with pseudo-labels. How these two methods differ in underlying motivation, method design, and experiments is not properly explored.\n2.\tThe term ‘generative’ would seriously confuse readers as it was not referring to data generation but to label generation, it would be better if the authors could use ‘label generative’ instead of ‘generative’.\n3.\tThe experiments are not convincing. What is the supervised learning method the authors used to compete with their weak supervision method? Is there any improvement for the supervision performance if the authors incorporate their weak supervision approach (or pseudo-labels) into a supervised setting?\n4.\tThe labeling functions seems to be rule-based functions that maps the features to the pseudo labels. I’m not sure how this can bring in new information to the downstream classifier when it is powerful enough. How do you avoid the error propagation problem? Also, if the pseudo labels are good enough, why do you need a downstream classifier as the pseudo labels are already the upper bound of the classifier?\n", "summary_of_the_paper": "The authors proposed a new approach called reinforced labeling to enhance the data programming weak-supervision approach. Labeling functions in data programming can only produce pseudo-labels for part of the dataset, and thus the authors propose to use similarity between two data features to find the lacking pseudo-labels, and thus enhance the existing data programming methods.", "summary_of_the_review": "The work may be interesting for the automatic label annotation area, but the idea novelty is a little trivial and experimental results are not convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an approach to weak supervision to address the possibly low-coverage of rule-based labeling functions, by assigning similar labels to similar instances (where the similarity is computed in feature space).

The reviewers main concerns were the presentation, as well as the experimental protocol and results. Several directions for improvement have been identified by the reviewers and acknowledged by the authors, but in the current state of the submission the consensus is that the paper is not ready for publication.
Decision: Reject