ACTIVE REFINEMENT OF WEAKLY SUPERVISED MODELS
Authors: Mononito Goswami
Chufan Gao
Benedikt Boecking
Saswati Ray
Artur Dubrawski
Venue: ICLR 2022 Submitted (2022)
Keywords: Weak Supervision
Active Learning
Fuzzy logic
AI in Healthcare

Abstract: Supervised machine learning (ML) has fueled major advances in several domains such as health, education and governance. However, most modern ML methods rely on vast quantities of point-by-point hand-labeled training data. In domains such as clinical research, where data collection and its careful characterization is particularly expensive and tedious, this reliance on pointillisticaly labeled data is one of the biggest roadblocks to the adoption of modern data-hungry ML algorithms. Data programming, a framework for learning from weak supervision, attempts to overcome this bottleneck by generating probabilistic training labels from simple yet imperfect heuristics obtained a priori from domain experts. We present WARM, Active Refinement of Weakly Supervised Models, a principled approach to iterative and interactive improvement of weakly supervised models via active learning. WARM directs domain experts' attention on a few selected data points that, when annotated, would improve the label model's probabilistic output in terms of accuracy the most. Gradient backpropagation is then used to iteratively update decision parameters of the heuristics of the label model. Experiments on multiple real-world medical classification datasets reveal that WARM can substantially improve the accuracy of probabilistic labels, a direct measure of training data quality, with as few as 30 queries to clinicians. Additional experiments with domain shift and artificial noise in the LFs, demonstrate WARM's ability to adapt heuristics and the end model to changing population characteristics as well as its robustness to mis-specification of domain-expert-acquired LFs. These capabilities make WARM a potentially useful tool for deploying, maintaining, and auditing weakly supervised systems in practice.

Under review as a conference paper at ICLR 2022
ACTIVE REFINEMENT OF WEAKLY SUPERVISED MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Supervised machine learning (ML) has fueled major advances in several domains such as health, education and governance. However, most modern ML methods rely on vast quantities of point-by-pon such as clinical research, where data collection and its careful characterization is particularly expensive and tedious, this reliance on pointillisticlly labeled data is one of the biggest roadblocks to the adoption of modern data-hungry ML al- gorithms. Data programming. a framework for learning from weak supervision, attempts overcome this bottleneck by generating probabilistic training labels from simple yet imperfect heuristics (or labelling functions) obtained a priori from domain experts. We present WARM, Active Refinement of Weakly Super- vised Models, a principled approach to iterative and interactive improvement weakly supervised models via active learning. WARM directs domain experts' at- tention on a few selected data points that, when annotated, would most improve the label model's probabilistic accuracy. Gradient updates then backpropa- gated to iteratively update the parameters of the individual expert labelling fune- tions in the weak supervision model. Experiments on multiple real-world medical classification datasets reveal that WARM can substantially improve the accuracy of probabilistic labels used to train downstream classifiers, with as few as 30 queries experts. Additional experiments with domain shift and artificial noise demon- strate WARM's ability to adapt to changing population characteristics as well as noisy initial labelling functions from the experts. These capabilities make WARM potentially useful tool for deploying, maintaining, and auditing weakly supervised systems in practice.
hand-labeled training data. domains
INTRODUCTION
Machine learning (ML) has seen widespread adoption several domains such as health, education and public policy. In the clinical context, supervised ML algorithms have been shown to be effective solutions to a gamut of problems, ranging from detecting abnormal heart rhythms in electrocar- diogram (Goswami al. 2022 Hannun et al., 2019) prognosticating neurological recovery using continuous electroencephalorrat (Elmer et al. 2016 2020) to detecting lung cancer in CT scans (Gao et al. 2019). However, most modern ML algorithms, especially large deep neural networks, rely on vast quantities of pointillistcally labeled training data. In our experience, while raw clinical data is abundant, its careful annotation not. Manually labeling large quantities of clinical data is often tedious, prohibitively expensive, prone to error, and acutely unscalable.
Recently, many studies have explored the use of cheap, potentially noisy sources of supervision to noisily annotate large training sets (Goswami et al.) 2022 Fries et al., 2019: Saab et al., 2020) using weak supervision. These methods combine these sources create noisy labels on unseen data, which can then be used to learn powerful discriminative end models. In principle, these methods resemble other repeated-labell supervision relying on external knowledge bases (De Sa et al. (016). In this paper, we leverage data programming (Ratner et al. 2016) which uses generative model to integrate expert knowledge expressed in the form of noisy heuristics (labeling functions or LFs) to probabilistically label training data. Since the quality of the training labels is strongly correlated with that of labeling functions, unrefined weak supervision may not always yield optimal performance. Hence, in practice, experts
techniques such as crowd-sourcing (Li et al., 2013), or distant

---

Under review as a conference paper at ICLR 2022
using data programming often spend several hours iteratively designing new heuristics or refining existing ones (Varma& & Ré 2018). Our work aims to streamline and reduce this effort.
Another closely related issue is that of domain shift (also referred to as knowledge shift). Imbalance in characteristics of source and target populations may render ML models trained on one domain perform poorly on another. This domain shift, especially in the form of population characteristics like gender or race, is prevalent in clinical settings (Thiagarajan et al. 2018). In the context of weak supervision, we are interested in the related problem of knowledge shift. Specifically, we want to train effective ML classifiers while tailoring knowledge of the source population to the target population (e.g. heart disease in young and old patients) via small changes in the LFs.
To this end. we propose WARM (Active Refinement of Weakly Supervised Models), a principled approach towards quick, iterative, and interactive refinement of weakly supervised model: WARM directs expert attention to specific data points in the training set, which if labeled, would improve the labeling functions and the resulting label model the most. Specifically, at each iteration WARM directs expert to label the most uncertain data point, and then updates decision parameters of the labeling functions to reduce the uncertainty.
Experiments on multiple real-world medical classification datasets of varying sizes and complexity, reveal that with as few as 30 queries to an expert, WARM can significantly improve label model ac- curacy resulting in higher quality training labels. Morcover, WARM outperforms existing approaches to active weak supervision (Biegel et al. 2021 Nashaat et al. 2018) and supports interpretability by allowing experts track changes in LFs. Moreover, auxiliary experiments on subset of our datasets show that WARM can not only combat knowledge shift but also de-noise LFs. For example, found that WARM could adapt LFs detect the presence of heart disease defined on young population of patients, to an older population, resulting in a significant 1 - 3% increase in label and end model performance. We also found that WARM was able to easily refine noisy LFs achieve label and end model performance on par with no noise.
2 RELATED WORK
There is small but growing body of work on active strategies for weak supervision. A few studies combine data programming with active learning on data points. For instance, [Nashaat et al. (2018) proposed a method which actively obtained expert-labeled data via uncertainty sampling and cor- rected LF votes on individual data points to reflect the true labels. While their method improved the probabilistic predictions, it did not change the label model per se. Biegel et al. (2021) recently proposed Act ive WeaSuL which uses experts labels to learn how to best combine LFs. Specifi- cally, they add penalty term to the data programming loss function to nudge its optimal parameters to a configuration where the label model predictions agree with the expert labels. They also pro- posed novel active sampling strategy tailored the problem, which splits the data into buckets and measures the KL-divergence between the expert and estimated probabilistic labels to query new labels.
Some other studies have instead used labeled data to inspire ideas for new labeling functions (Cohen- Wang et al. 2019), and yet others have used them automatically come up with new one (Awasthi et al. 2020; Boecking et al.) 2020). Cohen- Wang et al. (2019) for instance proposed two simple strategies (i.e. data points where L.Fs abstain or disagree the most) to iteratively select subsets of data to inspire domain experts to create new LFs. On the other hand, [Boecking et al. (2020) introduced an interactive weak supervision method that automatically improves a set of LFs based on feedback from users.
Alternatively, to reduce human effort to an absolute minimum, some studies have attempted to auto- mate weak supervision by automatically generating heuristics based on a small labeled subset of the data (Varma& & Ré 2018 Nashaat et al. 2020). For instance, Varma & Ré (2018) proposed Snuba which automatically trains decision trees, logistic regression models and k-nearest neighbor classi- fiers as LFs, tuned for accuracy and coverage (amount of data that the I.F predicts a class as oppose to abstaining from predicting any class), and later pruned ensure sufficient dissimilarity. Finally, Nashaat et al. 22000) proposed asterissk extending their prior work on active weak supervision
We will release the source code for WARM at www. git.ub. com anonymi zed_ for_revies after neer-review.
2

---

Under review as a conference paper at ICLR 2022
by now automatically generating heuristics from small labeled dataset. In our experience, these methods are highly dependent on the quality and size of the labeled validation set. Furthermore, they may require large diverse labeled datasets for complex prediction tasks evidenced by our experiments on the diabetes dataset. Additionally, Asteri isteriskk sk needs a large amount (up to ~ 7% of training data) of actively labeled data points for good performance.
METHODOLOGY
3.1 PROBLEM FORMULATION AND MODELING ASSUMPTIONS
Let (x,y) D denote the data generating distribution, where each data point x E I is as- sociated with a latent class label E (1,...,k) ... Users provide unlabeled training dataset S. {(x, y)) where y/ unobserved, and m soft unipolar labeling functions (LFs) which we define as follows.
Since we only consider LFs that predict a single class, let pol(f) E fl,...,, .) define the polarity (or the single class that it predicts) of LF f. We define I = {mra)j= Yin R + [0, 1]. such that
Tr.j c) A 1 [c==pol(ss.) P| = c x,), where p is any arbitrary scoring function differentiable with respect to its parameters
Specifically, each LF "rrjj depends on one or more parameters in the parameter vector and ex- presses the likelihood that a data point x, belongs particular class as identified by the polarity (onctionpo(>>..])) function pol(?),)) .) We drop the subscript т in the rest of our discussion for the sake brevity.
3.2
MOST COMMON CLASSES OF LABELING FUNCTIONS ARE DIFFERENTIABLE
Fig. Iiii) is an example of an expert-defined LF characterize normal clinical findings in elec-
def high_ aEEG bine wtt.a/aanneenwwLLLL.... NORWAL(X, paraneters]:
troencephalogram (EEG) recording: which
"""High shaggy aEEG base laseline constantly at anplitude of around 10-20 aV, then NORMAL EEG.
depends on the h high basel ine parameter.
xit, Average EEG baseline
Our methods assume that all the LFs in the label model are differentiable with re- Many
votes np.wereel
x[1, 9] paranparsss ('high__shhhhhhhh bastegnessslnne" return vates
spect to their decision parameters. frequently used classes of LFs are intrinsi- cally differentiable with respect to their pa- rameters, such as multi-layer perceptrons, con- volutional neural networks, log-linear mod- els, neural language models (Hooper et al.) 2020 Varma & Ré 2018). Other com- mon classes of LFs can be easily transformed into their soft counterparts. One example is
high baseline NoNMAL :Module): """High shaggy aE5G baseline constartly an
amplitude of around 18-28 nV, then NORNAL EEG. ... def
_self_ paranteers);
super(). super().__nit_..)) init_ selt.paraneters parameters
def forward(self, x): Average baseline
decision stumps (IF <CONDITION> THEN CLASS <I> ELSE CLASS <J>) (Varma & Ré. 2018). More generally, AND or OR could also be made differentiable by fuzzy logic. To transform crisp decision stumps into their soft equivalents, we can follow prior work on de- cision trees (Suárez& & Lutsko 1999). Specif- ically, consider heuristic of the form: À A d > a, where d defines a new compos- ite variable and
votes torcchsiggoidd
x(:, self, paraneters Whhighhaaselliee
return vates
Figure 1: Examples of (i) crisp and (ii) soft equiv- alents a labeling function to characterize nor- mal clinical findings in EEG data. While both sions make identical predictions, the latter is dif- ferentiable with respect to its decision parameter, high.baseln line.
is decision param- eter. Then we define its fuzzy equivalent as - sigmoid(d"xx - a), where sigmoid(x) = 1+exp Intuitively, while indicates which side of the hyperplane d point lies, ? encapsulates a probabilistic notion of distance from the decision boundary. Using the corresponding
"Specifically, the I.F is developed to work with amplitude-integraaee EEG, which is widely used quantita- tive summary of raw multi-channel EEG data.
3

---

Under review as a conference paper at ICLR 2022
Zadeh operators, conjunctions are converted via (AND( y) - MIN y)), disjunctions verted via (OR(z, y) + MA y)) and negations are converted via (NCT(z) - (1 - z)) Zadeh 1996). There are still other ways to obtain soft equivalents of crisp LFs, for instance by tramiin feed-forward neural networks to approximate classes of functions under reasonable assumptions on their continuity and topology.
This extension of crisp LFs into their soft counterparts not only enables them to express confidence on their predictions, but also better capture fuzzy decision boundaries, common in most practical settings. Additionally, restricting LFs to be unipolar can be done without any loss of generality, since complex LFs can be easily broken down into their unipolar components. Moreover, in our experience, several simple unipolar LFs tend to perform better than a few complex multipolar LFs in practice. Each soft L. " can converted to its hard equivalent X; TR + {0. pol(A;)) using a threshold function, where 0 means that the LF abstained from voting for class pol( ()() In particular, we have that
X = Cc if (t, 2 0.5
00 otherwise
3.3 PROBABILISTIC LABEL ESTIMATION
Given the unlabeled training dataset S. and an initial set of LFs denoted by [(),, our goal is to efficiently improve the accuracy of LFs by refining their parameters 7 based on expert feedback, and ultimately assign probabilistic labels p(y ( x) the training data. Here [(t) denotes the set of LFs after iteration of WARM The probabilistic labels are used to further train a downstream classifier f X' + y. We refine the LFs automatically by asking domain experts to provide labels for few highly uncertain data points, and back propagating LF errors to update their parameters т.
In order to estimate probabilistic labels p(y p((), x), we propose the following weighted majority- vote label model:
b + j=1 7i (6) (x,, 1)0(0)
Di Σ Ek (b. + Σグ= 95 c),
(I)
such that Pa p(y: =1
[| 1(0) xx) is the estimated probability that x, belongs class L, while b, = ply = 1) denotes its prevalence. 7,50 (x,) is the probabilistic vote of jth LF and ((9) is its unobserved accuracy, estimated using the data programming (DP) label model as defined in Eq. 2 (Ratner et al. (016)).
We ensure that Pa is a probability by passing p; through a softmax function, and note that our model is differentiable with respect to all LF parameters т. The DP label model is not differentiable with respect to LF parameters, and therefore cannot be directly used estimate probabilistic labels while still having the ability update LF parameters. However, data programming has been shown to perform well in several weak supervision applications and has some theoretical guarantees for estimating the latent accuracy of LFs by optionally modeling dependencies between them. Thus, in order to iteratively improve the accuracy of LFs and estimate probabilistic labels, we alternate between two objectives the tt time step:
Estimate the accuracy of LFs keeping their parameters т fixed. Let [A (0, 10.1....,k)) }] denote the observed matrix of crisp LF outputs at the 1th time step, such that [A = ( (x:) is the thresholded output of LF yi on x,. We define factors for LF accuracy and
propensity as & Ace ([A (4) sg y/) A L [[A and (==ab([A()) y1) - 1 (0)): respectively.
We then follow Ratner et al. (2016) to define the joint distribution of the latent class label y and [A as:
po(y, (A(0)) A 1
exp
Σ Σ
(g
ACc((A w:) 0z m (ab((((()))))))))))))))))))))))) 9i)) + by:
(2)
Zo
j=1
4

---

Under review as a conference paper at ICLR 2022
We use Snorkel (Ratner al. 2017) to learn 0 by minimizing the negative log marginal likelihood given the observed [A(O]:
((t) arg min
-
log
Cooou. (A(1).)
(3)
WEY
We then set the accuracy weights of the LFs to be equal to og
Update LE parameters т keeping their accuracy weights fixed. In this step, we use our label model (Eq. D probabilistically label the training dataset, gather expert labels for the most uncer- tain data points, and finally update the LF parameters by minimizing the cross-entropy between the observed expert labels and probabílistic label predictions of the data points. In particular, we update the LF parameters via gradient back propagation, so as to minimize the negative log likelihood of all the expert labels observed so far:
LCE = Σ
Σ - ((= log Pa
(4)
(E 1=1
where S₁ is the set of expert labeled data points. While in experiments, at each time step t, we gather an expert label for the data point with the highest entropy, our methods are generic and seamlessly incorporate other active learning acquisition functions. Our algorithm briefly summarised in Algorithm!]
Algorithm WARM: The proposed active weak supervision algorithm.
1: procedure WARM(I S, {(x,, 3/ b)
2: for E to do
3:
[A c(
Convert crisp votes
4:
((t) <- arg min log
Σ pa (Y Vey
5:
for i + to |u do
6:
pa + b, + Σx=
(x 1000) , VI [I....,k}
Aggregate weighted LF votes
7:
p. < softmax p:)
8:
E, Et- Pa log(pa)
Compute entropies
9:
end for
10:
V4 t query.user((x,))
arg max E})
Query user for label
sES
11:
S - St U (xq: vq); S. S (xg. My)
12:
LCE - Σ - - ((=1,) log Pit
(x..1)E2,
13:
((++)) <- backprop.erroors(()) LCB)
D Update LF parameters
14:
end for
15: end procedure
EXPERIMENTS
4.1 DATASETS AND WEAK SUPERVISION SOURCES
Like most prior work on active and automated weak supervision, we primarily focus on the bi- nary classification setting for consistency (Varma RR 2018, Biegel et al. [02T Nashaat et al.) 2018). Specifically, we carry out experiments of varying complexity and sizes. See Table | for summary of all the datasets used. All of the above datasets were taken from the UCI Machine Learning Repository (Dua & Graff 2017). Refer Appendix A. for dataset details.
various real-world clinical classification datasets
Since did not have the requisite clinical expertise to define good weak supervision sources, we trained a Random Forest (RF) classifier decision stumps (depth-one decision trees) to auto- matically to simulate expert heuristics for simple LFs. Specifically, trained Random Forest

---

Under review as a conference paper at ICLR 2022
Dutaset
Cl Clssification task
# Train
# Test
# FFeatures
VLFS
C'Cassess
I'+ve clase (16)
Wiscomsin Breast Cuncer
Benign malignant canoer
140
Wiscomsin Diagnasitc Breast Cancer
Benign I maliguant cancor
455
114
Heart Disease
Absence / Presence of heart discase
736
184
13
5
55
Diabetes
Patient re-admitted
81412
20154
47
EBG
Clinical findings in
4238
1058
(37, 25,
Syndwetic
Gonesian clasters
10000)
3000
2
3
50
Table I: Summary of the six classification datasets used for evaluation.
classifier with 5 or 10 decision stumps on a 50% sub-sample of the training set, such that each decision stump served LF. We used the Random Forest Classifier implementation from the Seikit-Lea (Pedregosa et al.) 2011) Python library for the same. For all datasets, we acquired set of 10 LFs, with the exception of the diabetes dataset for which we used set of 5 LFs only. Appendix contains examples of I.Fs for the Heart Disease dataset.
As in Biegel et al. (021), we also benchmark WARM on a synthetic dataset comprising of two classes modeled by 2-dimensional isotropic Gaussians, with a training and testing set of 10,000 and 3,000 data points, respectívely. We use the same set of 3 LFs as Biegel et al. 202 for our experiments.
Finally, to subject WARM to even more realistic settings, we used to iteratively refine expert-defined heuristics to classify clinical findings in electroencephalogram (EEG) recordings. The dataset com- prised of continuous 6-hour anplitude-integraat patients resuscitated after suffering from cardiac arrest, and admitted to the Intensive Care Unit of a large tertiary care hospital between February 2010 and April 2019. The dataset was jointly annotated by two experts in cardiac arrest care and clinical EEG interpretation, using a simplified multinomial labeling convention to summarize EEG findings that occur commonly post cardiac ar- rest. Specifically, each raw EEG record was broken into six disjoint 1-hour windows and classified as generalized suppression; burst suppression with epileptiform activity (including burst suppression with identical bursts); burst suppression without epileptiform activity; and, near-continuous or con- tinuous background activity (Elmer et al. 2016 2020). We also developed total of 17 LFs tailored to the prediction problem with help from one of the expert clinicians.
EEG (aEEG recordings from 1310 comatose
4.2 COMPARISON OF METHODS
We compared WARM with the two existing active weak supervision approaches, namely Act ive WeaSuL (Biegel al., (02T) and the method proposed by Nashhaa al. (2018). All the methods had access to the same set of initial LFs and were run for 30 active learning iterations, where they were allowed to query one ground truth label from the training set each iteration. We also bench- marked WARM against Snuba (Varma & Rd, 2018). a popular weak supervision approach which automatically synthesizes weak supervision sources from a small subset of labeled examples incor- porates them with the data programming label model to probabilistically label data. We ran Snuba with a random labeled subset of 10, 20 and 30 data points with decision tree stumps as heuristics. To report test set performance, we trained downstream logistic regression models on the crisp versions of the probabilistic labels assigned by each method. We also compared these methods against a lo- gistic regression (LR) model trained using vanilla ictive learning ing sampling strategy based on the distance from the decision boundary as in Biegel et al. (202I). and a sallyyssppery one trained using the entire labeled training set. The fully supervised LR model serves proxy for the maximum achievable performance with the given features, train-test split and downstream classifier.
Since existing online implementations of Snuba, Active WeaSuL and the method proposed by Nashaat et al. (2018) were designed exclusively for binary classification problems, we were only able to compare WARM with active learning and fully supervised baselines on the EEG dataset involving multi-class classification. For all our experiments. we used the logistic regression model implementation from Scikit -Lean (Pedregosa et al.] [20IT), trained for maximum of 500 iterations and optimized using LBFGS. WARM models were trained and built using PyTorch 1.8.1 Paszke et al. (2019) along with Python 3.8.1. All our experiments were carried out on a Linux desktop with typical machine having Intel(R) Core(TM) i7-6700K CPUs and 16 GB of RAM.
aFEG is widely used quantitative summary of multi-channel EEG data.

---

Under review as a conference paper at ICLR 2022
Test Discriminative
Test Discriminative
Model
Model
WARM seriminativve
91
WARM Discrim nartve netive
79
89
10 Number labeled
Number of labeled points
(ii)
Knowledge shift experiments on the (i) heart disease and (ii) synthetic datasets. The accuracy of end models increases with more active learning iterations as WARM adapts LFs and the resulting label model to the target population.
Figure 2:
4.3 EXPERIMENTAL DESIGN
We aim to answer the following research questions through our experiments.
Does HARM improve the quality of the training dataset and how does this improvement translate to downstream classifier performance? ? We evaluate the quality of the training dataset by comparing the crisp versions of the probabilistic labels assigned by all the weakly supervised upproaches with the ground truth training labels.
Can WARM combat the problem of knowledge shift and overcome noise in the initial LFs? To this end, we carry out experiments on the synthetic and Heart Disease dataset. To simulate knowledge shift in the synthetic dataset, we start with the three LFs tailored to the original (source) data distribution, and introduce two other (target) gaussian clusters with different centroids. For the heart disease dataset, we automatically acquire LFs from the young population i.e. all patients with less than median age. We then train and evaluate WARM on disjoint subsets of old patient population. Since weakly supervised models rely heavily on the quality of its weak supervision sources, we conduct additional experiments on the Heart Disease dataset by adding varying levels of uniform random noise, scaled by the range of the features, to the LF decision parameters initially obtained using the random forest. Le. 100% noise added could mean that the decision stump threshold is off by 100%.
5 RESULTS AND DISCUSSION
Table 2 summarizes the results of our experiments. The Train results represent the performance of the weak label models in comparison to ground truth training labels, and hence are a direct measure of label model quality. The Test results the other hand, from LR models trained on the crisp equivalents of the probabilistic labels and evaluated on the held-out test set. The latter set of results primarily reflect the quality of the downstream models trained using weak labels. We refer the
Methods
Synthetic
Heart Discase
Diagnostic
Wisconsin BC
Diabetes
EEG
Active WeaSuL
91.05 0.30
47.01 i 0.54
67.69 1 0.00
67.69 0.00
33.94 $5.91...11 0.14
Train
Nashaat el al. 2018
90.70 0.01
48.23 1.17
65.36 1 0.11
91.63 0.21
53.98
0.21
Snuba
95.83 0.39
75.62 1: 2.34
92.53 0.90
94.17 1 1.69
32.06 2 L.6
WARM
9..29:0.22
78.78 : 1.35
93.05 = 0.90
91.10
0.00
60.19
0.24
71.16 2.09
Active WeaSuL
95.51 10.52
50.43 1.55
88.95 1: 1.53
93.57
1.28
34.57
0.41
Nashant el al. 2018
92.93 1 0.28
53.59
7.07
94.74
1.57
92.29
1.05
54.27
0.32
Test
Snuba
95.51 0.337
T1.30
2.02
88.95
1 0.43
92.86
1.63
32.14
4.28
WARM
95.77 0.20
77.72 0.34
91.23
0.96
89.29
0.00
60.14
0.20
72.29 1.48
Active Learning
97. 0.15
76.63 =: 1.33
97.19
0.35
94.86
0.70
32.21 1.41
641.51 = 1.80
Fully Sup.
LR
97.50
77.72
99.12
95.71
62.38
73.25
Table 2: Final accuracy of models on all the datasets together with 95% confidence intervals com- puted over 5 bootstrapping iterations. WARM improves the quality of training data measured by the accuracy of label model on the Train set. This improvement further translates into better down- stream model performance on the Test set. Fully Sup. LR represents the performance of the fully-supervised logistic regression (LR) model trained with full access to ground truth labels.
7

---

Under review as a conference paper at ICLR 2022
interested reader to Appendix A.7 for supplementary results with additional metrics such Fl and percentage change in accuracy. Our primary findings from the experiments are summarized below.
WARM can improve the quality of training data. We found that on all the datasets, the probabilistic labels assigned post active weak supervision via RARM are significantly more accurate than the ones assigned by compared methods. However, we did observe some variance in accuracy and its improvement across different datasets, primarily due to the nature of the data and the quality of initial weak supervision sources. For instance on the breast cancer data, WARM improved the accuracy of the label model by over 7% across 30 active learning iterations, On the other hand, WARM could not improve the train generative performance on the heart disease data, but it also starts from a better higher accuracy. We observed similar trends for downstream model performance, albeit with smaller accuracy improvements. The smaller accuracy improvements can be attributed to the generalization capabilities of the downstream model over the labels it is trained on. Nevertheless, our findings underscore the importance of high quality labeled training sets. On the EEG dataset (Table 2 and Fig. 7 WARM achieved 4 - 5% increase in both label model and downstream classifier performance. These results are significant since even the initial set of LFs were developed by expert clinicians.
Moreover, we also found that traditional Active Learning strategy outperformed WARM on simpler classification tasks such as breast cancer diagnosis. However, for more complex prediction problems such as the predicting readmission on the Diabetes datasel and classifying clinical find- ings on the EEG dataset, active weak supervision via WARM significantly outperformed end models trained using active learning. Finally, experiment runtimes suggest that WARM is significantly faster than existing active weak supervision techniques across datasets (Appendix A.55
WARM can help combat knowledge shift and de-noise LFs. Results from knowledge shift exper- iments on the heart disease and diabetes dataset Fig. 2 and Table 45 reveal that WARM can help both weak labelers and the downstream classifier adapt to different population characteristics. We observed approximately a 3% and 6% increase in label and end model performance on the heart disease and synthetic datasets, respectively. See Appendix AA6 for all LFs defined on the heart disease dataset to detect the presence of heart disease the young and old population.
Fig. iii) and Tab. @ummmrrzee the performance of WARM across 50 active learning iterations, when varying levels of random noise is artificially added to LFs defined on the Wisconsin Diagnostic Breast Cancer dataset. Our results clearly show that noisy LFs hurt both label and end model performance. However, WARM can iteratively de-noise LFs by actively collecting a few more labeled examples. Even with 75% noise initially, WARM can refine LFs such that the resulting label and end model perform on par with those trained using LFs with no noise. Additionally, WARM consistently outperforms the other active weak supervision baselines
Changes to LFs brought about by WARM are interpretable. If the LFs themselves are inherently interpretable, WARM supports their interpretability by enabling domain experts to track changes in their decision parameters. Fig. (i) for instance shows LFs before and after refinement by WARM on the synthetic dataset. We also draw the reader's attention to the knowledge shift experiments on the heart disease dataset for more examples of interpretable changes to LF parameters. Appendix A.6 lists all the LFs tailored to the young and old population. For example, the rule If exercise induced ang i na <= 144.c 666 (1 123) then ABSENCE else PRESENCE clearly says that while exercise induced angina higher than 144.67 was indicative of heart disease in the young population, older people tend to naturally have higher values resulting in higher decision threshold.
Ablation experiments on WARM shows that it benefits from both active learning and weak su- pervision. We also perform ablation experiments to investigate if WARM performing as expected. In Appendix A.8 we see that WARM directly benefits from Uncertainty Sampling from its active learning component, outperforming the random sampling in every case. From Appendix A0..9 it is generally able to generally outperform Active Learning on complex large datasets due to its Weak Supervision component.
'Noe that active learning results are not affected because we only add noise to L.Fs, which are only used by weak supervision methods
8

---

Under review as a conference paper at ICLR 2022
Discriminative
LF1 before
LF3 before
LF1 after
LF3 after
Feature x
75
Model Noise
25%
70
75%
100% Nolse
65
20
Number of labeled points
Feature
iii
Figure 3: (i): End model testing accuracy on the Wisconsin Diagnostic Breast Cancer dataset as function of uniform random noise and active learning iterations. With few expert-labeled data points WARM can effectively de-noise LFs to eventually improve end model performance. (ii): Plot of the synthetic dataset and its three LFs before and after 30 active refinement iterations. WARM supports interpretability by allowing experts to track changes in LFs.
CONCLUSION
With more medical data being collected, methods that address the lack of labeled data in clinical applications of machine learning are becoming increasingly vital. If the vast amounts of unlabeled clinical data could be utilized its full potential, the possibility of augmenting physicians with ML to increase positive patient outcomes is substantial. In addition, lot of of useful information can be provided to ML systems from expert guidance, whether it be via heuristic design or via hand- labeling individual data points; making such capabilities efficient and effective is greatly needed practice. WARM (Active Refinement of Weakly Supervised Models) is a fast active learning ad- dition to data programming that substantially and inexpensively improves the empirical accuracy of training labels and downstream classifiers. Our experiments show that WARM has the capability to address the common problems of knowledge shift and rule denoising with minimal additional labeled data. ran on interpretable LFs, WARM is even able to reveal strategic, data-driven, inter- pretable updates the I.F thresholds and parameters. Limitations of WARM include its reliance differentiable and reasonably accurate initial LFs, which may occasionally be more expensive to acquire than pointillistic labels. However, have shown that most frequently used classes of LFs are intrinsically differentiable or can be made so. WARM is already a step towards more efficient and effective algorithms to tackle labeled-data-scarce ML problems in healthcare and beyond.
Additional experiments would benefit precise assessment of properties and scoping of practical util- ity WARM in wider range of its possible applications. Future work should also involve user studies to gauge attainable efficiency gains and utility of interpretable adjustments to the LFs, and whether they meaningfully boost perceived usability of the weak supervision process in practice. Other future directions could include using meta-AL to learn query selection strategies based off of previous AL outcomes as in (Konyushkova et al. 2017 Bachman et al. 2017) or AL with partial feedback approaches in (Hu al. 2018). Another challenge to address is mitigating overfitting to the active learning samples; we could perform early stopping, learning rate decay, gradient clip- ping, cross validation to better generalize models. Furthermore, would like to research the connection between our framework and automated (armm & Ré 2018) or active (Boecking et al. 2020) discovery of new I.Fs.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi. Learning from rules generalizing labeled exemplars. arXiv preprint arXiv:2004.0603 2020.
Philip Bachman, Alessandro Sordoni, and Adam Trischler. Learning algorithms for active learning. In international conference on machine learning, pp. 301-310. PMLR, 2017.
Samantha Biegel, Rafah El-Khatib, Luiz Otavio Vilas Boas Oliveira, Max Baak, and Nanne Aben. Active weasul: Improving weak supervision with active learning. arXiv preprint arXiv:2104. 14847, 2021.
Benedikt Boecking, Willie Neiswanger, Eric Xing, and Artur Dubrawski. Interactive weak super- vision: Learning useful heuristics for data labeling. In International Conference on Learning Representations (ICLR), 2020.
Benjamin Cohen- Cohen-Wang, Stephen Mussmann, Alex Ratner, and Chris Ré. Interactive programmatic labeling for weak supervision. In Proc. KDD DCCL Workshop, 2019.
Christopher De Sa, Alex Ratner, Christopher Ré, Jaeho Shin, Feiran Wang, Sen Wu, and Ce Zhang. Deepdive: Declarative knowledge base construction. ACM SIGMOD Record, 45(1):60 6666006,660 2016.
Robert Detrano, Andras Janosi, Walter Steinbrunn, Matthias Pfisterer, Johann-Jakob Schmid, Sar- bjit Sandhu, Kern H Guppy, Stella Lee, and Victor Froelicher. International application of new probability algorithm for the diagnosis of coronary artery disease. The American journal of cardiology, 6455)3304-3 1989.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL BRLLtttpp//achiiv
ics. www..edimmmm
Jonathan Elmer, John J Gianakas, Jon C Rittenberger, Maria Baldwin, John Faro, Cheryl Plum- mer, Lori A Shutter, Christina Wassel, Clifton W Callaway, and Anthony Fabio. Group-based trajectory modeling of suppression ratio after cardiac arrest. Neurocritical care, 2553)4415-423, 2016.
Jonathan Elmer, Patrick Coppler, Pawan Solanki, M Brandon Westover, Aaron F Struck, Maria Baldwin, Michael C Kurz, and Clifton W Callaway. Sensitivity of continuous electroenccehalogg raphy to detect ictal activity before cardiac arrest. JAMA network open, 3(4):2203751-e203751 2020.
Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications image analysis and automated cartography. Communications of the ACM, 24 (6):381-395 1981.
Jason A Fries, Paroma Varma, Vincent S Chen, Ke Xiao, Heliodoro Tejeda, Priyanka Saha, Jared Dunnmon, Henry Chubb, Shiraz Maskatia, Madalina Fiterau, et al. Weakly supervised classifi cation of aortic valve malformations using unlabeled cardiac mri sequences. Nature communica- tions, 10(1):1-10, 2019.
Chufan Gao, Stephen Clark, Jacob Furst, and Daniela Raicu. Augmenting lide dataset using 3d generative adversarial networks to improve lung nodule detection. In Medical Imaging 2019: Computer-Aided Diagnosis, volume 10950, pp. 109501K. International Society for Optics and Photonics, 2019.
Hannah C Glass, Courtney Wusthoff, and Renée A Shellhaas. Amplitude-ineeraaeed electro- encephalography: the child neurologist's perspective. Journal of child neurology. 28(10):1342 1350, 2013.
Mononito Goswami, Benedikt Boecking, and Artur Dubrawski. Weak supervision for affordable modeling of electrocardiogram data. In AMIA Annual Symposium Proceedings, volume 2022. American Medical Informatics Association, 2022.
10

---

Under review as a conference paper at ICLR 2022
Awni Y Hannun, Pranav Rajpurkar, Masoumeh Haghpanahi, Geoffrey H Tison, Codie Bour, Mintu P Turakhia, and Andrew Y Ng. Cardiologist-lewe arrhythmia detection and classification in ambulatory electrocardiograms using deep neural network. Nature medicine, 25(1):65-69, 2019.
LJ Hirsch, SM LaRoche, N Gaspard, E Gerard, A Svoronos, ST Herman, Ram Mani, H Arif, N Jette, Y Minazad, et al. American clinical neurophysiology socicty's ' standardized critical care ceg terminology: 2012 version. Journal of clinical neurophysiology. 30(1):1-27, 2013.
Sarah Hooper, Michael Wornow, Ying Hang Seah, Peter Kellman, Hui Xue, Frederic Sala, Curtis Langlotz, and Christopher Re. Cut out the annotator, keep the cutout: better segmentation with weak supervision. International Conference on Learning Representations, 2020.
Peiyun Hu. Zachary C Lipton, Anima Anandkumar, and Deva Ramanan. Active learning with partial 2018.
feedback. arXiv preprint
Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. arXiv preprint 1703. 2017.
Hongwei Li, Bin Yu, and Dengyong Zhou Error rate analysis of labeling by crowdsourcing. In ICML Workshop: Machine Learning Meets Crowdsourcing. Atalanta, Georgia, USA. Citeseer, 2013.
Johan Löfhede, Nils Löfgren, Magnus Thordstein, Anders Flisberg, Ingemar Kjellmer, and Kaj Lin- decrantz. Classification burst and suppression in the neonatal electroencephalogran. Journal of neural engineering. 5(4):402, 2008.
Olvi L Mangasarian and William H Wolberg. Cancer diagnosis via linear programming. Technical report, University of Wisconsin- Madiison Department of Computer Sciences, 1990.
Mona Nashaat, Aindrila Ghosh, James Miller, Shaikh Quader, Chad Marston, and Jean-Francois Puget. Hybridization of active learning and data programming for labeling large industrial datasets. In 2018 IEEE International Conference on Big Data (Big Data), pp. 46-55. IEEE, 2018.
Mona Nashaat, Aindrila Ghosh, James Miller, and Shaikh Quader. Asterisk: Generating large train- ing datasets with automatic active supervision. ACM Transactions on Data Science, 1(2):1225, 2020.
Sang Hoon Oh, Kyu Nam Park, Young-Min Shon, Young-Min Kim, Han Joon Kim, Chun Song Youn. Soo Hyun Kim. Seung Pill Choi, and Scok Chan Kim. Continuous elecrroeneephalorrphiie monitoring is a useful prognostic tool for hypothermia-trean treattd cardiac arrest patients. Circulation, (2(12))1094-111 103, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Veto,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, -uc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http: 9015 pytorch- an anwpchh-nn-mmmetatiiiii imperative style www.-igggghhtee perforrancooooeoeeeeeeooeee dewpeaaaneeeeeeeeeeeeeeeeeeeeeee learning Librarrry........... pdf
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Seikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the VLDB En- dowment. International Conference on Very Large Data Bases, volume 11, pp. 269, NIH Public Access, 2017.
11

---

Under review as a conference paper at ICLR 2022
Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. In Advances in neural information processing systems. pp. 3567-3575, 2016.
Khaled Saab, Jared Dunnmon, Christopher Ré, Daniel Rubin, and Christopher MesMers Weak supervision as an efficient approach for automated seizure detection in electroencephalograapyy npj Digital Medicine, 3(1):1-12, 2020.
Beata Strack, Jonathan P DeShazo, Chris Gennings, Juan L Olmo, Sebastian Ventura, Krzysztof J Cios, and John N Clore. Impact of hbalc measurement on hospital readmission rates: analysis of 70,000 clinical database patient records. BioMed research international, 2014, 2014.
W Nick Street, William H Wolberg, and Olvi L Mangasarian. Nuclear feature extraction for breast tumor diagnosis. In Biomedical image processing and biomedical visualization, volume 1905, pp. 861-870. International Society for Optics and Photonics, 1993.
Alberto Suárez and James F Lutsko. Globally optimal fuzzy decision trees for classification and regression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21912311297-1311, 1999.
Jayaraman Thiagarajan, Deepta Rajan, and Prasanna Sattigeri. Understanding behavior of clinical models under domain shifts. arXiv preprint 3xX1::/809.07066, 2018.
Paroma Varma and Christopher Ré. Snuba: Automating weak supervision to label training data. In Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases, volume 12, pp. 223. NIH Public Access, 2018
Lotfi A Zadeh. Fuzzy sets. In Fuzzy sets, fuzzy logic, and fuzzy systems: selected papers by Lotfi A Zadeh, pp. 394-432. World Scientific, 1996.
A APPENDIX
A.1 DATASET DETAILS
The Heart Disease dataset (Detrano et al., 1989) contains clinical and noninvasive test results from patients undergoing angiography at the Cleveland Clinic in Cleveland, Ohio, Hungarian Institute of Cardiology in Budapest, Hungary, Veterans Administration Medical Center in Long Beach, Califor- nia; and University Hospitals Zurich and Basel, Switzerland. For our experiments, we concatenate all the databases into one and predict whether or not an individual has heart disease. The Diabetes dataser (Strack et al. 2014) comprises of clinical records from over million patients admitted 130 US hospitals and integrated delivery networks between 1999 to 2008. Given features representing patients such as their age, race, gender, time hospital, diagnosis etc., the goal predict whether they are re-admitted for further treatment. We also use two breast cancer datasets, namely Wisconsin Diagnostic Breast Cancer dataset (Street et al. 1993) and Wisconsin Breast Cancer Database (Man-) gasarian & Wolberg 1990). For both the datasets, the prediction task is to distinguish between sam- ples of malignant and benign breast cancer using different sets of features. The diagnostic dataset comprises of features characterizing cell nuclei present in digitized image of a fine needle aspirate (FNA) of a breast mass. The latter dataset instead contains 9 cytological characteristics of breast FNA graded between 1-10 at the time of sample collection.
A.2 EEG DATA COLLECTION
EEG monitoring was initiated median of 10 hours after initial collapse standard routine of care on arrival to the intensive care unit (Elmer et al., (0166.. EEG was recorded using XLTech Natus Neuroworks digital video/EEG systems (Natus Medical Inc) 256Hz from 22 gold-plated cup electrodes placed in the international 10-20 system positions.
Löfhede al. 2008) has shown near-perfect spatial correlation of many clinically important EEG features in patients after cardiac arrest, allowing the 22-sample signal to be down-sampled without significant information loss. Additionally, after cardiac arrest, many prognostic clinical findings can
12

---

Under review as a conference paper at ICLR 2022
be identified accurately from lower-resolution quantitative summaries (e.g. 1Hz aEEG, Fig. Oh et al. 2015 Glass et al. 2013). Thus, to optimize computational efficiency, we aimed to develop Similar ar heuristics as labeling functions to classify raw EEG signals from contemporaneous aEEG summaries.
time
Figure 4: An example aEEG waveform (T = 21600 seconds). Each alternating pink and grey region marks one-hour segment. The dashed red line represents baseline aEEG value determined using robust linear regression (aka RANSAC (Fischler & Bolles 1981).
Characteristic
Number (%)
Number of patients
1310
Age, mean (std), years
57 (17)
Female
503 (38)
Initial arrest rhythm
VT/VF
370 (28)
PEA
482 (37)
Asystole
371 (28)
Unknown
87 (6)
Arrest location
In-hospital cardiae
243 (19)
Out-of-hospital cardiac
1067 (81)
Hours from arrest to start, median (IQR)
(6)
Survived to hospital discharge
387 (30)
Disposition
Home
111 (29)
Acute rehabilitation
112 (29)
Skilled nursing facility
78 (20)
Long-term acute care
46 (12)
Hospice
22 (6)
Other
18 (4)
Cireumstances of death
Rearrested, intractable shock, multi-system organ failure
184 (20)
Withdrawal for non-neuroloiical reasons
114 (10)
Brain death
96 (12)
Withdrawal for perceived poor neurological prognosis
$29 (57)
Table 3: Clinical Characteristics and Patient Outcomes: A summary of the patients characteristics in the EEG dataset.
For our experiments, we used 6-hour aEEG time series starting at the initiation of EEG monitoring from 1310 patients with mean age 57 years of whom 503 (38%) were female. Approximately 30% survived to discharge from the hospital. A large subset of our data (approximately 70%) was initially jointly annotated by two experts cardiac arrest care and clinical EEG interpretation at UPMC. They manually annotated EEG findings using adaptation of the 2016 American Clinical Neurophysiology Society guidelines (Hirsch et al. 2013).
Based on prior research and expert clinicians, convention to summarize prognostic HEG findings that oceur commonly after cardiac arrest. Specif- ically, we classified the raw EEG waveforms as: generalized suppression; burst suppression with cpileptiform activity (including burst suppression with identical bursts); burst suppression without epileptiform activity; and, near-continuous or continuous background activity (Elmer et al. (020). Because continuous background activity with epileptiform activity occurred infrequently in this data set, did not further subdivide categories of patients with continuous background activity.
developed simplified multinomial labeling
13

---

Under review as a conference paper at ICLR 2022
A.3 KNOWLEDGE SHIFT EXPERIMENTS
Synthetic Dataset
Methods
Accuracy
AAccuracy
F1
Active WeaSuL
90.29 + 7.57
17.77 7.57
88.94 9.30
Train Generative
Nashaat et al.
71.17
-1.35
68.80
4.78
WARM
91.66 + 0.05
4.72
91.85 0.03
Active WeaSuL
92.83 2.880
9.47
92.46 3.03
Test Discriminative
Nashaat et al.
82.42 + 3.22 3.22
80.62 = -4.4
Active learning
96.85 + 0.09
46.85 0.09
96.85 0.08
WARM
91.71 + 0.11
5.95 = 0.15
91.60 = 0.14
Table 4: Knowledge Shift experiments on the Synthetic Dataset. Accuracy is final accuracy, and AAcouracy change from initial to final accuracy.
Heart Disease Dataset
Models
Accuracy
AAccuracy
F1
Active WeaSuL Nashaat et al.
36.31 = 0.00
0.00 = 0.00
5.47 = 0.38
Train Generative
37.26 = 0.52
0.95 0.52
8.02 = 1.05
WARM
80.56 + 1.88
1.06 + 0.65
+.22
Active WeaSuL
30.35 + 2.07
4.82 2.07
13.16
Test Discriminative
Nashaat et Active learning
35.32 + 3.15
9.79 + 3.15
21.27 7.59
79.01 = 1.65
29.01 + 1.65
85.65 + 1.42
WARM
80.14 + 0.78
2.55 1.24
86.97 + 0.33
Table 5: Knowledge Shift experiments on the Heart Disease Dataset. Accuracy final accuracy, and AAccuracy is change from initial to final accuracy.
14

---

Under review as a conference paper at ICLR 2022
A.4
ADDING ARTIFICIAL NOISE TO LFS
Heart Disease Dataset
Methods
Accuracy
AAcouracy
F1
0% Noise
Active WeaSuL
46.20 = 0.54
0.27 = 0.54
4.90
Train Generative
Nashaat et al.
48.23 = 1.17
2.31 1.17
11.75
WARM
78.61 + 1.56
0.00 + 0.00
80.50
Active WeaSuL
48.26 = 1.59
5.87 1.59
28.25
Nashaat et al.
53.59 = 7.07
11.20 +
Test Discriminative
WARM
78.59 = 0.43
0.00 = 0.00
1.64+00.4
25% Noise
Acti ve WeaSul.
48.34 + 0.53
0.65 + 0.53
12.00 2.12
Train Generative
Nashaat et al.
50.11 = 0.83
2.42
19.22 3.79
WARM Active WeaSuL
79.62 = 1.09
81.39 1.25
58.26 = 1.47
10.43
49.03 + 2.56
Test Discriminative
Nashaat et al.
59.13 = 5.34
11.30
50.36 9.79
WARM
77.83 = 0.72
0.54 = 0.60
81.84 + 0.87
50%
Noise
Active WeaSuL
47.15 + 0.00
0.00 0.00
10.16 + 0.00
Train Generative
Nashaat et al. WARM
48.26 + 0.25
1.21
15.00 0.74
78.89 = 0.96
-.333
80.78 +
Active WeaSuL
46.09 = 1.11
1.13
21.67 3.55
Test Discriminative
Nashaat et
52.07 = 7.89
11.30
31.31 + 18.65
WARM
77.17 + 1.50
- 11.41 + 1.12
80.84 + 1.58
75%
Noise
Active WeaSuL
43.29 = 1.47
-0.73 1.47
15.23 2.84
Train Generative
Nashaat et al.
48.45 = 4.21
4.43 4.211
25.20 6.44
WARM
79.54 + 1.25
0.43
$1.79
Active WeaSuL
42.61 = 0.81
1.30
11.35 2.56
Test Discriminative
Nashaat et al.
48.80 = 7.89
7.89
25.01 21.41
WARM
77.61 = 0.41
1.30 = 0.55
81.53 + 0.58
100% Noise
Active WeaSuL
49.86 + 0.00
0.00 0.00
35.83 0.00
Train Generative
Nashaat et al.
53.26 = 2.31
+
35.62 - 4.92
WARM
78.42 = 1.21
0.82
79.00 0.77
Active WeaSuL
37.83 + 2.24
2.20
13.52 4.19
Test Discriminative
Nashaat et al.
54.35 + 5.00
15.22
5.00
42.74
WARM
71.96 = 1.31
-1.41 0.55
74.64 1.36
Table 6: Experiments on Heart Disease dataset by adding varying levels of uniform random noise to LF decision parameters. Accuracy is final accuracy, and AAccuracy is change from initial to final accuracy. Noisy LFs hurt both label and end model performance. However, WARM can iteratively de-noise LFs by actively collecting few more labeled examples. Even with 75% noise initially, WARM can refine LFs such that the resulting label and end model perform on par with those trained using LFs with no noise. WARM also consistently outperforms other baselines.
15

---

Under review as a conference paper at ICLR 2022
A.5 RUNTIMES
Runtime (secandss)
20
Active WeaSuL
302
Nashaat
Diegnostic UANIscomsin BC
Dtebetes Datasets
Heart Unsease
bynthetic
Figure 5: Active weak supervision runtimes (in seconds on log scale). WARM is significantly faster than compared existing methods across all datasets.
16

---

Under review as a conference paper at ICLR 2022
A.6 EXAMPLE RULES
On the hearl disease dataset, we carried out our knowledge shift experiments, where the initial labeling functions were based the young population, but the target population was that of old patients. Below is a list of initial rules tailored to the young patients. The decision parameters in parenthesis reflect the modified labeling function parameters, tailored to older patients.
Chest pain type 0.898 (C. 978) then ARSENCE else PRESENCE
If Serum cholesterol in mg/dl <= 124.057 (119.793) then PRESENCE else ABSENCE If Maximum heart rate achieved <= 0.244 (0. then PRESENCE else ABSENCE then ABSENCE else PRESENCE then ABSENCE else PRESENCE
If Exercise induced angina <= 144. (150.
If thal <= 00.46
If Chest pain type <= 0.
then ABSENCE else PRESENCE
If ST depression induced by exercise relative to rest <= 00.73(0.383) then ABSENCE else PRESENCE
If Exercise induced angina <= 146.666
(150.123) (14.6661155.11111 then ABSENCE else PRESENCE thal <= 0.646 then ABSENCE else PRESENCE
If Maximum heart rate achieved <= 0.533 (0.367) then PRESENCE else ABSENCE
17

---

Under review as a conference paper at ICLR 2022
A.7
WARM COMPARISON TO OTHER MODELS
70
Train Generative
Train Generative
Model
60
VARM Generative
Accurncy
70
Active
60
Nashast et al.
WARM Gonerative
Snuha
66
30
points
points
(i) Diabetes
(ii) EEG
Generative
100
Active WeaSul.
Neshast et
Accurscy
Snuba
WARM Generative
Model
20
Active Westiul
50
Comerative
Number of abeled points
Number of labaled points
(iii) Wisconsin Breast Cancer
(iv) Wisconsin Diagnostic Breast Cancer
Train Generative
Train Generative
100
100
Model
Nashaat
WARM Gemerativee
Active WeaSul
Nashast at al.
Snuba
WARM Generative
5
15
25
30
15
25
Number laeeled points
Namber labeleed points
(v) Heart disease
(vi) Synthetic
Figure 6: Results of label model predictions the training datasets. These results are direct measure training set quality.
Wisconsin Breast Cancer Database
Methods
Accuracy
AAcouracy
Active WeaSuL (Biegel et
2021
67.69
85.88
Nashaat 2018
91.63 0.21
86.75
0.40
Train Generative
Snuba Varma Re, 2018
94.17 1.69
4.83 4.24
91.40
2.71
WARU (Ours)
94.10
7.00
91.08 0.02
Active WeaSuL (Biegel al 2021
93.57 1.28
0.00 1.28
90.20 1.95
Nashaat 2018
92.29 1.05
-1.29 1.05
88.08 1.79
Snuba Varma Ke 2018
1.66611.63
3.29 3.777
88.71 2.98
Test Discriminative
WARM (Ours)
50.29-10000
1.07 - 0.36
82.76 0.00
Active Searning
94.86 1 0.70
44.86 + 0.70
92.39 1.11
ES Logistic Regression
Table 7: Performance metrics for the Wisconsin Breast Cancer database. Accuracy is final accuracy, and AAccuracy is change from initial to final accuracy.
18

---

Under review as a conference paper at ICLR 2022
Test Discrininative
Test Diseriminatinn
70
60
60
50
40
Nashast al.
WeadaL
Active leaming by lssil
Model
Sruba
Active leamiing itsef
VMAMM Disctminattve
WMRM Discriminative
20
10
15
20
25
30
5
25
25
30
Number labeled points
Namber labeleed points
ii) Diabetes
(ii) EEG
100
Test Discriminative
100
Test Discriminative
90
90
80
80
Accurocy
70
Active Wealul.
Model
70
Model
Wea leamiing
60
60
lcaming by itslf
50
Snule
50
WARM Disciminative
WARM Discrimiratie
40
40
Number lbbeled points
Number labeled
(iii) Wisconsin Breast Cancer
(iv) Wisconsin Diagnostic Breast Cancer
Discriminative
Test Discrtminative
Active WeaSuL
Nashastt et al.
Active learning by ibsel
Accurary
Snuba
Model
WARM Discriminaivve
Active WeaSuL
Nashaat
Active leaming itser
5
10
15
20
25
30
5
30
35
20
25
30
Number of abbled points
Number of labaled points
(v) Heart disease
(vi) Synthetic
Figure 7: Results of the logistic regression end model evaluated the testing dataset. datasets.
Wisconsin Diagnostic Breast Cancer Dataset
Methods
Accuracy
AAcouracy
F1
Active WeaSuL (Biegel ct al
2021
67.69 + 0.00
0.00 0.00
28.29 + 0.00
Nashaat et al 2018
65.36 0.11
-2,33 0.11
18.93
Train Generative
Snuba Varma & Ke 2018
92.53 + 0.90
2.37 1.69
90.07 : 0.86
(Ours)
93.05 : 0.90
1.25 7.78
90.52 1 1.74
Active WeaSuL (Biegel et al 2021
88.95 1 1.53
0.35 1.53
19.28 1 3.23
Nashaat et al 2018
9..74
6.54
90.83 3.01
Test Discriminative
Snuba Varma Re 2018
88.95 + 0.43
0.88 +.09
82.15 0.39
WARM (Ours)
91.23 0.96
3.16
87.27
Active Searning
97.19 0.35
47.19 1 0.35
95.45 0.59
ES Logistic Regression
Table Performance metrics for the Wisconsin Diagnostic Breast Cancer dataset. Accuracy is final accuracy, and AAccuracy is change from initial to final accuracy.
19

---

Under review as a conference paper at ICLR 2022
Heart Disease Dataset
Methods
Accuracy
ARccuracy
F1
Active WeaSub (Biegel et
2021
47.01 1.5.4
0.14 0.54
6.61 1.14
Train Generative
Nashaat et al 2018
48.23 1.17
1.36 1.17
11.75
Snuba (Varma
2018
75.62 2.34
12.07 + 9.97
75.95
(Our))
7
0.08 0.24
1..
Active ive WeaSaa (Biegel et 2021
50.43
1.55
33.21
Nashaat al. (2018
53.59 7.07
38.54
Test Discriminative
Snapa
(Várma Ke 2018
71.30
9.56
73.01 4.73
MARM (Ours)
77.72 0.34
-0.87 +.27
81.74 0.55
Active Learning
76.63 + 1.33
26.63 1.33
79.34 1.17
FS Logistic Regression
Table 9: Performance metrics for the Heart disease dataset. A Accuracy is change from initial to final accuracy.
Accuracy final accuracy, and
Diabetes Dataset
Methods
Accuracy
AAcouracy
Active WeaSuL (Biegel ul. 2021
53.94 = 0.14
0.10
7.77=
Nashaat al. 2018
53.98 = 0.21
0.11
7.35
Train Generative
Snuba Varma Rc 2018
52.06 = 4.06
0.41
50.16
(Our)
60.19 + 0.24
0.29
48.53
Active WeaSuL [Biegel et al. 2021
54.37 0.41
00.47
3.73
Nashaat et al. 2018
54.27 + 0.32
0.37
2.62 2.37
Snuba Varrm Re 2018
52.14 4.28
5.44
50.04 5.94
Test Discriminative
(Ours(Orr)
60.14 0.20
0.21
48. 1 2.44
Active Learning
52.21 + 1.41
2.21
50.23 + 7.44
ES Logistic Regression
Table 10: Performance metries for the Diabetes dataset. Accuracy is final accuracy, and AAccuracy change from initial to final accuracy.
Synthetic Dataset
Methods
Accuracy
AAcouracy
F1
Act ve MeaSuL. Biegel al. 2021
91.05 + 0.00
0.00+0..00
85.88 + 0.00
Nashaat al. 2018
90.70 + 0.01
0.03 0.01
90.13 + 0.01
Train Generative
Snuba Varma Ré 2018
95.83 + 0.39
2.28 3.63
9
WARM((Ors))
95.39 + 0.22
1.52
05.55
Active WeaSuL (Biegel 2021
95.51 1 0.52
2.88
95.36
Nashaat
92.93 0.28
0.30
92.71 + 0.29
Snuba Varma Kc 2018
95.51 0.37
1.65
95.49 - 0.26
Test Discriminative
(Ours)
95.77 0.20
2.21
0.19
95.80 + 0.23
Active Learning
97.33 0.15
47.33 0.15
97.34 + 0.13
Logistic Regression
Table 11:
Performance metries for the Syntheric dataset. AAccuracy is change from initial to final accuracy.
Accuracy final accuracy, and
EEG Dataset Accuracy
Methods
AAccuracy
F1
Train Generative
WARK (Ours)
71.16 2.09
5,00 + 1.65
69.30 4: 2.66
WARK (Ours)
72.29 t 1.48
3.89 1.12
70.19
2.14
Test Discriminative
Active Learning
64.50 1.80
14.50 1.80
62.79 16.78
Logistic Regression
Table 12:
Performance metrics for the EEG dataset. Accuracy is final accuracy, and AAccuracy change from initial to final accuracy.
20

---

Under review as a conference paper at ICLR 2022
A.8
WARM RANDOM SAMPLING VS UNCERTAINTY SAMPLING
Synthetic Dataset
Methods
Accuracy
AAccuracy
F1 93.67 = 2.75
WARM
93.77 = 2.29
0.34 2.19
Train Generative
WARM (US)
95.39 + 0.22
1.52 + 0.26
95.55 + 0.17
WARM
93.72 2.52
0.35 + 2.44
93.39 3.03
Test Discriminative
WARM (US)
95.77 + 0.20
2.21 + 0.19
95.80 + 0.23
Table 13:
Effect of random sampling Synthetic Dataset. Accuracy is final accuracy, and AAccuracy is change from initial to final accuracy. Accuracy and score metrics are much lower for random sampling (RS) than the uncertainty sampling (US).
Wisconsin Diagnostic Breast Cancer
Methods
Accuracy
AAccuracy
F1
WARM (RS)
82.86 + 3.80
-3.82 3.63
69.49 8.79
Train Generative
WARM (US)
93.05 = 0.90
7.78 = 1.25
90.52 1.74
WARM (RS)
83.16 + 3.74
-4.91 - 3.75
62.77 + 10.81
Test Discriminative
WARM (US)
91.23 0.96
3.16 = 1.31
87.27 + 1.19
Table 14:
Effect of random sampling on Wisconsin Diagnostic Breast Cancer Dataset. Accuracy is final accuracy, and AAccuracy is change from initial final accuracy. Accuracy and Fl score metries are much lower for random sampling (RS) than the uncertainty sampling (US).
21

---

Under review as a conference paper at ICLR 2022
A.9 WARM ABLATION EXPERIMENTS ON DATASET SIZE AND DOWNSTREAM MODELS
For the rest of our experiments, we created synthetic datasets using sklearn's make.classiffcati
method. The datasets comprise of clusters of points normally distributed (with standard deviation 1) about vertices of 25-dimensional hypercube with sides of length 2, with an equal number of clusters (= to each class, We used 25 decision tree stumps as LFs. In order to automatically create the simple LFs, we followed the same procedure outlined in Section [.1 with the only difference being that the Random Forest classifier was trained using 250 data points sampled from the training dataset. This was done to ensure that WARM received the same amount of weak supervision regardless of the size the dataset, and the differences in performance were solely due to the size of the dataset and not the quality of weak supervision.
Downstream Model
Dataset size
Methods
Accuracy
AAccuracy
F1
Active learning
2.001
7.70 1 2.01
62.55 1 4.30
1000
WARM
58.79 + 1.70
0136 = 2.30
58.51 + 3.88
Active learning
56.88 20.88:111
1.12 1.12
48.64 + 2.06
5000
Logistic Regression
WARM
5..80
-2.96 4.99
46.81 11.11
Active learning
54.14
4.14
54.10 14.08
10000
WARM
61.20
1.75
67.98 + 1.08
Active learning
62.35 1: 1.55
12.35
63.86 1 0.76
20000
WARM
0.97
0.22=1007
61,82 = 4.97
Activv learning
50.79
6.79 2.38
51.09 + 4.15
1000
WARM
58.67 + 1.87
2.30 + 4.69
57.87 + 4.22
Active learning
60.27
10.27
52.21 = 0.59
5000
Multi-Layer Perceptron
WARM
1:.20
1.13 2.577
44.82 10.96
Active learning
2..08
5.28
63.70 + 2.60
10000
WARM
61.82
1.16
68.24 +
Active learning
10.10 6.11
57.41 5.04
20000
WARM
62.9
0.05 = 1.53
62.11 + 4.87
Act f Tea
55.64+ 1.64
5.64 + 1.64
59.37 + 8.67
1000
WARM
58.91 = 1.42
0.8
59.08 = 4.09
Active learning
59.02 1.87
9.02 10221188 1.87
61.98 + 2.09
5000
WARM
2.33
3,33 5.32
41.49 = 12.12
Random Forests
Act ve learning ng
55.71 2-117
5.71 4.117
56.82 11.39
10000
WARM
60.25 1.38
1.50
67.37 1.64
Active learning
56.42 2.19
6.42 2.19
44.88 4.35
20000
WARM
62.05 1.07
0.52 1.76
61.24 5.66
Testing performance of different downstream models datasets of different sizes. Ac- curacy is final accuracy, and AAccuracy is change from initial to final accuracy. Due to the complexity of the dataset, WARM outperforms Active Learning on most occasions. Moreover, WARM's performance improves as the size of the dataset increases, as it is able to better estimate the accuracies of LFs.
Table 15:
att.://eeee..kaaaaaaaaaaaaaaa dataseta.
make_classi ficatio htm]
22

---

Under review as a conference paper at ICLR 2022
A.10 WARM EXPERIMENTS ON COMPLEXITY OF DATASET
The following results are reported on a datasets with 10000 samples and varying numbers of clusters per class, created using sklearn's 's make_clase assssfcaas ficatiir ion method. The number of clusters per class controls the complexity of the problem, and more clusters per class is a harder classification problem than fewer clusters per class. The weak supervision sources were created in exactly the same way as mentioned in the previous section.
Downstream Model
Complexity
Methods
Accuracy
AAccuracy
F1
Active learning
55.45 + 1.34
5.45 + 1.34
62.99
9.72
8
WARM
60.30 + 3.26
3.70 2.96
54.29 + 12.00
Active learning
61.45 t 4.55
4.55
49.31 + 10.97
6
Logistic Regression
WARM
63.09 +: 2.24
-1.09
55.64 8.66
Active learning
53.76 +: 5.58
3.76
59.41 5.17
4
WARM
59.88 +: 1.06
5.52 1.69
63.93 3.02
2 Active learning
80.67 + 3.04
30.67 3.04
80.07 3.15
WARM
76.73 +: 2.18
0.18 = 1.88
76.52 4.43
Table 16: Testing performance of downstream models on datasets with increasing complexity. Ac- curacy is final accuracy. and AAccuracy is change from initial to final accuracy. The more clusters per class, the more complex the dataset is, as can also be seen by the deteriorating performance of the models. When the dataset is less complex, Active learning does well, but the complex- ity increase WARM does better. We expect Active learning to take many more data points to learn decision boundaries in more complex datasets.
23


Reviews:
[{"review_id": "ygNI_oGHEs-", "reviewer": "Reviewer_aMJi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In equation (1), the \\propto seems like the wrong relation since the left hand side is actually the softmax of the right hand side.\n\nWhy are the labels not used in estimating the labeling function accuracies? \n\nIn figure 3.(i), why is the performance not monotonic? In particular, the curve without noise peaks with 20 labels and then starts to deteriorate.\n\nIt's interesting to note that WARM's downstream performance never improves over the accuracy of the label model. It seems that a piece of the weak supervision pipeline is broken here. Perhaps using a more expressive model (random forests?) would be more appropriate for the downstream model.\n\nI'm surprised that the active learning baseline generally outperformed the weak supervision methods even though the weak supervision methods have access to extra information (the labeling functions trained on the whole dataset).\n\n\n\n\nMinor things:\nI think the arguments of p_\\theta are swapped throughout the paper. Look at eq (2), eq (3), and line 4 of the algorithm.\n", "summary_of_the_paper": "This paper proposes an algorithm for choosing a small set of labels to improve labeling function model performance both directly and for downstream tasks. Additionally, the authors provide a general method to convert standard labeling functions to \"soft\" labeling functions which are differentiable with respect to some parameters (e.g. a threshold). If the labeling functions are differentiable, this paper provides a method to update the labeling function parameters. Finally, experimental results show that the method introduced outperforms other active labeling approaches for weak supervision.", "summary_of_the_review": "The method proposed in this paper does not contain any particularly novel ideas and seems to be based on heuristics (maybe the proposed quantities could be derived from more general principles?). Additionally, it appears that weak supervision is not appropriate for the paper's empirical settings as seen by the lack of improvement from the downstream model and the stronger performance of non-weakly-supervised methods (active learning).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mXwmHX4o3tu", "reviewer": "Reviewer_JFqf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nThe proposed method is clearly described and the writing is easy to follow.\n\nWeakness:\n1. The theoretical analysis is insufficient and I recommend the authors provide more analyses on their main contributions.\n2. The general idea of active learning with weak supervision is not novel, which can be seen in [1]. The methods proposed in this work are combinations of existing ones and the authors also fail to contribute novel theoretical results.\n3. The layout should be improved. I recommend the authors separate the key notations, e.g., labeling functions, from the contexts for better presentation. There are also some typos, e.g., 'dependant' should be 'dependent' in the last paragraph of page 2.\n\n[1]. Chicheng Zhang and Kamalika Chaudhuri, Active Learning from Weak and Strong Labelers. NIPS 2015: 703-711\n", "summary_of_the_paper": "In this work, the authors propose the WARM method to help conduct iterative and interactive weakly-supervised learning. Active learning is used here to refine the labeling functions by focusing on data points that are once labeled. The authors further incorporated gradient propagation to alternatively update the LF parameters and the DP model. Experimental results show that the WARM method can improve the quality of training data.", "summary_of_the_review": "The authors should compare with more related works and provide some theoretical analyses to make this work more convincing. The ideas are totally heuristic and the experimental results are also not satisfying. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bTe_sloBxnv", "reviewer": "Reviewer_bVTy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. The proposed method would be of great use in many real-world scenarios where labelled data is scarce (e.g. medicine).\n2. To my knowledge, the authors’ proposed method (i.e., actively refining the voting weights of each labeling function and the parameters of the labeling function) is novel.\n3. Generally I found the writing in the manuscript to be of high quality. As someone who is not an expert in data programming I found the manuscript easy to follow with some minor exceptions (see point 3 in “Cons”).\n4. The authors experiment with their method on a variety of datasets, including one for which they use human domain experts to craft labeling functions. I greatly appreciate the application to a real-world scenario!\n\nCons (listed in order of importance to my score):\n\n1. I found it difficult to assess the significance of the knowledge shift experiment results presented in Figure 2 and Figure 3 due to a lack of any results from baseline models. As such, I would appreciate it if the authors could add results from their baseline models to these Figures to (1) illustrate the severity of the knowledge shift problem and (2) (potentially) better illustrate the advantages of WARM over previous work.\n2. As mentioned by the authors in Section 5, the active learning baseline outperforms WARM on half of the tested datasets. The authors say that this is due to some datasets being “simpler” than others, though I was not clear as to what “simpler” meant here. I would be willing to raise my score if the authors could provide experimental results that clearly illustrate the reasons behind the poor performance of WARM on these “simpler” datasets. For example, if “simpler” means having fewer data points, an experiment that assesses WARM's performance on simulated data with varying dataset size would be helpful to better understand when WARM may outperform existing methods vs. when it may underperform them.\n3. Some of the terminology relating to the problem formulation/labeling functions is not precisely defined within the manuscript and can be confusing for readers not already familiar with data programming (e.g. I had to look outside the manuscript for a definition of the “polarity” of a labeling function).\n4. The literature review in Section 2 is very data-programming specific, and does not discuss other recent approaches to active learning (e.g. [1,2,3]). Expanding the related works section would help put WARM in a broader context and assist the reviewer in assessing the significance of WARM.\n\n[1]: “Learning Active Learning from Data” (NeurIPS 2017)\n[2]: “Learning Algorithms for Active Learning” (ICML 2017)\n[3]: “Active Learning with Partial Feedback” (ICLR 2019)\n", "summary_of_the_paper": "This paper proposes a new method for data programming, i.e., using weak supervision to generate probabilistic training labels for unlabelled points using heuristics devised by domain experts. In particular, the authors propose WARM, a framework for iteratively improving these weakly supervised models by modifying the parameters of labeling functions and directing users to a subset of data points that, when labelled, would most improve the model.\n", "summary_of_the_review": "Overall I enjoyed reading this paper. The authors’ method appears methodologically sound and it seems to provide considerable benefits when applied to complex datasets. However, there are some weaknesses in the manuscript that somewhat undermine the paper’s story. For now I am recommending a weak accept, and I am willing to revise my score upwards if the authors address my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SovRYRQyyCR", "reviewer": "Reviewer_379M", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strengths of the paper lies in effectively combining active learning and weak supervision to generate high quality labels. The proposed approach incrementally improves the labeling functions by using the true labels obtained in each round. To achieve this they need LFs to be differentiable, which can be a drawback in some cases. Except these few changes, it is still using most of the existing weak-supervision machinery (i.e. label model, accuracy estimation and weighted combination of labeling functions to generate final labels etc.). So while the novelty might seem limited, I think it is useful in the sense that it may require minimal changes in existing weak-supervision setups to enhance it using active learning. \nI don't see major problems with this work, except a few. Firstly, interpreting the accuracy results in Table 2, is slightly difficult since class distributions in the datasets are not provided i.e. are the datasets imbalanced?  From the results it looks like the cases where one can obtain good labeling functions from experts then this approach works better than active learning otherwise not. There might be cases where obtaining expert labeling functions is more costly than obtaining labels ( which is probably easier than writing LFs). It might be worthwhile to discuss these limitations in the paper.     ", "summary_of_the_paper": "The paper gives a method to iteratively and interactively improve the label model in weak supervision. The approach consists of two steps, first is standard weak supervision way of weighted combination of labelling functions to generate labels. Novelty and improvement mainly comes from the second step, where true label for most uncertain data point is queried using which the parameters of the labelling functions are improved, which in turn lead to a more accurate weak supervision model. A key requirement and assumption in this paper's setup is that labelling functions are given by some learnable parameters ( i.e. they can be differentiated w.r.t. their parameters), which allows parameters updates using the true labels acquired. Empirical results on various real world datasets in medical domain show that in some cases this approach can yield a more accurate model in comparison to pure active learning approach and some recent baselines which combine weak supervision with active learning. These results also show that the paper's approach can get accuracy comparable to fully supervised model as well.  ", "summary_of_the_review": "Overall, I think its a nice paper with a sound and simple approach to improve weak supervision's label quality using active learning. The contributions are novel and useful in practice. I am inclined towards accepting this paper. I need some clarifications in the experiments section to be more confident in this assessment. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5Ig1AHj2rU", "reviewer": "Reviewer_jkgK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- (S1) This paper tackles an important problem with an intuitive approach of complementing recent programmatic/weak supervision approaches with expert feedback via an active learning-style approach.\n- (S2) This paper introduces a clean formulation of / argument for LFs being cast as differentiable functions, whereas to date most LFs have been non-differentiable\n- (S3) The paper shows some strong results relative to recent approaches.\n- (S4) The paper includes a range of datasets from synthetic (data + LFs), to 'semi-synthetic' (real data + synthetically generated LFs), to a real EEG task/dataset + LFs developed in conjunction with medical SMEs- an impressive and real world-relevant contribution.\n\nWeaknesses:\n- (W1) Simple method: While not a major drawback in isolation, it is worth nothing that the proposed approach is fairly simple and standard from a methodological/algorithmic standpoint.  From an active learning perspective: the query function is just based on model uncertainty, which is the most basic type of active learning query function (the only tweak being that it is the label model, i.e. model over LFs, but this does not change anything from an algorithm perspective).  Then, these data points are used to tune the parameters of the LFs in a manner that is also straightforward (the only tweak here being that the approach alternates between two formulations of the label model objective which is not necessarily needed... see W#1.a).\n- (W1.a) The authors state that the data programming label model is not differentiable with respect to the tunable LF parameters introduced in WARM, which is not true.\n- (W2) Lack of exploration of effect of differentiable LFs: Given the above lack of methodological novelty, this reviewer at least saw one of the main points of novelty and overall contribution of the paper being around the differentiable LFs themselves, and the overall setup here.  However, unfortunately this contribution was not explored in any depth (e.g. what are the tradeoffs of \"softening\" LFs to make them differentiable?  How should we think about this more broadly beyond the medical settings treated?  etc), which would have been interesting and significantly strengthened this aspect of the overall contribution.\n- (W3) Lack of relevant ablations: In general, there were a range of ablations of the overall approach I would have thought natural- for example, what is the impact of an active learning setup vs. just using some randomly sampled labeled data to tune the LF's internal parameters?  Could these internal parameters also be learned without labeled data, following the basic Snorkel modeling approach, and how would that do?  How would the approach do without tuning the internal LF params?  Etc.\n- (W4) Weak/improper comparisons: Since two of the other approaches compared to have no access to tune the internal params of the LFs and WARM does, this seems like a somewhat handicapped comparison...", "summary_of_the_paper": "This paper proposes WARM, an active learning approach to weakly/programmatically supervised learning.  In the WARM approach, which bases off of the data programming/Snorkel paradigm for weak supervision, users write labeling functions (LFs) to programmatically label training data; these labeling functions are then modeled by the Snorkel framework for weak supervision and used to train downstream models.  In the WARM setup, these LFs are assumed to be, or cast as, differentiable.  The paper then proposes an active learning approach to sampling labeled data points to tune the parameters of these LFs, and validates this approach on several medical datasets.", "summary_of_the_review": "Overall, the proposed approach introduces some interesting and practical ideas with some exciting experimental applications, however does not sufficiently explore the most novel elements of the contribution, ablate them sufficiently, or compare them to prior methods appropriately. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose WARM, a novel method that actively queries a small set of true labels to improve the label function in weak supervision. In particular, the authors propose a methodology that converts the label function to "soft" versions that are differentiable, which are in term learnable with true labels using proper updates of parameters. Empirical results on several real-world data sets demonstrate that the method yields a pretty strong performance.

The reviewers generally agree that the idea of making the labeling functions differentiable is conceptually interesting. They are also positive about the simplicity and the promising performance. They share joint concerns on whether the idea has been sufficiently studied in terms of the design choices and completeness of the experiments. For instance, the authors can conduct deeper exploration of the trade-off for differentiable LFs. They can also study active learning strategies that are beyond basic uncertainty sampling. While the authors have provided more studies about those exploration and ablation studies during the rebuttal, generally the results are not sufficient to convince most of the reviewers. In future revisions, the authors are encouraged to clarify its position with respect to existing works that combine active learning and weakly-supervised learning.

The authors position the paper as more empirical than theoretical. So the suggestion from some reviewers about more theoretical study is viewed as nice-to-have but not a must.
Decision: Reject