Year: 2021

Abstract:
Supervised machine learning (ML) has fueled major advances in several domains such as health, education and governance. However, most modern ML methods rely on vast quantities of point-by-point hand-labeled training data. In domains such as clinical research, where data collection and its careful characterization is particularly expensive and tedious, this reliance on pointillistically labeled data is one of the biggest roadblocks to the adoption of modern data-hungry ML algorithms. Data programming, a framework for learning from weak supervision, attempts to overcome this bottleneck by generating probabilistic training labels from simple yet imperfect heuristics (or labelling functions) obtained a priori from domain experts. We present WARM, Active Refinement of Weakly Supervised Models, a principled approach to iterative and interactive improvement of weakly supervised models via active learning. WARM directs domain experts’ attention on a few selected data points that, when annotated, would most improve the label model’s probabilistic accuracy. Gradient updates are then backpropagated to iteratively update the parameters of the individual expert labelling functions in the weak supervision model. Experiments on multiple real-world medical classification datasets reveal that WARM can substantially improve the accuracy of probabilistic labels used to train downstream classifiers, with as few as 30 queries to experts. Additional experiments with domain shift and artificial noise demonstrate WARM’s ability to adapt to changing population characteristics as well as noisy initial labelling functions from the experts. These capabilities make WARM a potentially useful tool for deploying, maintaining, and auditing weakly supervised systems in practice.

heading: 1 INTRODUCTION
text: Machine learning (ML) has seen widespread adoption in several domains such as health, education and public policy. In the clinical context, supervised ML algorithms have been shown to be effective solutions to a gamut of problems, ranging from detecting abnormal heart rhythms in electrocardiogram (Goswami et al., 2022; Hannun et al., 2019), prognosticating neurological recovery using continuous electroencephalogram (Elmer et al., 2016; 2020) to detecting lung cancer in CT scans (Gao et al., 2019). However, most modern ML algorithms, especially large deep neural networks, rely on vast quantities of pointillistically labeled training data. In our experience, while raw clinical data is abundant, its careful annotation is not. Manually labeling large quantities of clinical data is often tedious, prohibitively expensive, prone to error, and acutely unscalable.
Recently, many studies have explored the use of cheap, potentially noisy sources of supervision to noisily annotate large training sets (Goswami et al., 2022; Fries et al., 2019; Saab et al., 2020) using weak supervision. These methods combine these sources to create noisy labels on unseen data, which can then be used to learn powerful discriminative end models. In principle, these methods resemble other repeated-labeling techniques such as crowd-sourcing (Li et al., 2013), or distant supervision relying on external knowledge bases (De Sa et al., 2016). In this paper, we leverage data programming (Ratner et al., 2016) which uses a generative model to integrate expert knowledge expressed in the form of noisy heuristics (labeling functions or LFs) to probabilistically label training data. Since the quality of the training labels is strongly correlated with that of labeling functions, unrefined weak supervision may not always yield optimal performance. Hence, in practice, experts
using data programming often spend several hours iteratively designing new heuristics or refining existing ones (Varma & Ré, 2018). Our work aims to streamline and reduce this effort.
Another closely related issue is that of domain shift (also referred to as knowledge shift). Imbalance in characteristics of source and target populations may render ML models trained on one domain perform poorly on another. This domain shift, especially in the form of population characteristics like gender or race, is prevalent in clinical settings (Thiagarajan et al., 2018). In the context of weak supervision, we are interested in the related problem of knowledge shift. Specifically, we want to train effective ML classifiers while tailoring knowledge of the source population to the target population (e.g. heart disease in young and old patients) via small changes in the LFs.
To this end, we propose WARM (Active Refinement of Weakly Supervised Models), a principled approach towards quick, iterative, and interactive refinement of weakly supervised models1. WARM directs expert attention to specific data points in the training set, which if labeled, would improve the labeling functions and the resulting label model the most. Specifically, at each iteration WARM directs an expert to label the most uncertain data point, and then updates decision parameters of the labeling functions to reduce the uncertainty.
Experiments on multiple real-world medical classification datasets of varying sizes and complexity, reveal that with as few as 30 queries to an expert, WARM can significantly improve label model accuracy resulting in higher quality training labels. Moreover, WARM outperforms existing approaches to active weak supervision (Biegel et al., 2021; Nashaat et al., 2018) and supports interpretability by allowing experts to track changes in LFs. Moreover, auxiliary experiments on a subset of our datasets show that WARM can not only combat knowledge shift but also de-noise LFs. For example, we found that WARM could adapt LFs to detect the presence of heart disease defined on a young population of patients, to an older population, resulting in a significant 1− 3% increase in label and end model performance. We also found that WARM was able to easily refine noisy LFs to achieve label and end model performance on par with no noise.

heading: 2 RELATED WORK
text: There is a small but growing body of work on active strategies for weak supervision. A few studies combine data programming with active learning on data points. For instance, Nashaat et al. (2018) proposed a method which actively obtained expert-labeled data via uncertainty sampling and corrected LF votes on individual data points to reflect the true labels. While their method improved the probabilistic predictions, it did not change the label model per se. Biegel et al. (2021) recently proposed Active WeaSuL which uses experts labels to learn how to best combine LFs. Specifically, they add a penalty term to the data programming loss function to nudge its optimal parameters to a configuration where the label model predictions agree with the expert labels. They also proposed a novel active sampling strategy tailored to the problem, which splits the data into buckets and measures the KL-divergence between the expert and estimated probabilistic labels to query new labels.
Some other studies have instead used labeled data to inspire ideas for new labeling functions (CohenWang et al., 2019), and yet others have used them to automatically come up with new one (Awasthi et al., 2020; Boecking et al., 2020). Cohen-Wang et al. (2019) for instance proposed two simple strategies (i.e. data points where LFs abstain or disagree the most) to iteratively select subsets of data to inspire domain experts to create new LFs. On the other hand, Boecking et al. (2020) introduced an interactive weak supervision method that automatically improves a set of LFs based on feedback from users.
Alternatively, to reduce human effort to an absolute minimum, some studies have attempted to automate weak supervision by automatically generating heuristics based on a small labeled subset of the data (Varma & Ré, 2018; Nashaat et al., 2020). For instance, Varma & Ré (2018) proposed Snuba which automatically trains decision trees, logistic regression models and k-nearest neighbor classifiers as LFs, tuned for accuracy and coverage (amount of data that the LF predicts a class as oppose to abstaining from predicting any class), and later pruned to ensure sufficient dissimilarity. Finally, Nashaat et al. (2020) proposed Asterisk, extending their prior work on active weak supervision
1We will release the source code for WARM at www.github.com/anonymized_for_review after peer-review.
by now automatically generating heuristics from a small labeled dataset. In our experience, these methods are highly dependent on the quality and size of the labeled validation set. Furthermore, they may require large diverse labeled datasets for complex prediction tasks as evidenced by our experiments on the diabetes dataset. Additionally, Asterisk needs a large amount (up to ∼ 7% of training data) of actively labeled data points for good performance.

heading: 3 METHODOLOGY

heading: 3.1 PROBLEM FORMULATION AND MODELING ASSUMPTIONS
text: Let (x, y) ∼ D denote the data generating distribution, where each data point x ∈ Rd is associated with a latent class label y ∈ {1, . . . , k}. Users provide an unlabeled training dataset Su = {(xi, yi)}ni=1, where yi is unobserved, and m soft unipolar labeling functions (LFs) Γ which we define as follows.
Since we only consider LFs that predict a single class, let pol(f) ∈ {1, . . . , k} define the polarity (or the single class that it predicts) of a LF f . We define Γ = {γτ,j}mj=1, γτ,j : Rd −→ [0, 1], such that γτ,j(xi, c) , 1{c=pol(γτ,j)}p̂(yi = c | xi), where p̂ is any arbitrary scoring function differentiable with respect to its parameters τ .
Specifically, each LF γτ,j depends on one or more parameters in the parameter vector τ and expresses the likelihood that a data point xi belongs to a particular class as identified by the polarity function pol(γτ,j). We drop the subscript τ in the rest of our discussion for the sake of brevity.
3.2 MOST COMMON CLASSES OF LABELING FUNCTIONS ARE DIFFERENTIABLE
Fig. 1(ii) is an example of an expert-defined LF to characterize normal clinical findings in electroencephalogram (EEG) recordings2 which depends on the high baseline parameter.
Our methods assume that all the LFs in the label model are differentiable with respect to their decision parameters. Many frequently used classes of LFs are intrinsically differentiable with respect to their parameters, such as multi-layer perceptrons, convolutional neural networks, log-linear models, neural language models (Hooper et al., 2020; Varma & Ré, 2018). Other common classes of LFs can be easily transformed into their soft counterparts. One example is decision stumps (IF <CONDITION> THEN CLASS <I> ELSE CLASS <J>) (Varma & Ré, 2018). More generally, a AND or OR could also be made differentiable by fuzzy logic. To transform crisp decision stumps into their soft equivalents, we can follow prior work on decision trees (Suárez & Lutsko, 1999). Specifically, consider a heuristic of the form: λ , dᵀx > a, where dᵀx defines a new composite variable and a ∈ τ is a decision parameter. Then we define its fuzzy equivalent as γ , sigmoid(dᵀx − a), where sigmoid(x) =
1 1+exp (−x) . Intuitively, while λ indicates which side of the hyperplane d ᵀx = a a point lies, γ encapsulates a probabilistic notion of distance from the decision boundary. Using the corresponding
2Specifically, the LF is developed to work with amplitude-integrated EEG, which is a widely used quantitative summary of raw multi-channel EEG data.
Zadeh operators, conjunctions are converted via (AND(x, y) → MIN(x, y)), disjunctions are converted via (OR(x, y) → MAX(x, y)) and negations are converted via (NOT(x) → (1 − x)) (Zadeh, 1996). There are still other ways to obtain soft equivalents of crisp LFs, for instance by training feed-forward neural networks to approximate classes of functions under reasonable assumptions on their continuity and topology.
This extension of crisp LFs into their soft counterparts not only enables them to express confidence on their predictions, but also better capture fuzzy decision boundaries, common in most practical settings. Additionally, restricting LFs to be unipolar can be done without any loss of generality, since complex LFs can be easily broken down into their unipolar components. Moreover, in our experience, several simple unipolar LFs tend to perform better than a few complex multipolar LFs in practice. Each soft LF γj can be converted to its hard equivalent λj : Rd −→ {0, pol(λj)} using a threshold function, where 0 means that the LF abstained from voting for class pol(λj). In particular, we have that
λj = { c, if γτ,j(x, c) ≥ 0.5 0, otherwise

heading: 3.3 PROBABILISTIC LABEL ESTIMATION
text: Given the unlabeled training dataset Su and an initial set of LFs denoted by Γ(1), our goal is to efficiently improve the accuracy of LFs by refining their parameters τ based on expert feedback, and ultimately assign probabilistic labels p̂(y | Γ(t),x) to the training data. Here Γ(t) denotes the set of LFs after tth iteration of WARM. The probabilistic labels are used to further train a downstream classifier f : X −→ Y . We refine the LFs automatically by asking domain experts to provide labels for a few highly uncertain data points, and back propagating LF errors to update their parameters τ .
In order to estimate probabilistic labels p(y | Γ(t),x), we propose the following weighted majorityvote label model:
p̂il = bl +
∑m j=1 γ (t) j (xi, l)θ̂
(t) j∑k c=1(bc + ∑m j=1 γ (t) j (xi, c)θ̂ (t) j )
(1)
such that p̂il = p(yi = l | Γ(t),xi) is the estimated probability that xi belongs to class l, while bl = p(y = l) denotes its prevalence. γ (t) j (xi) is the probabilistic vote of j
th LF and θ̂(t)j is its unobserved accuracy, estimated using the data programming (DP) label model as defined in Eq. 2 (Ratner et al., 2016).
We ensure that p̂ik is a probability by passing p̂i through a softmax function, and note that our model is differentiable with respect to all LF parameters τ . The DP label model is not differentiable with respect to LF parameters, and therefore cannot be directly used to estimate probabilistic labels while still having the ability to update LF parameters. However, data programming has been shown to perform well in several weak supervision applications and has some theoretical guarantees for estimating the latent accuracy of LFs by optionally modeling dependencies between them. Thus, in order to iteratively improve the accuracy of LFs and estimate probabilistic labels, we alternate between two objectives at the tth time step:
Estimate the accuracy of LFs keeping their parameters τ fixed. Let [Λ(t)]n×m = {0, 1, . . . , k}n×m denote the observed matrix of crisp LF outputs at the tth time step, such that [Λ(t)]ij = λ (t) j (xi) is the thresholded output of LF γj on xi. We define factors for LF accuracy and propensity as φAcc([Λ(t)]ij , yi) , 1{[Λ(t)]ij=yi} and φ Lab([Λ(t)]ij , yi) , 1{[Λ(t)]ij 6=0}, respectively. We then follow Ratner et al. (2016) to define the joint distribution of the latent class label y and [Λ(t)] as:
pθ(y, [Λ (t)]) ,
1
Zθ exp  m∑ j=1 n∑ i=1 (θjφ Acc([Λ(t)]ij , yi) + θj+mφ Lab([Λ(t)]ij , yi)) + byi  (2)
We use Snorkel (Ratner et al., 2017) to learn θ by minimizing the negative log marginal likelihood given the observed [Λ(t)]:
θ̂(t) = arg min θ n∑ i=1 − log ∑ y∈Y pθ(y, [Λ (t)]i)  (3) We then set the accuracy weights of the LFs to be equal to θ̂(t)j .
Update LF parameters τ keeping their accuracy weights fixed. In this step, we use our label model (Eq. 1) to probabilistically label the training dataset, gather expert labels for the most uncertain data points, and finally update the LF parameters by minimizing the cross-entropy between the observed expert labels and probabilistic label predictions of the data points. In particular, we update the LF parameters τ via gradient back propagation, so as to minimize the negative log likelihood of all the expert labels observed so far:
LCE = ∑
(xi,yi)∈Sl
k∑ l=1 −1{l=yi} log p̂il (4)
where Sl is the set of expert labeled data points. While in our experiments, at each time step t, we gather an expert label for the data point with the highest entropy, our methods are generic and can seamlessly incorporate other active learning acquisition functions. Our algorithm is briefly summarised in Algorithm 1.
Algorithm 1 WARM: The proposed active weak supervision algorithm. 1: procedure WARM([Γ(1)],Su = {(xi, yi)}ni=1,b) 2: for t ∈ 1 to T do 3: [Λ(t)]← c([Γ(t)]) . Convert to crisp votes
4: θ̂(t) ← arg min θ − log ( ∑ Y ∈Y pθ(Y, [Λ (t)]) ) 5: for i← 1 to |Su| do 6: p̂il ← bl + ∑m j=1 γ (t) j (xi, c)θ̂ (t) j ,∀l ∈ {1, . . . , k} . Aggregate weighted LF votes 7: p̂i ← softmax(p̂i) 8: Ei ← − ∑k l=1 p̂il log(p̂il) . Compute entropies
9: end for 10: yq ← query user({xq|q ← arg max
i∈Su E}) . Query user for label
11: Sl ← Sl ∪ (xq, yq); Su ← Su/(xq, yq) 12: LCE ← ∑ (xi,yi)∈Sl ∑k l=1−1{l=yi} log p̂il 13: Γ(t+1) ← backprop errors(Γ(t),LCE) . Update LF parameters 14: end for 15: end procedure

heading: 4 EXPERIMENTS

heading: 4.1 DATASETS AND WEAK SUPERVISION SOURCES
text: Like most prior work on active and automated weak supervision, we primarily focus on the binary classification setting for consistency (Varma & Ré, 2018; Biegel et al., 2021; Nashaat et al., 2018). Specifically, we carry out experiments on various real-world clinical classification datasets of varying complexity and sizes. See Table 1 for a summary of all the datasets used. All of the above datasets were taken from the UCI Machine Learning Repository (Dua & Graff, 2017). Refer Appendix A.1 for dataset details.
Since we did not have the requisite clinical expertise to define good weak supervision sources, we trained a Random Forest (RF) classifier of decision stumps (depth-one decision trees) to automatically to simulate expert heuristics for simple LFs. Specifically, we trained a Random Forest
classifier with 5 or 10 decision stumps on a 50% sub-sample of the training set, such that each decision stump served as a LF. We used the Random Forest Classifier implementation from the Scikit-Learn (Pedregosa et al., 2011) Python library for the same. For all datasets, we acquired a set of 10 LFs, with the exception of the diabetes dataset for which we used a set of 5 LFs only. Appendix A.6 contains examples of LFs for the Heart Disease dataset.
As in Biegel et al. (2021), we also benchmark WARM on a synthetic dataset comprising of two classes modeled by 2-dimensional isotropic Gaussians, with a training and testing set of 10,000 and 3,000 data points, respectively. We use the same set of 3 LFs as Biegel et al. (2021) for our experiments.
Finally, to subject WARM to even more realistic settings, we used it to iteratively refine expert-defined heuristics to classify clinical findings in electroencephalogram (EEG) recordings. The dataset comprised of continuous 6-hour amplitude-integrated EEG (aEEG)3 recordings from 1310 comatose patients resuscitated after suffering from cardiac arrest, and admitted to the Intensive Care Unit of a large tertiary care hospital between February 2010 and April 2019. The dataset was jointly annotated by two experts in cardiac arrest care and clinical EEG interpretation, using a simplified multinomial labeling convention to summarize EEG findings that occur commonly post cardiac arrest. Specifically, each raw EEG record was broken into six disjoint 1-hour windows and classified as generalized suppression; burst suppression with epileptiform activity (including burst suppression with identical bursts); burst suppression without epileptiform activity; and, near-continuous or continuous background activity (Elmer et al., 2016; 2020). We also developed a total of 17 LFs tailored to the prediction problem with help from one of the expert clinicians.

heading: 4.2 COMPARISON OF METHODS
text: We compared WARM with the two existing active weak supervision approaches, namely Active WeaSuL (Biegel et al., 2021) and the method proposed by Nashaat et al. (2018). All the methods had access to the same set of initial LFs and were run for 30 active learning iterations, where they were allowed to query one ground truth label from the training set at each iteration. We also benchmarked WARM against Snuba (Varma & Ré, 2018), a popular weak supervision approach which automatically synthesizes weak supervision sources from a small subset of labeled examples incorporates them with the data programming label model to probabilistically label data. We ran Snuba with a random labeled subset of 10, 20 and 30 data points with decision tree stumps as heuristics. To report test set performance, we trained downstream logistic regression models on the crisp versions of the probabilistic labels assigned by each method. We also compared these methods against a logistic regression (LR) model trained using a vanilla active learning sampling strategy based on the distance from the decision boundary as in Biegel et al. (2021), and a fully-supervised one trained using the entire labeled training set. The fully supervised LR model serves as a proxy for the maximum achievable performance with the given features, train-test split and downstream classifier.
Since existing online implementations of Snuba, Active WeaSuL and the method proposed by Nashaat et al. (2018) were designed exclusively for binary classification problems, we were only able to compare WARM with active learning and fully supervised baselines on the EEG dataset involving multi-class classification. For all our experiments, we used the logistic regression model implementation from Scikit-Learn (Pedregosa et al., 2011), trained for a maximum of 500 iterations and optimized using LBFGS. WARM models were trained and built using PyTorch 1.8.1 Paszke et al. (2019) along with Python 3.8.1. All our experiments were carried out on a Linux desktop with a typical machine having 8 Intel(R) Core(TM) i7-6700K CPUs and 16 GB of RAM.
3aEEG is a widely used quantitative summary of multi-channel EEG data.

heading: 4.3 EXPERIMENTAL DESIGN
text: We aim to answer the following research questions through our experiments.
Does WARM improve the quality of the training dataset and how does this improvement translate to downstream classifier performance? We evaluate the quality of the training dataset by comparing the crisp versions of the probabilistic labels assigned by all the weakly supervised approaches with the ground truth training labels.
Can WARM combat the problem of knowledge shift and overcome noise in the initial LFs? To this end, we carry out experiments on the synthetic and Heart Disease dataset. To simulate knowledge shift in the synthetic dataset, we start with the three LFs tailored to the original (source) data distribution, and introduce two other (target) gaussian clusters with different centroids. For the heart disease dataset, we automatically acquire LFs from the young population i.e. all patients with less than median age. We then train and evaluate WARM on disjoint subsets of old patient population. Since weakly supervised models rely heavily on the quality of its weak supervision sources, we conduct additional experiments on the Heart Disease dataset by adding varying levels of uniform random noise, scaled by the range of the features, to the LF decision parameters initially obtained using the random forest. I.e. 100% noise added could mean that the decision stump threshold is off by 100%.

heading: 5 RESULTS AND DISCUSSION
text: Table 2 summarizes the results of our experiments. The Train results represent the performance of the weak label models in comparison to ground truth training labels, and hence are a direct measure of label model quality. The Test results on the other hand, are from LR models trained on the crisp equivalents of the probabilistic labels and evaluated on the held-out test set. The latter set of results primarily reflect the quality of the downstream models trained using weak labels. We refer the
interested reader to Appendix A.7 for supplementary results with additional metrics such as F1 and percentage change in accuracy. Our primary findings from the experiments are summarized below.
WARM can improve the quality of training data. We found that on all the datasets, the probabilistic labels assigned post active weak supervision via WARM are significantly more accurate than the ones assigned by compared methods. However, we did observe some variance in accuracy and its improvement across different datasets, primarily due to the nature of the data and the quality of initial weak supervision sources. For instance on the breast cancer data, WARM improved the accuracy of the label model by over 7% across 30 active learning iterations. On the other hand, WARM could not improve the train generative performance on the heart disease data, but it also starts from a better higher accuracy. We observed similar trends for downstream model performance, albeit with smaller accuracy improvements. The smaller accuracy improvements can be attributed to the generalization capabilities of the downstream model over the labels it is trained on. Nevertheless, our findings underscore the importance of high quality labeled training sets. On the EEG dataset (Table 2 and Fig. 6–7), WARM achieved a 4 − 5% increase in both label model and downstream classifier performance. These results are significant since even the initial set of LFs were developed by expert clinicians.
Moreover, we also found that traditional Active Learning strategy outperformed WARM on simpler classification tasks such as breast cancer diagnosis. However, for more complex prediction problems such as the predicting readmission on the Diabetes dataset and classifying clinical findings on the EEG dataset, active weak supervision via WARM significantly outperformed end models trained using active learning. Finally, experiment runtimes suggest that WARM is significantly faster than existing active weak supervision techniques across datasets (Appendix A.5).
WARM can help combat knowledge shift and de-noise LFs. Results from knowledge shift experiments on the heart disease and diabetes dataset in Fig. 2 and Table 4,5 reveal that WARM can help both weak labelers and the downstream classifier adapt to different population characteristics. We observed approximately a 1− 3% and 4− 6% increase in label and end model performance on the heart disease and synthetic datasets, respectively. See Appendix A.6 for all LFs defined on the heart disease dataset to detect the presence of heart disease in the young and old population.
Fig. 3(ii) and Tab. 6 summarizes the performance of WARM across 50 active learning iterations, when varying levels of random noise is artificially added to LFs defined on the Wisconsin Diagnostic Breast Cancer dataset. Our results clearly show that noisy LFs hurt both label and end model performance. However, WARM can iteratively de-noise LFs by actively collecting a few more labeled examples. Even with 75% noise initially, WARM can refine LFs such that the resulting label and end model perform on par with those trained using LFs with no noise. Additionally, WARM consistently outperforms the other active weak supervision baselines4.
Changes to LFs brought about by WARM are interpretable. If the LFs themselves are inherently interpretable, WARM supports their interpretability by enabling domain experts to track changes in their decision parameters. Fig. 3(ii) for instance shows LFs before and after refinement by WARM on the synthetic dataset. We also draw the reader’s attention to the knowledge shift experiments on the heart disease dataset for more examples of interpretable changes to LF parameters. Appendix A.6 lists all the LFs tailored to the young and old population. For example, the rule If exercise induced angina <= 144.666(150.123) then ABSENCE else PRESENCE clearly says that while exercise induced angina higher than 144.67 was indicative of heart disease in the young population, older people tend to naturally have higher values resulting in a higher decision threshold.
Ablation experiments on WARM shows that it benefits from both active learning and weak supervision. We also perform ablation experiments to investigate if WARM is performing as expected. In Appendix A.8, we see that WARM directly benefits from Uncertainty Sampling from its active learning component, outperforming the random sampling in every case. From Appendix A.10,A.9 it is generally able to generally outperform Active Learning on complex or large datasets due to its Weak Supervision component.
4Note that active learning results are not affected because we only add noise to LFs, which are only used by weak supervision methods

heading: 6 CONCLUSION
text: With more medical data being collected, methods that address the lack of labeled data in clinical applications of machine learning are becoming increasingly vital. If the vast amounts of unlabeled clinical data could be utilized to its full potential, the possibility of augmenting physicians with ML to increase positive patient outcomes is substantial. In addition, a lot of of useful information can be provided to ML systems from expert guidance, whether it be via heuristic design or via handlabeling individual data points; making such capabilities efficient and effective is greatly needed in practice. WARM (Active Refinement of Weakly Supervised Models) is a fast active learning addition to data programming that substantially and inexpensively improves the empirical accuracy of training labels and downstream classifiers. Our experiments show that WARM has the capability to address the common problems of knowledge shift and rule denoising with minimal additional labeled data. If ran on interpretable LFs, WARM is even able to reveal strategic, data-driven, interpretable updates to the LF thresholds and parameters. Limitations of WARM include its reliance on differentiable and reasonably accurate initial LFs, which may occasionally be more expensive to acquire than pointillistic labels. However, we have shown that most frequently used classes of LFs are intrinsically differentiable or can be made so. WARM is already a step towards more efficient and effective algorithms to tackle labeled-data-scarce ML problems in healthcare and beyond.
Additional experiments would benefit precise assessment of properties and scoping of practical utility of WARM in a wider range of its possible applications. Future work should also involve user studies to gauge attainable efficiency gains and utility of interpretable adjustments to the LFs, and whether they meaningfully boost perceived usability of the weak supervision process in practice. Other future directions could include using meta-AL to learn query selection strategies based off of previous AL outcomes as in (Konyushkova et al., 2017; Bachman et al., 2017) or AL with partial feedback approaches as in (Hu et al., 2018). Another challenge to address is mitigating overfitting to the active learning samples; we could perform early stopping, learning rate decay, gradient clipping, or cross validation to better generalize our models. Furthermore, we would like to research the connection between our framework and automated (Varma & Ré, 2018) or active (Boecking et al., 2020) discovery of new LFs.

heading: A APPENDIX
text: A.1 DATASET DETAILS
The Heart Disease dataset (Detrano et al., 1989) contains clinical and noninvasive test results from patients undergoing angiography at the Cleveland Clinic in Cleveland, Ohio, Hungarian Institute of Cardiology in Budapest, Hungary, Veterans Administration Medical Center in Long Beach, California; and University Hospitals in Zurich and Basel, Switzerland. For our experiments, we concatenate all the databases into one and predict whether or not an individual has heart disease. The Diabetes dataset (Strack et al., 2014) comprises of clinical records from over million patients admitted to 130 US hospitals and integrated delivery networks between 1999 to 2008. Given features representing patients such as their age, race, gender, time in hospital, diagnosis etc., the goal is to predict whether they are re-admitted for further treatment. We also use two breast cancer datasets, namely Wisconsin Diagnostic Breast Cancer dataset (Street et al., 1993) and Wisconsin Breast Cancer Database (Mangasarian & Wolberg, 1990). For both the datasets, the prediction task is to distinguish between samples of malignant and benign breast cancer using different sets of features. The diagnostic dataset comprises of features characterizing cell nuclei present in a digitized image of a fine needle aspirate (FNA) of a breast mass. The latter dataset instead contains 9 cytological characteristics of breast FNA graded between 1–10 at the time of sample collection.
A.2 EEG DATA COLLECTION
EEG monitoring was initiated a median of 10 hours after initial collapse as standard routine of care on arrival to the intensive care unit (Elmer et al., 2016). EEG was recorded using XLTech Natus Neuroworks digital video/EEG systems (Natus Medical Inc) at 256Hz from 22 gold-plated cup electrodes placed in the international 10-20 system positions.
Löfhede et al. (2008) has shown near-perfect spatial correlation of many clinically important EEG features in patients after cardiac arrest, allowing the 22-sample signal to be down-sampled without significant information loss. Additionally, after cardiac arrest, many prognostic clinical findings can
be identified accurately from lower-resolution quantitative summaries (e.g. 1Hz aEEG, Fig. 4) (Oh et al., 2015; Glass et al., 2013). Thus, to optimize computational efficiency, we aimed to develop similar heuristics as labeling functions to classify raw EEG signals from contemporaneous aEEG summaries.
For our experiments, we used 6-hour aEEG time series starting at the initiation of EEG monitoring from 1310 patients with mean age 57 years of whom 503 (38%) were female. Approximately 30% survived to discharge from the hospital. A large subset of our data (approximately 70%) was initially jointly annotated by two experts in cardiac arrest care and clinical EEG interpretation at UPMC. They manually annotated EEG findings using an adaptation of the 2016 American Clinical Neurophysiology Society guidelines (Hirsch et al., 2013).
Based on prior research and our expert clinicians, we developed a simplified multinomial labeling convention to summarize prognostic EEG findings that occur commonly after cardiac arrest. Specifically, we classified the raw EEG waveforms as: generalized suppression; burst suppression with epileptiform activity (including burst suppression with identical bursts); burst suppression without epileptiform activity; and, near-continuous or continuous background activity (Elmer et al., 2020). Because continuous background activity with epileptiform activity occurred infrequently in this data set, we did not further subdivide categories of patients with continuous background activity.
A.3 KNOWLEDGE SHIFT EXPERIMENTS
A.4 ADDING ARTIFICIAL NOISE TO LFS
A.5 RUNTIMES
A.6 EXAMPLE RULES
On the heart disease dataset, we carried out our knowledge shift experiments, where the initial labeling functions were based on the young population, but the target population was that of old patients. Below is a list of initial rules tailored to the young patients. The decision parameters in parenthesis reflect the modified labeling function parameters, tailored to older patients.

heading: If Chest pain type <= 0.898(0.978) then ABSENCE else PRESENCE

heading: If Serum cholesterol in mg/dl <= 124.057(119.793) then PRESENCE else ABSENCE

heading: If Maximum heart rate achieved <= 0.244(0.079) then PRESENCE else ABSENCE

heading: If Exercise induced angina <= 144.666(150.123) then ABSENCE else PRESENCE

heading: If thal <= 0.646(0.796) then ABSENCE else PRESENCE

heading: If Chest pain type <= 0.898(0.978) then ABSENCE else PRESENCE

heading: If ST depression induced by exercise relative to rest <= 0.273(0.383)
text: then ABSENCE else PRESENCE

heading: If Exercise induced angina <= 144.666(150.123) then ABSENCE else PRESENCE

heading: If thal <= 0.646(0.796) then ABSENCE else PRESENCE
text: If Maximum heart rate achieved <= 0.533(0.367) then PRESENCE else ABSENCE
A.7 WARM COMPARISON TO OTHER MODELS
A.8 WARM RANDOM SAMPLING VS UNCERTAINTY SAMPLING
A.9 WARM ABLATION EXPERIMENTS ON DATASET SIZE AND DOWNSTREAM MODELS
For the rest of our experiments, we created synthetic datasets using sklearn’s make classification5 method. The datasets comprise of clusters of points normally distributed (with standard deviation 1) about vertices of a 25-dimensional hypercube with sides of length 2, with an equal number of clusters (= 5) to each class. We used 25 decision tree stumps as LFs. In order to automatically create the simple LFs, we followed the same procedure outlined in Section 4.1, with the only difference being that the Random Forest classifier was trained using 250 data points sampled from the training dataset. This was done to ensure that WARM received the same amount of weak supervision regardless of the size of the dataset, and the differences in performance were solely due to the size of the dataset and not the quality of weak supervision.
5https://scikit-learn.org/stable/modules/generated/sklearn.datasets. make_classification.html
A.10 WARM EXPERIMENTS ON COMPLEXITY OF DATASET
The following results are reported on a datasets with 10000 samples and varying numbers of clusters per class, created using sklearn’s make classification method. The number of clusters per class controls the complexity of the problem, and more clusters per class is a harder classification problem than fewer clusters per class. The weak supervision sources were created in exactly the same way as mentioned in the previous section.

Review summaries:
In this work, the authors propose the WARM method to help conduct iterative and interactive weakly-supervised learning. Active learning is used here to refine the labeling functions by focusing on data points that are once labeled. The authors further incorporated gradient propagation to alternatively update the LF parameters and the DP model. Experimental results show that the WARM method can improve the quality of training data.

This paper proposes WARM, an active learning approach to weakly/programmatically supervised learning.  In the WARM approach, which bases off of the data programming/Snorkel paradigm for weak supervision, users write labeling functions (LFs) to programmatically label training data; these labeling functions are then modeled by the Snorkel framework for weak supervision and used to train downstream models.  In the WARM setup, these LFs are assumed to be, or cast as, differentiable.  The paper then proposes an active learning approach to sampling labeled data points to tune the parameters of these LFs, and validates this approach on several medical datasets.

This paper proposes an algorithm for choosing a small set of labels to improve labeling function model performance both directly and for downstream tasks. Additionally, the authors provide a general method to convert standard labeling functions to "soft" labeling functions which are differentiable with respect to some parameters (e.g. a threshold). If the labeling functions are differentiable, this paper provides a method to update the labeling function parameters. Finally, experimental results show that the method introduced outperforms other active labeling approaches for weak supervision.

The paper gives a method to iteratively and interactively improve the label model in weak supervision. The approach consists of two steps, first is standard weak supervision way of weighted combination of labelling functions to generate labels. Novelty and improvement mainly comes from the second step, where true label for most uncertain data point is queried using which the parameters of the labelling functions are improved, which in turn lead to a more accurate weak supervision model. A key requirement and assumption in this paper's setup is that labelling functions are given by some learnable parameters ( i.e. they can be differentiated w.r.t. their parameters), which allows parameters updates using the true labels acquired. Empirical results on various real world datasets in medical domain show that in some cases this approach can yield a more accurate model in comparison to pure active learning approach and some recent baselines which combine weak supervision with active learning. These results also show that the paper's approach can get accuracy comparable to fully supervised model as well.