ACTIVE REFINEMENT OF WEAKLY SUPERVISED MODELS

1 INTRODUCTION . Machine learning ( ML ) has seen widespread adoption in several domains such as health , education and public policy . In the clinical context , supervised ML algorithms have been shown to be effective solutions to a gamut of problems , ranging from detecting abnormal heart rhythms in electrocardiogram ( Goswami et al. , 2022 ; Hannun et al. , 2019 ) , prognosticating neurological recovery using continuous electroencephalogram ( Elmer et al. , 2016 ; 2020 ) to detecting lung cancer in CT scans ( Gao et al. , 2019 ) . However , most modern ML algorithms , especially large deep neural networks , rely on vast quantities of pointillistically labeled training data . In our experience , while raw clinical data is abundant , its careful annotation is not . Manually labeling large quantities of clinical data is often tedious , prohibitively expensive , prone to error , and acutely unscalable . Recently , many studies have explored the use of cheap , potentially noisy sources of supervision to noisily annotate large training sets ( Goswami et al. , 2022 ; Fries et al. , 2019 ; Saab et al. , 2020 ) using weak supervision . These methods combine these sources to create noisy labels on unseen data , which can then be used to learn powerful discriminative end models . In principle , these methods resemble other repeated-labeling techniques such as crowd-sourcing ( Li et al. , 2013 ) , or distant supervision relying on external knowledge bases ( De Sa et al. , 2016 ) . In this paper , we leverage data programming ( Ratner et al. , 2016 ) which uses a generative model to integrate expert knowledge expressed in the form of noisy heuristics ( labeling functions or LFs ) to probabilistically label training data . Since the quality of the training labels is strongly correlated with that of labeling functions , unrefined weak supervision may not always yield optimal performance . Hence , in practice , experts using data programming often spend several hours iteratively designing new heuristics or refining existing ones ( Varma & Ré , 2018 ) . Our work aims to streamline and reduce this effort . Another closely related issue is that of domain shift ( also referred to as knowledge shift ) . Imbalance in characteristics of source and target populations may render ML models trained on one domain perform poorly on another . This domain shift , especially in the form of population characteristics like gender or race , is prevalent in clinical settings ( Thiagarajan et al. , 2018 ) . In the context of weak supervision , we are interested in the related problem of knowledge shift . Specifically , we want to train effective ML classifiers while tailoring knowledge of the source population to the target population ( e.g . heart disease in young and old patients ) via small changes in the LFs . To this end , we propose WARM ( Active Refinement of Weakly Supervised Models ) , a principled approach towards quick , iterative , and interactive refinement of weakly supervised models1 . WARM directs expert attention to specific data points in the training set , which if labeled , would improve the labeling functions and the resulting label model the most . Specifically , at each iteration WARM directs an expert to label the most uncertain data point , and then updates decision parameters of the labeling functions to reduce the uncertainty . Experiments on multiple real-world medical classification datasets of varying sizes and complexity , reveal that with as few as 30 queries to an expert , WARM can significantly improve label model accuracy resulting in higher quality training labels . Moreover , WARM outperforms existing approaches to active weak supervision ( Biegel et al. , 2021 ; Nashaat et al. , 2018 ) and supports interpretability by allowing experts to track changes in LFs . Moreover , auxiliary experiments on a subset of our datasets show that WARM can not only combat knowledge shift but also de-noise LFs . For example , we found that WARM could adapt LFs to detect the presence of heart disease defined on a young population of patients , to an older population , resulting in a significant 1− 3 % increase in label and end model performance . We also found that WARM was able to easily refine noisy LFs to achieve label and end model performance on par with no noise . 2 RELATED WORK . There is a small but growing body of work on active strategies for weak supervision . A few studies combine data programming with active learning on data points . For instance , Nashaat et al . ( 2018 ) proposed a method which actively obtained expert-labeled data via uncertainty sampling and corrected LF votes on individual data points to reflect the true labels . While their method improved the probabilistic predictions , it did not change the label model per se . Biegel et al . ( 2021 ) recently proposed Active WeaSuL which uses experts labels to learn how to best combine LFs . Specifically , they add a penalty term to the data programming loss function to nudge its optimal parameters to a configuration where the label model predictions agree with the expert labels . They also proposed a novel active sampling strategy tailored to the problem , which splits the data into buckets and measures the KL-divergence between the expert and estimated probabilistic labels to query new labels . Some other studies have instead used labeled data to inspire ideas for new labeling functions ( CohenWang et al. , 2019 ) , and yet others have used them to automatically come up with new one ( Awasthi et al. , 2020 ; Boecking et al. , 2020 ) . Cohen-Wang et al . ( 2019 ) for instance proposed two simple strategies ( i.e . data points where LFs abstain or disagree the most ) to iteratively select subsets of data to inspire domain experts to create new LFs . On the other hand , Boecking et al . ( 2020 ) introduced an interactive weak supervision method that automatically improves a set of LFs based on feedback from users . Alternatively , to reduce human effort to an absolute minimum , some studies have attempted to automate weak supervision by automatically generating heuristics based on a small labeled subset of the data ( Varma & Ré , 2018 ; Nashaat et al. , 2020 ) . For instance , Varma & Ré ( 2018 ) proposed Snuba which automatically trains decision trees , logistic regression models and k-nearest neighbor classifiers as LFs , tuned for accuracy and coverage ( amount of data that the LF predicts a class as oppose to abstaining from predicting any class ) , and later pruned to ensure sufficient dissimilarity . Finally , Nashaat et al . ( 2020 ) proposed Asterisk , extending their prior work on active weak supervision 1We will release the source code for WARM at www.github.com/anonymized_for_review after peer-review . by now automatically generating heuristics from a small labeled dataset . In our experience , these methods are highly dependent on the quality and size of the labeled validation set . Furthermore , they may require large diverse labeled datasets for complex prediction tasks as evidenced by our experiments on the diabetes dataset . Additionally , Asterisk needs a large amount ( up to ∼ 7 % of training data ) of actively labeled data points for good performance . 3 METHODOLOGY . 3.1 PROBLEM FORMULATION AND MODELING ASSUMPTIONS . Let ( x , y ) ∼ D denote the data generating distribution , where each data point x ∈ Rd is associated with a latent class label y ∈ { 1 , . . . , k } . Users provide an unlabeled training dataset Su = { ( xi , yi ) } ni=1 , where yi is unobserved , and m soft unipolar labeling functions ( LFs ) Γ which we define as follows . Since we only consider LFs that predict a single class , let pol ( f ) ∈ { 1 , . . . , k } define the polarity ( or the single class that it predicts ) of a LF f . We define Γ = { γτ , j } mj=1 , γτ , j : Rd −→ [ 0 , 1 ] , such that γτ , j ( xi , c ) , 1 { c=pol ( γτ , j ) } p̂ ( yi = c | xi ) , where p̂ is any arbitrary scoring function differentiable with respect to its parameters τ . Specifically , each LF γτ , j depends on one or more parameters in the parameter vector τ and expresses the likelihood that a data point xi belongs to a particular class as identified by the polarity function pol ( γτ , j ) . We drop the subscript τ in the rest of our discussion for the sake of brevity . 3.2 MOST COMMON CLASSES OF LABELING FUNCTIONS ARE DIFFERENTIABLE Fig . 1 ( ii ) is an example of an expert-defined LF to characterize normal clinical findings in electroencephalogram ( EEG ) recordings2 which depends on the high baseline parameter . Our methods assume that all the LFs in the label model are differentiable with respect to their decision parameters . Many frequently used classes of LFs are intrinsically differentiable with respect to their parameters , such as multi-layer perceptrons , convolutional neural networks , log-linear models , neural language models ( Hooper et al. , 2020 ; Varma & Ré , 2018 ) . Other common classes of LFs can be easily transformed into their soft counterparts . One example is decision stumps ( IF < CONDITION > THEN CLASS < I > ELSE CLASS < J > ) ( Varma & Ré , 2018 ) . More generally , a AND or OR could also be made differentiable by fuzzy logic . To transform crisp decision stumps into their soft equivalents , we can follow prior work on decision trees ( Suárez & Lutsko , 1999 ) . Specifically , consider a heuristic of the form : λ , dᵀx > a , where dᵀx defines a new composite variable and a ∈ τ is a decision parameter . Then we define its fuzzy equivalent as γ , sigmoid ( dᵀx − a ) , where sigmoid ( x ) = 1 1+exp ( −x ) . Intuitively , while λ indicates which side of the hyperplane d ᵀx = a a point lies , γ encapsulates a probabilistic notion of distance from the decision boundary . Using the corresponding 2Specifically , the LF is developed to work with amplitude-integrated EEG , which is a widely used quantitative summary of raw multi-channel EEG data . Zadeh operators , conjunctions are converted via ( AND ( x , y ) → MIN ( x , y ) ) , disjunctions are converted via ( OR ( x , y ) → MAX ( x , y ) ) and negations are converted via ( NOT ( x ) → ( 1 − x ) ) ( Zadeh , 1996 ) . There are still other ways to obtain soft equivalents of crisp LFs , for instance by training feed-forward neural networks to approximate classes of functions under reasonable assumptions on their continuity and topology . This extension of crisp LFs into their soft counterparts not only enables them to express confidence on their predictions , but also better capture fuzzy decision boundaries , common in most practical settings . Additionally , restricting LFs to be unipolar can be done without any loss of generality , since complex LFs can be easily broken down into their unipolar components . Moreover , in our experience , several simple unipolar LFs tend to perform better than a few complex multipolar LFs in practice . Each soft LF γj can be converted to its hard equivalent λj : Rd −→ { 0 , pol ( λj ) } using a threshold function , where 0 means that the LF abstained from voting for class pol ( λj ) . In particular , we have that λj = { c , if γτ , j ( x , c ) ≥ 0.5 0 , otherwise

Summary: This paper proposes WARM, an active learning approach to weakly/programmatically supervised learning.  In the WARM approach, which bases off of the data programming/Snorkel paradigm for weak supervision, users write labeling functions (LFs) to programmatically label training data; these labeling functions are then modeled by the Snorkel framework for weak supervision and used to train downstream models.  In the WARM setup, these LFs are assumed to be, or cast as, differentiable.  The paper then proposes an active learning approach to sampling labeled data points to tune the parameters of these LFs, and validates this approach on several medical datasets.