ACTIVE REFINEMENT OF WEAKLY SUPERVISED MODELS

1 INTRODUCTION . Machine learning ( ML ) has seen widespread adoption in several domains such as health , education and public policy . In the clinical context , supervised ML algorithms have been shown to be effective solutions to a gamut of problems , ranging from detecting abnormal heart rhythms in electrocardiogram ( Goswami et al. , 2022 ; Hannun et al. , 2019 ) , prognosticating neurological recovery using continuous electroencephalogram ( Elmer et al. , 2016 ; 2020 ) to detecting lung cancer in CT scans ( Gao et al. , 2019 ) . However , most modern ML algorithms , especially large deep neural networks , rely on vast quantities of pointillistically labeled training data . In our experience , while raw clinical data is abundant , its careful annotation is not . Manually labeling large quantities of clinical data is often tedious , prohibitively expensive , prone to error , and acutely unscalable . Recently , many studies have explored the use of cheap , potentially noisy sources of supervision to noisily annotate large training sets ( Goswami et al. , 2022 ; Fries et al. , 2019 ; Saab et al. , 2020 ) using weak supervision . These methods combine these sources to create noisy labels on unseen data , which can then be used to learn powerful discriminative end models . In principle , these methods resemble other repeated-labeling techniques such as crowd-sourcing ( Li et al. , 2013 ) , or distant supervision relying on external knowledge bases ( De Sa et al. , 2016 ) . In this paper , we leverage data programming ( Ratner et al. , 2016 ) which uses a generative model to integrate expert knowledge expressed in the form of noisy heuristics ( labeling functions or LFs ) to probabilistically label training data . Since the quality of the training labels is strongly correlated with that of labeling functions , unrefined weak supervision may not always yield optimal performance . Hence , in practice , experts using data programming often spend several hours iteratively designing new heuristics or refining existing ones ( Varma & Ré , 2018 ) . Our work aims to streamline and reduce this effort . Another closely related issue is that of domain shift ( also referred to as knowledge shift ) . Imbalance in characteristics of source and target populations may render ML models trained on one domain perform poorly on another . This domain shift , especially in the form of population characteristics like gender or race , is prevalent in clinical settings ( Thiagarajan et al. , 2018 ) . In the context of weak supervision , we are interested in the related problem of knowledge shift . Specifically , we want to train effective ML classifiers while tailoring knowledge of the source population to the target population ( e.g . heart disease in young and old patients ) via small changes in the LFs . To this end , we propose WARM ( Active Refinement of Weakly Supervised Models ) , a principled approach towards quick , iterative , and interactive refinement of weakly supervised models1 . WARM directs expert attention to specific data points in the training set , which if labeled , would improve the labeling functions and the resulting label model the most . Specifically , at each iteration WARM directs an expert to label the most uncertain data point , and then updates decision parameters of the labeling functions to reduce the uncertainty . Experiments on multiple real-world medical classification datasets of varying sizes and complexity , reveal that with as few as 30 queries to an expert , WARM can significantly improve label model accuracy resulting in higher quality training labels . Moreover , WARM outperforms existing approaches to active weak supervision ( Biegel et al. , 2021 ; Nashaat et al. , 2018 ) and supports interpretability by allowing experts to track changes in LFs . Moreover , auxiliary experiments on a subset of our datasets show that WARM can not only combat knowledge shift but also de-noise LFs . For example , we found that WARM could adapt LFs to detect the presence of heart disease defined on a young population of patients , to an older population , resulting in a significant 1− 3 % increase in label and end model performance . We also found that WARM was able to easily refine noisy LFs to achieve label and end model performance on par with no noise . 2 RELATED WORK . There is a small but growing body of work on active strategies for weak supervision . A few studies combine data programming with active learning on data points . For instance , Nashaat et al . ( 2018 ) proposed a method which actively obtained expert-labeled data via uncertainty sampling and corrected LF votes on individual data points to reflect the true labels . While their method improved the probabilistic predictions , it did not change the label model per se . Biegel et al . ( 2021 ) recently proposed Active WeaSuL which uses experts labels to learn how to best combine LFs . Specifically , they add a penalty term to the data programming loss function to nudge its optimal parameters to a configuration where the label model predictions agree with the expert labels . They also proposed a novel active sampling strategy tailored to the problem , which splits the data into buckets and measures the KL-divergence between the expert and estimated probabilistic labels to query new labels . Some other studies have instead used labeled data to inspire ideas for new labeling functions ( CohenWang et al. , 2019 ) , and yet others have used them to automatically come up with new one ( Awasthi et al. , 2020 ; Boecking et al. , 2020 ) . Cohen-Wang et al . ( 2019 ) for instance proposed two simple strategies ( i.e . data points where LFs abstain or disagree the most ) to iteratively select subsets of data to inspire domain experts to create new LFs . On the other hand , Boecking et al . ( 2020 ) introduced an interactive weak supervision method that automatically improves a set of LFs based on feedback from users . Alternatively , to reduce human effort to an absolute minimum , some studies have attempted to automate weak supervision by automatically generating heuristics based on a small labeled subset of the data ( Varma & Ré , 2018 ; Nashaat et al. , 2020 ) . For instance , Varma & Ré ( 2018 ) proposed Snuba which automatically trains decision trees , logistic regression models and k-nearest neighbor classifiers as LFs , tuned for accuracy and coverage ( amount of data that the LF predicts a class as oppose to abstaining from predicting any class ) , and later pruned to ensure sufficient dissimilarity . Finally , Nashaat et al . ( 2020 ) proposed Asterisk , extending their prior work on active weak supervision 1We will release the source code for WARM at www.github.com/anonymized_for_review after peer-review . by now automatically generating heuristics from a small labeled dataset . In our experience , these methods are highly dependent on the quality and size of the labeled validation set . Furthermore , they may require large diverse labeled datasets for complex prediction tasks as evidenced by our experiments on the diabetes dataset . Additionally , Asterisk needs a large amount ( up to ∼ 7 % of training data ) of actively labeled data points for good performance . 3 METHODOLOGY . 3.1 PROBLEM FORMULATION AND MODELING ASSUMPTIONS . Let ( x , y ) ∼ D denote the data generating distribution , where each data point x ∈ Rd is associated with a latent class label y ∈ { 1 , . . . , k } . Users provide an unlabeled training dataset Su = { ( xi , yi ) } ni=1 , where yi is unobserved , and m soft unipolar labeling functions ( LFs ) Γ which we define as follows . Since we only consider LFs that predict a single class , let pol ( f ) ∈ { 1 , . . . , k } define the polarity ( or the single class that it predicts ) of a LF f . We define Γ = { γτ , j } mj=1 , γτ , j : Rd −→ [ 0 , 1 ] , such that γτ , j ( xi , c ) , 1 { c=pol ( γτ , j ) } p̂ ( yi = c | xi ) , where p̂ is any arbitrary scoring function differentiable with respect to its parameters τ . Specifically , each LF γτ , j depends on one or more parameters in the parameter vector τ and expresses the likelihood that a data point xi belongs to a particular class as identified by the polarity function pol ( γτ , j ) . We drop the subscript τ in the rest of our discussion for the sake of brevity . 3.2 MOST COMMON CLASSES OF LABELING FUNCTIONS ARE DIFFERENTIABLE Fig . 1 ( ii ) is an example of an expert-defined LF to characterize normal clinical findings in electroencephalogram ( EEG ) recordings2 which depends on the high baseline parameter . Our methods assume that all the LFs in the label model are differentiable with respect to their decision parameters . Many frequently used classes of LFs are intrinsically differentiable with respect to their parameters , such as multi-layer perceptrons , convolutional neural networks , log-linear models , neural language models ( Hooper et al. , 2020 ; Varma & Ré , 2018 ) . Other common classes of LFs can be easily transformed into their soft counterparts . One example is decision stumps ( IF < CONDITION > THEN CLASS < I > ELSE CLASS < J > ) ( Varma & Ré , 2018 ) . More generally , a AND or OR could also be made differentiable by fuzzy logic . To transform crisp decision stumps into their soft equivalents , we can follow prior work on decision trees ( Suárez & Lutsko , 1999 ) . Specifically , consider a heuristic of the form : λ , dᵀx > a , where dᵀx defines a new composite variable and a ∈ τ is a decision parameter . Then we define its fuzzy equivalent as γ , sigmoid ( dᵀx − a ) , where sigmoid ( x ) = 1 1+exp ( −x ) . Intuitively , while λ indicates which side of the hyperplane d ᵀx = a a point lies , γ encapsulates a probabilistic notion of distance from the decision boundary . Using the corresponding 2Specifically , the LF is developed to work with amplitude-integrated EEG , which is a widely used quantitative summary of raw multi-channel EEG data . Zadeh operators , conjunctions are converted via ( AND ( x , y ) → MIN ( x , y ) ) , disjunctions are converted via ( OR ( x , y ) → MAX ( x , y ) ) and negations are converted via ( NOT ( x ) → ( 1 − x ) ) ( Zadeh , 1996 ) . There are still other ways to obtain soft equivalents of crisp LFs , for instance by training feed-forward neural networks to approximate classes of functions under reasonable assumptions on their continuity and topology . This extension of crisp LFs into their soft counterparts not only enables them to express confidence on their predictions , but also better capture fuzzy decision boundaries , common in most practical settings . Additionally , restricting LFs to be unipolar can be done without any loss of generality , since complex LFs can be easily broken down into their unipolar components . Moreover , in our experience , several simple unipolar LFs tend to perform better than a few complex multipolar LFs in practice . Each soft LF γj can be converted to its hard equivalent λj : Rd −→ { 0 , pol ( λj ) } using a threshold function , where 0 means that the LF abstained from voting for class pol ( λj ) . In particular , we have that λj = { c , if γτ , j ( x , c ) ≥ 0.5 0 , otherwise

Summary: This paper proposes an algorithm for choosing a small set of labels to improve labeling function model performance both directly and for downstream tasks. Additionally, the authors provide a general method to convert standard labeling functions to "soft" labeling functions which are differentiable with respect to some parameters (e.g. a threshold). If the labeling functions are differentiable, this paper provides a method to update the labeling function parameters. Finally, experimental results show that the method introduced outperforms other active labeling approaches for weak supervision.