Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling

1 INTRODUCTION . The performance of supervised machine learning ( ML ) hinges on the availability of labeled data in sufficient quantity and quality . However , labeled data for applications of ML can be scarce , and the common process of obtaining labels by having annotators inspect individual samples is often expensive and time consuming . Additionally , this cost is frequently exacerbated by factors such as privacy concerns , required expert knowledge , and shifting problem definitions . Weak supervision provides a promising alternative , reducing the need for humans to hand label large datasets to train ML models ( Riedel et al. , 2010 ; Hoffmann et al. , 2011 ; Ratner et al. , 2016 ; Dehghani et al. , 2018 ) . A recent approach called data programming ( Ratner et al. , 2016 ) combines multiple weak supervision sources by using an unsupervised label model to estimate the latent true class label , an idea that has close connections to modeling workers in crowd-sourcing ( Dawid & Skene , 1979 ; Karger et al. , 2011 ; Dalvi et al. , 2013 ; Zhang et al. , 2014 ) . The approach enables subject matter experts to specify labeling functions ( LFs ) —functions that encode domain knowledge and noisily annotate subsets of data , such as user-specified heuristics or external knowledge bases—instead of needing to inspect and label individual samples . These weak supervision approaches have been used on a wide variety of data types such as MRI sequences and unstructured text , and in various domains such as healthcare and e-commerce ( Fries et al. , 2019 ; Halpern et al. , 2014 ; Bach et al. , 2019 ; Ré et al. , 2020 ) . Not only does the use of multiple sources of weak supervision provide a scalable framework for creating large labeled datasets , but it can also be viewed as a vehicle to incorporate high level , conceptual feedback into the data labeling process . In data programming , each LF is an imperfect but reasonably accurate heuristic , such as a pre-trained classifier or keyword lookup . For example , for the popular 20 newsgroups dataset , an LF to identify the class ‘ sci.space ’ may look for the token ‘ launch ’ in documents and would be right about 70 % of the time . While data programming can be very effective when done right , experts may spend a significant amount of time designing the weak supervision sources ( Varma & Ré , 2018 ) and must often inspect samples at random to generate ideas ( Cohen-Wang et al. , 2019 ) . In our 20 newsgroups example , we may randomly see a document mentioning ‘ Salman Rushdie ’ and realize that the name of a famous atheist could be a good heuristic to identify posts in ‘ alt.atheism ’ . While such a heuristic seems obvious after the fact , we have to chance upon the right documents to generate these ideas . In practice , coming up with effective LFs becomes difficult after the first few . Substantial foresight ( Ramos et al. , 2020 ) is required to create a new function that applies to a non-negligible subset of given data , is novel , and adds predictive value . We propose a new approach for training supervised ML models with weak supervision through an interactive process , supporting domain experts in fast discovery of good LFs . The method queries users in an active fashion for feedback about candidate LFs , from which a model learns to identify LFs likely to have good accuracy . Upon completion , our approach produces a final set of LFs . We use this set to create an estimate of the latent class label via an unsupervised label model and train a final , weakly supervised end classifier using a noise aware loss function on the estimated labels as in Ratner et al . ( 2016 ) . The approach relies on the observation that many applications allow for heuristics of varying quality to be generated at scale ( similar to Varma & Ré ( 2018 ) ) , and that experts can provide good judgment by identifying some LFs that have reasonable accuracy . The full pipeline of the proposed approach , termed Interactive Weak Supervision ( IWS ) 1 , is illustrated in Fig . 1 . Our contributions are : 1 . We propose , to the best of our knowledge , the first interactive method for weak supervision in which queries to be annotated are not data points but labeling functions . This approach automates the discovery of useful data labeling heuristics . 2 . We conduct experiments with real users on three classification tasks , using both text and image datasets . Our results support our modeling assumptions , demonstrate competitive test set performance of the downstream end classifier , and show that users can provide accurate feedback on automatically generated LFs . 3 . In our results , IWS shows superior performance compared to standard active learning , i.e . we achieve better test set performance with a smaller number of queries to users . In text experiments with real users , IWS achieves a mean test set AUC after 200 LF annotations that requires at least three times as many active learning iterations annotating data points . In addition , the average user response time for LF queries was shorter than for the active learning queries on data points . 2 RELATED WORK . Active strategies for weak supervision sources have largely focused on combinations of data programming with traditional active learning on data points , while our work has similarities to active learning on features ( Druck et al. , 2009 ) and active learning of virtual evidence ( Lang & Poon , 2021 ) . In Nashaat et al . ( 2018 ) , a pool of samples is created on which LFs disagree , and active learning strategies are then applied to obtain labels for some of the samples . In Cohen-Wang et al . ( 2019 ) , samples where LFs abstain or disagree most are selected and presented to users in order to inspire the creation of new LFs . In Hancock et al . ( 2018 ) , natural language explanations provided during text labeling are used to generate heuristics . The proposed system uses a semantic parser to convert explanations into logical forms , which represent labeling functions . 1Code is available at https : //github.com/benbo/interactive-weak-supervision Prior work has emphasized that LFs defined by experts frequently have a recurring structure in which elements are swapped to change the higher level concept a function corresponds to ( Varma & Ré , 2018 ; Varma et al. , 2017 ; Bach et al. , 2019 ) . As an example , in tasks involving text documents , LFs often follow a repetitive structure in which key terms or phrases and syntactical relationships change , e.g . mentions of specific words ( Varma & Ré , 2018 ; Cohen-Wang et al. , 2019 ; Varma et al. , 2019 ) . Prior work relies on this observation to create heuristic generators ( Varma & Ré , 2018 ) , LF templates ( Bach et al. , 2019 ) , and domain-specific primitives ( Varma et al. , 2017 ) . In particular , in a semi-supervised data programming setting , Varma & Ré ( 2018 ) propose a system for automatic generation of labeling functions without user interaction , by using a small set of labeled data . Additional related work has investigated weak supervision for neural networks in information retrieval ( Dehghani et al. , 2017 ; Zamani et al. , 2018 ; Zamani & Croft , 2018 ) , the modeling of dependencies among heuristics in data programming ( Bach et al. , 2017 ; Varma et al. , 2019 ) , the multi-task data programming setting ( Ratner et al. , 2019 ) , handling of multi-resolution sources ( Sala et al. , 2019 ) , the use of noisy pairwise labeling functions ( Boecking & Dubrawski , 2019 ) , addressing latent subsets in the data ( Varma et al. , 2016 ) , LFs with noisy continuous scores ( Chatterjee et al. , 2020 ) , and fast model iteration via the use of pre-trained embeddings ( Chen et al. , 2020 ) . 3 METHODS . We propose an interactive weak supervision ( IWS ) approach to assist experts in finding good labeling functions ( LFs ) for training a classifier on datasets without ground truth labels . We will first describe the general problem setting of learning to classify without ground truth samples by modeling multiple weak supervision sources , as well as the concept of LF families . We then dive into the details of the proposed IWS approach . For brevity , we limit the scope of the end classifier to binary classification , but the presented background and ideas do extend to the multi-class settings . 3.1 PRELIMINARIES . Learning with Multiple Weak Supervision Sources Assume each data point x ∈ X has a latent class label y∗ ∈ Y = { −1 , 1 } . Given n unlabeled , i.i.d . datapoints X = { xi } ni=1 , our goal is to train an end classifier f : X → Y such that f ( x ) = y∗ . In data programming ( Ratner et al. , 2016 ; 2020 ) , a user provides m LFs { λj } mj=1 , where λj : X → Y ∪ { 0 } . An LF λj noisily labels the data with λj ( x ) ∈ Y or abstains with λj ( x ) = 0 . The corresponding LF output matrix is Λ ∈ { −1 , 0 , 1 } n×m , where Λi , j = λj ( xi ) . In this paper , we assume that each LF λj has the same accuracy on each class , αj = P ( λj ( x ) = y ∗|λj ( x ) 6= 0 ) , where accuracy is defined on items where j does not abstain . Further , we denote by lj = P ( λj ( x ) 6= 0 ) the LF propensity ( sometimes called LF coverage ) , i.e . the frequency at which LF j does not abstain . In data programming , an unsupervised label model pθ ( Y , Λ ) produces probabilistic estimates of the latent class labels Y ∗ = { y∗i } ni=1 using the observed LF outputs Λ by modeling the LF accuracies , propensities , and possibly their dependencies . A number of label model approaches exist in the crowdsourcing ( Dawid & Skene , 1979 ; Zhang et al. , 2014 ) and the weak supervision literature ( Ratner et al. , 2020 ) . In this paper , we use a factor graph as proposed in Ratner et al . ( 2016 ; 2020 ) to obtain probabilistic labels by modeling the LF accuracies via factor φAcci , j ( Λ , Y ) , 1 { Λij = yi } and labeling propensity by factor φLabi , j ( Λ , Y ) , 1 { Λij 6= 0 } , and for simplicity assume LFs are independent conditional on Y . The label model is defined as pθ ( Y , Λ ) , Z −1 θ exp ( n∑ i=1 θ > φi ( Λi , yi ) ) , ( 1 ) where Zθ is a normalizing constant and φi ( Λi , yi ) defined to be the concatenation of the factors for all LFs j = 1 , . . . , m for sample i . We learn θ by minimizing the negative log marginal likelihood given the observed Λ . Finally , following Ratner et al . ( 2016 ) an end classifier f is trained using probabilistic labels pθ ( Y |Λ ) . Labeling Function Families We define LF families as sets of expert-interpretable LFs described by functions zφ : X 7→ { −1 , 0 , 1 } , for parameters φ ∈ Φ . An example are shallow decision trees zφ parameterized by variables and splitting rules φ ( Varma & Ré , 2018 ) , or a function zφ defining a regular expression for two words where φ parameterizes the word choices from a vocabulary and the target label . Given such an LF family , we can generate a large set of p candidate heuristics L = { λj ( x ) = zφj ( x ) } p j=1 , where φj ∈ Φ , e.g . by sampling from Φ and pruning low coverage candidates . These families often arise naturally in the form of LFs with repetitive structure that experts write from scratch , where template variables—such as keywords—can be sampled from the unlabeled data to create candidates . For text , we can find n-grams within a document frequency range to generate key term lookups , fill placeholders in regular expressions , or generate shallow decision trees ( Ratner et al. , 2016 ; Varma & Ré , 2018 ; Varma et al. , 2019 ) . For time series , we can create a large set of LFs based on motifs ( Lonardi & Patel , 2002 ) or graphs of temporal constraints ( GuillameBert & Dubrawski , 2017 ) . For images , we can create a library of pre-trained object detectors as in Chen et al . ( 2019 ) , or in some applications combine primitives of geometric properties of the images ( Varma & Ré , 2018 ) . An LF family has to be chosen with domain expert input . Compared to standard data programming , the burden of creating LFs from scratch is shifted to choosing an appropriate LF family and then judging recommended candidates . We argue that domain experts often have the foresight to choose an LF family such that a sufficiently sized subset of LFs is predictive of the latent class label . Such LF families may not exist for all data types and classification tasks . But when they exist they offer the opportunity to quickly build large , labeled datasets . Once created , it is reasonable to expect that the same LF generation procedure can be reused for similar classification tasks without additional effort ( e.g . we use a single LF family procedure for all text datasets in our experiments ) .

Summary: This paper proposes a new framework for interactively selecting labeling heuristics in a weakly supervised setting. The main idea of the proposed approach is to combine weak supervision and active learning. Compared to the previous work which relies on human manually create labeling functions (the abstraction of the weak supervision), this work defines a family of labeling function and uses an active learning method to interactively identify a set of labeling functions that maximizes the utility based on the usefulness by the users. The experiment results showed that the proposed approach outperforms other baseline methods.