Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling

1 INTRODUCTION . The performance of supervised machine learning ( ML ) hinges on the availability of labeled data in sufficient quantity and quality . However , labeled data for applications of ML can be scarce , and the common process of obtaining labels by having annotators inspect individual samples is often expensive and time consuming . Additionally , this cost is frequently exacerbated by factors such as privacy concerns , required expert knowledge , and shifting problem definitions . Weak supervision provides a promising alternative , reducing the need for humans to hand label large datasets to train ML models ( Riedel et al. , 2010 ; Hoffmann et al. , 2011 ; Ratner et al. , 2016 ; Dehghani et al. , 2018 ) . A recent approach called data programming ( Ratner et al. , 2016 ) combines multiple weak supervision sources by using an unsupervised label model to estimate the latent true class label , an idea that has close connections to modeling workers in crowd-sourcing ( Dawid & Skene , 1979 ; Karger et al. , 2011 ; Dalvi et al. , 2013 ; Zhang et al. , 2014 ) . The approach enables subject matter experts to specify labeling functions ( LFs ) —functions that encode domain knowledge and noisily annotate subsets of data , such as user-specified heuristics or external knowledge bases—instead of needing to inspect and label individual samples . These weak supervision approaches have been used on a wide variety of data types such as MRI sequences and unstructured text , and in various domains such as healthcare and e-commerce ( Fries et al. , 2019 ; Halpern et al. , 2014 ; Bach et al. , 2019 ; Ré et al. , 2020 ) . Not only does the use of multiple sources of weak supervision provide a scalable framework for creating large labeled datasets , but it can also be viewed as a vehicle to incorporate high level , conceptual feedback into the data labeling process . In data programming , each LF is an imperfect but reasonably accurate heuristic , such as a pre-trained classifier or keyword lookup . For example , for the popular 20 newsgroups dataset , an LF to identify the class ‘ sci.space ’ may look for the token ‘ launch ’ in documents and would be right about 70 % of the time . While data programming can be very effective when done right , experts may spend a significant amount of time designing the weak supervision sources ( Varma & Ré , 2018 ) and must often inspect samples at random to generate ideas ( Cohen-Wang et al. , 2019 ) . In our 20 newsgroups example , we may randomly see a document mentioning ‘ Salman Rushdie ’ and realize that the name of a famous atheist could be a good heuristic to identify posts in ‘ alt.atheism ’ . While such a heuristic seems obvious after the fact , we have to chance upon the right documents to generate these ideas . In practice , coming up with effective LFs becomes difficult after the first few . Substantial foresight ( Ramos et al. , 2020 ) is required to create a new function that applies to a non-negligible subset of given data , is novel , and adds predictive value . We propose a new approach for training supervised ML models with weak supervision through an interactive process , supporting domain experts in fast discovery of good LFs . The method queries users in an active fashion for feedback about candidate LFs , from which a model learns to identify LFs likely to have good accuracy . Upon completion , our approach produces a final set of LFs . We use this set to create an estimate of the latent class label via an unsupervised label model and train a final , weakly supervised end classifier using a noise aware loss function on the estimated labels as in Ratner et al . ( 2016 ) . The approach relies on the observation that many applications allow for heuristics of varying quality to be generated at scale ( similar to Varma & Ré ( 2018 ) ) , and that experts can provide good judgment by identifying some LFs that have reasonable accuracy . The full pipeline of the proposed approach , termed Interactive Weak Supervision ( IWS ) 1 , is illustrated in Fig . 1 . Our contributions are : 1 . We propose , to the best of our knowledge , the first interactive method for weak supervision in which queries to be annotated are not data points but labeling functions . This approach automates the discovery of useful data labeling heuristics . 2 . We conduct experiments with real users on three classification tasks , using both text and image datasets . Our results support our modeling assumptions , demonstrate competitive test set performance of the downstream end classifier , and show that users can provide accurate feedback on automatically generated LFs . 3 . In our results , IWS shows superior performance compared to standard active learning , i.e . we achieve better test set performance with a smaller number of queries to users . In text experiments with real users , IWS achieves a mean test set AUC after 200 LF annotations that requires at least three times as many active learning iterations annotating data points . In addition , the average user response time for LF queries was shorter than for the active learning queries on data points . 2 RELATED WORK . Active strategies for weak supervision sources have largely focused on combinations of data programming with traditional active learning on data points , while our work has similarities to active learning on features ( Druck et al. , 2009 ) and active learning of virtual evidence ( Lang & Poon , 2021 ) . In Nashaat et al . ( 2018 ) , a pool of samples is created on which LFs disagree , and active learning strategies are then applied to obtain labels for some of the samples . In Cohen-Wang et al . ( 2019 ) , samples where LFs abstain or disagree most are selected and presented to users in order to inspire the creation of new LFs . In Hancock et al . ( 2018 ) , natural language explanations provided during text labeling are used to generate heuristics . The proposed system uses a semantic parser to convert explanations into logical forms , which represent labeling functions . 1Code is available at https : //github.com/benbo/interactive-weak-supervision Prior work has emphasized that LFs defined by experts frequently have a recurring structure in which elements are swapped to change the higher level concept a function corresponds to ( Varma & Ré , 2018 ; Varma et al. , 2017 ; Bach et al. , 2019 ) . As an example , in tasks involving text documents , LFs often follow a repetitive structure in which key terms or phrases and syntactical relationships change , e.g . mentions of specific words ( Varma & Ré , 2018 ; Cohen-Wang et al. , 2019 ; Varma et al. , 2019 ) . Prior work relies on this observation to create heuristic generators ( Varma & Ré , 2018 ) , LF templates ( Bach et al. , 2019 ) , and domain-specific primitives ( Varma et al. , 2017 ) . In particular , in a semi-supervised data programming setting , Varma & Ré ( 2018 ) propose a system for automatic generation of labeling functions without user interaction , by using a small set of labeled data . Additional related work has investigated weak supervision for neural networks in information retrieval ( Dehghani et al. , 2017 ; Zamani et al. , 2018 ; Zamani & Croft , 2018 ) , the modeling of dependencies among heuristics in data programming ( Bach et al. , 2017 ; Varma et al. , 2019 ) , the multi-task data programming setting ( Ratner et al. , 2019 ) , handling of multi-resolution sources ( Sala et al. , 2019 ) , the use of noisy pairwise labeling functions ( Boecking & Dubrawski , 2019 ) , addressing latent subsets in the data ( Varma et al. , 2016 ) , LFs with noisy continuous scores ( Chatterjee et al. , 2020 ) , and fast model iteration via the use of pre-trained embeddings ( Chen et al. , 2020 ) . 3 METHODS . We propose an interactive weak supervision ( IWS ) approach to assist experts in finding good labeling functions ( LFs ) for training a classifier on datasets without ground truth labels . We will first describe the general problem setting of learning to classify without ground truth samples by modeling multiple weak supervision sources , as well as the concept of LF families . We then dive into the details of the proposed IWS approach . For brevity , we limit the scope of the end classifier to binary classification , but the presented background and ideas do extend to the multi-class settings . 3.1 PRELIMINARIES . Learning with Multiple Weak Supervision Sources Assume each data point x ∈ X has a latent class label y∗ ∈ Y = { −1 , 1 } . Given n unlabeled , i.i.d . datapoints X = { xi } ni=1 , our goal is to train an end classifier f : X → Y such that f ( x ) = y∗ . In data programming ( Ratner et al. , 2016 ; 2020 ) , a user provides m LFs { λj } mj=1 , where λj : X → Y ∪ { 0 } . An LF λj noisily labels the data with λj ( x ) ∈ Y or abstains with λj ( x ) = 0 . The corresponding LF output matrix is Λ ∈ { −1 , 0 , 1 } n×m , where Λi , j = λj ( xi ) . In this paper , we assume that each LF λj has the same accuracy on each class , αj = P ( λj ( x ) = y ∗|λj ( x ) 6= 0 ) , where accuracy is defined on items where j does not abstain . Further , we denote by lj = P ( λj ( x ) 6= 0 ) the LF propensity ( sometimes called LF coverage ) , i.e . the frequency at which LF j does not abstain . In data programming , an unsupervised label model pθ ( Y , Λ ) produces probabilistic estimates of the latent class labels Y ∗ = { y∗i } ni=1 using the observed LF outputs Λ by modeling the LF accuracies , propensities , and possibly their dependencies . A number of label model approaches exist in the crowdsourcing ( Dawid & Skene , 1979 ; Zhang et al. , 2014 ) and the weak supervision literature ( Ratner et al. , 2020 ) . In this paper , we use a factor graph as proposed in Ratner et al . ( 2016 ; 2020 ) to obtain probabilistic labels by modeling the LF accuracies via factor φAcci , j ( Λ , Y ) , 1 { Λij = yi } and labeling propensity by factor φLabi , j ( Λ , Y ) , 1 { Λij 6= 0 } , and for simplicity assume LFs are independent conditional on Y . The label model is defined as pθ ( Y , Λ ) , Z −1 θ exp ( n∑ i=1 θ > φi ( Λi , yi ) ) , ( 1 ) where Zθ is a normalizing constant and φi ( Λi , yi ) defined to be the concatenation of the factors for all LFs j = 1 , . . . , m for sample i . We learn θ by minimizing the negative log marginal likelihood given the observed Λ . Finally , following Ratner et al . ( 2016 ) an end classifier f is trained using probabilistic labels pθ ( Y |Λ ) . Labeling Function Families We define LF families as sets of expert-interpretable LFs described by functions zφ : X 7→ { −1 , 0 , 1 } , for parameters φ ∈ Φ . An example are shallow decision trees zφ parameterized by variables and splitting rules φ ( Varma & Ré , 2018 ) , or a function zφ defining a regular expression for two words where φ parameterizes the word choices from a vocabulary and the target label . Given such an LF family , we can generate a large set of p candidate heuristics L = { λj ( x ) = zφj ( x ) } p j=1 , where φj ∈ Φ , e.g . by sampling from Φ and pruning low coverage candidates . These families often arise naturally in the form of LFs with repetitive structure that experts write from scratch , where template variables—such as keywords—can be sampled from the unlabeled data to create candidates . For text , we can find n-grams within a document frequency range to generate key term lookups , fill placeholders in regular expressions , or generate shallow decision trees ( Ratner et al. , 2016 ; Varma & Ré , 2018 ; Varma et al. , 2019 ) . For time series , we can create a large set of LFs based on motifs ( Lonardi & Patel , 2002 ) or graphs of temporal constraints ( GuillameBert & Dubrawski , 2017 ) . For images , we can create a library of pre-trained object detectors as in Chen et al . ( 2019 ) , or in some applications combine primitives of geometric properties of the images ( Varma & Ré , 2018 ) . An LF family has to be chosen with domain expert input . Compared to standard data programming , the burden of creating LFs from scratch is shifted to choosing an appropriate LF family and then judging recommended candidates . We argue that domain experts often have the foresight to choose an LF family such that a sufficiently sized subset of LFs is predictive of the latent class label . Such LF families may not exist for all data types and classification tasks . But when they exist they offer the opportunity to quickly build large , labeled datasets . Once created , it is reasonable to expect that the same LF generation procedure can be reused for similar classification tasks without additional effort ( e.g . we use a single LF family procedure for all text datasets in our experiments ) .

Summary: This paper proposes a new approach for active learning by interactively discovering weak supervision. Instead of asking human to annotate data points, the method collects feedback about candidate label functions, from which a model learns to identify promising label functions. With the final set of label functions, they train a classifier with the estimated labels on unlabeled data. They conduct experiments on text classification datasets with both oracle and human feedback, and show a large improvement compared with traditional active learning.