End-to-End Weak Supervision

1 Introduction . The success of supervised machine learning methods relies on the availability of large amounts of labeled data . The common process of manual data annotation by humans , especially when domain experts need to be involved , is expensive , both in terms of effort and cost , and as such presents a major bottleneck for deploying supervised learning methods to new domains and applications . Recently , data programming , a paradigm that makes use of multiple sources of noisy labels , has emerged as a promising alternative to manual data annotation [ 30 ] . It encompasses previous paradigms such as distant supervision from external knowledge bases [ 29 , 34 ] , crowdsourcing [ 15 , 26 , 14 , 42 ] , and general heuristic and rule-based labeling of data [ 23 , 20 ] . In the data programming framework , users encode domain knowledge into so called labeling functions ( LFs ) , which are functions ( e.g . domain heuristics or knowledge base derived rules ) that noisily label subsets of data . The main task for learning from multiple sources of weak supervision is then to recover the sources ’ accuracies in order to estimate the latent true label , without access to ground truth data . In previous work [ 30 , 32 , 19 ] , this is achieved by first learning a generative probabilistic graphical model ( PGM ) over the weak supervision sources and the latent true label to estimate probabilistic labels , which are then used in the second step to train a downstream model via a noise-aware loss function . Data programming has led to a wide variety of success stories in domains such as healthcare [ 18 , 17 ] and e-commerce [ 5 ] , but the existing PGM based frameworks still come with a number of drawbacks . The separate PGM does not take the predictions of the downstream model into account , and indeed this model is trained independently of the PGM . In addition , current approaches for estimating the unknown class label via a PGM need to rely on computationally expensive approximate sampling methods [ 30 ] , estimation of the full inverse of the LFs covariance matrix [ 32 ] , or they need to make strong independence assumptions [ 19 ] . Furthermore , existing prior work and the associated theoretical analyses make assumptions that may not hold in practice [ 30 , 32 , 19 ] , such as availability ∗salvaruehling @ gmail.com . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of a well-specified generative model structure ( i.e . that the dependencies and correlations between the weak sources have been correctly specified by the user ) , that LF errors are randomly distributed across samples , and that the latent label is independent of the features given the weak labels ( i.e . only the joint distribution between the sources and labels needs to be modeled ) . We introduce WeaSEL , our Weakly Supervised End-to-end Learner model for training neural networks with , exclusively , multiple sources of weak supervision as noisy signals for the latent labels . WeaSEL is based on 1 ) reparameterizing previous PGM based posteriors with a neural encoder network that produces accuracy scores for each weak supervision source ; and 2 ) training the encoder and downstream model on the same target loss , using the other model ’ s predictions as constant targets , to maximize the agreement between both models . The proposed method needs no labeled training data , and neither assumes sample-independent source accuracies nor redundant features for latent label modeling . We show empirically that it is not susceptible to highly correlated LFs . In addition , the proposed approach can learn from multiple probabilistic sources of weak supervision . Our contributions include : • We introduce a flexible , end-to-end method for learning models from multiple sources of weak supervision . • We empirically demonstrate that the method is naturally robust to adversarial sources as well as highly correlated weak supervision sources . • We release an open-source , end-to-end system for arbitrary PyTorch downstream models that will allow practitioners to take advantage of our approach2 . • We show that our method outperforms , by as much as 6.1 F1 points , state-of-the-art latent label modeling approaches on 4 out of 5 relevant benchmark datasets , and achieves state-of-the-art performance on a crowdsourcing dataset against methods specifically designed for this setting . 2 Related Work . Multi-source Weak Supervision The data programming paradigm [ 30 ] allows users to programmatically label data through multiple noisy sources of labels , by treating the true label as a latent variable of a generative PGM . Several approaches for learning the parameters of the generative model have been introduced [ 32 , 19 , 11 ] to address computational complexity issues . Existing methods are susceptible to misspecification of the dependencies and correlations between the LFs , which can lead to substantial losses in performance [ 8 ] . Indeed , it is common practice to assume a conditionally independent model – without any dependencies between the sources – in popular libraries [ 5 , 31 ] and related research [ 15 , 2 , 37 , 7 ] , even though methods to learn the intra-LF structure have been proposed [ 4 , 39 , 38 ] . In natural language processing , [ 40 ] consider more relaxed and broader forms 2https : //github.com/autonlab/weasel of weak supervision and introduce a framework that uses virtual evidence as prior belief over latent labels and their interdependencies , learning an end model jointly with the label model via variational EM . As in the approach proposed in this paper , the aforementioned methods do not assume any labeled training data , i.e . the downstream model is learned based solely on outputs of multiple LFs on unlabeled data . The traditional co-training paradigm [ 6 ] on the other hand is similar in spirit but requires some labeled data to be available . Recent methods that study the co-training setup where labeled training data supplements multiple WS sources , include [ 3 , 25 ] . Note that the experiments in [ 3 , 25 ] rely on large pre-trained language models , making the applicability of the approach without such models or to non-text domains unclear . Crowdsourcing Aggregating multiple noisy labels is also a core problem studied in the crowdsourcing literature . Common approaches model worker performance and the unknown label jointly [ 15 , 14 , 42 ] using expectation maximization ( EM ) or similar approaches . Some core differences to learning from weak supervision sources are that errors by crowdworkers are usually assumed to be random , and that task assignment is not always fixed but can be optimized for . The benefits of jointly optimizing the downstream model and the aggregator of the weak sources have been recognized in multiple end-to-end methods that have been proposed for the crowdsourcing problem setting [ 33 , 22 , 41 , 27 , 35 , 9 ] . They often focus on image labeling and EM-like algorithms for modeling and aggregating the workers . Importantly , our proposed approach can be used in general applications with weak supervision from multiple sources without any restrictive assumptions specific to crowdsourcing , and we show that our approach outperforms the aforementioned methods on a crowdsourcing benchmark task . 3 End-to-End Weak Supervision . Algorithm 1 WeaSEL : The proposed Weakly Supervised End-to-end Learning algorithm for learning from multiple weak supervision sources . input : batch size n , networks e , f , inverse temperatures τ1 , τ2 , noise-aware loss function L , class balance P ( y ) . for sampled minibatch { z ( k ) = ( x ( k ) , λ ( k ) ) } nk=1 do for all k ∈ { 1 , . . . , n } do # Produce accuracy scores for all weak sources θ ( z ( k ) ) = softmax ( e ( z ( k ) ) τ1 ) # Generate probabilistic labels define s ( k ) as s ( k ) = θ ( z ( k ) ) Tλ ( k ) y ( k ) e = Pθ ( y|λ ( k ) ) = softmax ( s ( k ) τ2 ) P ( y ) # Downstream model forward pass y ( k ) f = f ( x ( k ) ) end for Lf = 1n ∑n k=1 L ( y ( k ) f , stop-grad ( y ( k ) e ) ) Le = 1n ∑n k=1 L ( y ( k ) e , stop-grad ( y ( k ) f ) ) update e to minimize Le , and f to minimize Lf end for return downstream network f ( · ) In this section we present our flexible base algorithm that we call WeaSEL , which can be extended to probabilistic sources and other network architectures ( Section 7 ) . See Algorithm 1 for its pseudocode . 3.1 Problem Setup . Let ( x , y ) ∼ D be the data generating distribution , where the unknown labels belong to one of C classes : y ∈ Y = { 1 , ... , C } . As in [ 30 ] , users provide an unlabeled training set X = { xi } Ni=1 , and m labeling functions λ = λ ( x ) ∈ { 0 , 1 , ... , C } m , where 0 means that the LF abstained from labeling for any class . We write λ = ( 1 { λ = 1 } , . . . ,1 { λ = C } ) ∈ { 0 , 1 } m×C for the one-hot representation of the LF votes provided by the m LFs for C classes . Our goal is to train a downstream model f : X → Y on a noise-aware loss L ( yf , ye ) that operates on the model ’ s predictions yf = f ( x ) and probabilistic labels ye generated by an encoder model e that has access to LF votes , λ , and features , x . Note that prior work restricts the probabilistic labels to only being estimated from the LFs . 3.2 Posterior Reparameterization . Previous PGM based approaches assume that the joint distribution p ( λ , y ) of the LFs and the latent true label can be modeled as a Markov Random Field ( MRF ) with pairwise dependencies between weak supervision sources [ 30 , 31 , 32 , 19 , 11 ] . These models are parameterized by a set of LF accuracy and intra-LF correlation parameters and in some cases by additional parameters to model LF and class label propensity . Note however , that the aforementioned models ignore features X when modeling the latent labels and therefore disregard that LFs may differ in their accuracy across samples and data slices . We relax these assumptions , and instead view the latent label as an aggregation of the LF votes that is a function of the entire set of LF votes and features , on a sample-by-sample basis . That is , we model the probability of a particular sample x having the class label c ∈ Y as Pθ ( y = c|λ ) = softmax ( s ) c P ( y = c ) , ( 1 ) s = θ ( λ , x ) Tλ ∈ RC . ( 2 ) where θ ( λ , x ) ∈ Rm weighs the LF votes on a sample-by-sample basis and the softmax for class c on s is defined as softmax ( s ) c = exp ( θ ( λ , x ) T1 { λ = c } ) ∑C j=1 exp ( θ ( λ , x ) T1 { λ = j } ) . While we do not use the class balance P ( y ) in our experiments for our own model , WeaSEL , it is frequently assumed to be known [ 32 , 19 , 11 ] , and can be estimated from a small validation set , or using LF outputs as described in [ 32 ] . Our formulation can be seen as a reparameterization of the posterior of the pairwise MRFs in [ 31 , 32 , 19 ] , where θ corresponds to the LF accuracies that are fixed across the dataset and are solely learned via LF agreement and disagreement signals , ignoring the informative features . We further motivate this formulation and expand upon this connection in the appendix A .

Summary: This paper proposes a new method for weak supervision with multiple noisy labeling functions. The method jointly trains an encoder network that can estimate sample-by-sample accuracies with the final downstream model, and maximizes the agreement between the two models. The authors show improved performance on several weak supervision data sets and further show that their method is robust against both adversarial labeling functions and highly-correlated labeling functions.