End-to-End Weak Supervision

1 Introduction . The success of supervised machine learning methods relies on the availability of large amounts of labeled data . The common process of manual data annotation by humans , especially when domain experts need to be involved , is expensive , both in terms of effort and cost , and as such presents a major bottleneck for deploying supervised learning methods to new domains and applications . Recently , data programming , a paradigm that makes use of multiple sources of noisy labels , has emerged as a promising alternative to manual data annotation [ 30 ] . It encompasses previous paradigms such as distant supervision from external knowledge bases [ 29 , 34 ] , crowdsourcing [ 15 , 26 , 14 , 42 ] , and general heuristic and rule-based labeling of data [ 23 , 20 ] . In the data programming framework , users encode domain knowledge into so called labeling functions ( LFs ) , which are functions ( e.g . domain heuristics or knowledge base derived rules ) that noisily label subsets of data . The main task for learning from multiple sources of weak supervision is then to recover the sources ’ accuracies in order to estimate the latent true label , without access to ground truth data . In previous work [ 30 , 32 , 19 ] , this is achieved by first learning a generative probabilistic graphical model ( PGM ) over the weak supervision sources and the latent true label to estimate probabilistic labels , which are then used in the second step to train a downstream model via a noise-aware loss function . Data programming has led to a wide variety of success stories in domains such as healthcare [ 18 , 17 ] and e-commerce [ 5 ] , but the existing PGM based frameworks still come with a number of drawbacks . The separate PGM does not take the predictions of the downstream model into account , and indeed this model is trained independently of the PGM . In addition , current approaches for estimating the unknown class label via a PGM need to rely on computationally expensive approximate sampling methods [ 30 ] , estimation of the full inverse of the LFs covariance matrix [ 32 ] , or they need to make strong independence assumptions [ 19 ] . Furthermore , existing prior work and the associated theoretical analyses make assumptions that may not hold in practice [ 30 , 32 , 19 ] , such as availability ∗salvaruehling @ gmail.com . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . of a well-specified generative model structure ( i.e . that the dependencies and correlations between the weak sources have been correctly specified by the user ) , that LF errors are randomly distributed across samples , and that the latent label is independent of the features given the weak labels ( i.e . only the joint distribution between the sources and labels needs to be modeled ) . We introduce WeaSEL , our Weakly Supervised End-to-end Learner model for training neural networks with , exclusively , multiple sources of weak supervision as noisy signals for the latent labels . WeaSEL is based on 1 ) reparameterizing previous PGM based posteriors with a neural encoder network that produces accuracy scores for each weak supervision source ; and 2 ) training the encoder and downstream model on the same target loss , using the other model ’ s predictions as constant targets , to maximize the agreement between both models . The proposed method needs no labeled training data , and neither assumes sample-independent source accuracies nor redundant features for latent label modeling . We show empirically that it is not susceptible to highly correlated LFs . In addition , the proposed approach can learn from multiple probabilistic sources of weak supervision . Our contributions include : • We introduce a flexible , end-to-end method for learning models from multiple sources of weak supervision . • We empirically demonstrate that the method is naturally robust to adversarial sources as well as highly correlated weak supervision sources . • We release an open-source , end-to-end system for arbitrary PyTorch downstream models that will allow practitioners to take advantage of our approach2 . • We show that our method outperforms , by as much as 6.1 F1 points , state-of-the-art latent label modeling approaches on 4 out of 5 relevant benchmark datasets , and achieves state-of-the-art performance on a crowdsourcing dataset against methods specifically designed for this setting . 2 Related Work . Multi-source Weak Supervision The data programming paradigm [ 30 ] allows users to programmatically label data through multiple noisy sources of labels , by treating the true label as a latent variable of a generative PGM . Several approaches for learning the parameters of the generative model have been introduced [ 32 , 19 , 11 ] to address computational complexity issues . Existing methods are susceptible to misspecification of the dependencies and correlations between the LFs , which can lead to substantial losses in performance [ 8 ] . Indeed , it is common practice to assume a conditionally independent model – without any dependencies between the sources – in popular libraries [ 5 , 31 ] and related research [ 15 , 2 , 37 , 7 ] , even though methods to learn the intra-LF structure have been proposed [ 4 , 39 , 38 ] . In natural language processing , [ 40 ] consider more relaxed and broader forms 2https : //github.com/autonlab/weasel of weak supervision and introduce a framework that uses virtual evidence as prior belief over latent labels and their interdependencies , learning an end model jointly with the label model via variational EM . As in the approach proposed in this paper , the aforementioned methods do not assume any labeled training data , i.e . the downstream model is learned based solely on outputs of multiple LFs on unlabeled data . The traditional co-training paradigm [ 6 ] on the other hand is similar in spirit but requires some labeled data to be available . Recent methods that study the co-training setup where labeled training data supplements multiple WS sources , include [ 3 , 25 ] . Note that the experiments in [ 3 , 25 ] rely on large pre-trained language models , making the applicability of the approach without such models or to non-text domains unclear . Crowdsourcing Aggregating multiple noisy labels is also a core problem studied in the crowdsourcing literature . Common approaches model worker performance and the unknown label jointly [ 15 , 14 , 42 ] using expectation maximization ( EM ) or similar approaches . Some core differences to learning from weak supervision sources are that errors by crowdworkers are usually assumed to be random , and that task assignment is not always fixed but can be optimized for . The benefits of jointly optimizing the downstream model and the aggregator of the weak sources have been recognized in multiple end-to-end methods that have been proposed for the crowdsourcing problem setting [ 33 , 22 , 41 , 27 , 35 , 9 ] . They often focus on image labeling and EM-like algorithms for modeling and aggregating the workers . Importantly , our proposed approach can be used in general applications with weak supervision from multiple sources without any restrictive assumptions specific to crowdsourcing , and we show that our approach outperforms the aforementioned methods on a crowdsourcing benchmark task . 3 End-to-End Weak Supervision . Algorithm 1 WeaSEL : The proposed Weakly Supervised End-to-end Learning algorithm for learning from multiple weak supervision sources . input : batch size n , networks e , f , inverse temperatures τ1 , τ2 , noise-aware loss function L , class balance P ( y ) . for sampled minibatch { z ( k ) = ( x ( k ) , λ ( k ) ) } nk=1 do for all k ∈ { 1 , . . . , n } do # Produce accuracy scores for all weak sources θ ( z ( k ) ) = softmax ( e ( z ( k ) ) τ1 ) # Generate probabilistic labels define s ( k ) as s ( k ) = θ ( z ( k ) ) Tλ ( k ) y ( k ) e = Pθ ( y|λ ( k ) ) = softmax ( s ( k ) τ2 ) P ( y ) # Downstream model forward pass y ( k ) f = f ( x ( k ) ) end for Lf = 1n ∑n k=1 L ( y ( k ) f , stop-grad ( y ( k ) e ) ) Le = 1n ∑n k=1 L ( y ( k ) e , stop-grad ( y ( k ) f ) ) update e to minimize Le , and f to minimize Lf end for return downstream network f ( · ) In this section we present our flexible base algorithm that we call WeaSEL , which can be extended to probabilistic sources and other network architectures ( Section 7 ) . See Algorithm 1 for its pseudocode . 3.1 Problem Setup . Let ( x , y ) ∼ D be the data generating distribution , where the unknown labels belong to one of C classes : y ∈ Y = { 1 , ... , C } . As in [ 30 ] , users provide an unlabeled training set X = { xi } Ni=1 , and m labeling functions λ = λ ( x ) ∈ { 0 , 1 , ... , C } m , where 0 means that the LF abstained from labeling for any class . We write λ = ( 1 { λ = 1 } , . . . ,1 { λ = C } ) ∈ { 0 , 1 } m×C for the one-hot representation of the LF votes provided by the m LFs for C classes . Our goal is to train a downstream model f : X → Y on a noise-aware loss L ( yf , ye ) that operates on the model ’ s predictions yf = f ( x ) and probabilistic labels ye generated by an encoder model e that has access to LF votes , λ , and features , x . Note that prior work restricts the probabilistic labels to only being estimated from the LFs . 3.2 Posterior Reparameterization . Previous PGM based approaches assume that the joint distribution p ( λ , y ) of the LFs and the latent true label can be modeled as a Markov Random Field ( MRF ) with pairwise dependencies between weak supervision sources [ 30 , 31 , 32 , 19 , 11 ] . These models are parameterized by a set of LF accuracy and intra-LF correlation parameters and in some cases by additional parameters to model LF and class label propensity . Note however , that the aforementioned models ignore features X when modeling the latent labels and therefore disregard that LFs may differ in their accuracy across samples and data slices . We relax these assumptions , and instead view the latent label as an aggregation of the LF votes that is a function of the entire set of LF votes and features , on a sample-by-sample basis . That is , we model the probability of a particular sample x having the class label c ∈ Y as Pθ ( y = c|λ ) = softmax ( s ) c P ( y = c ) , ( 1 ) s = θ ( λ , x ) Tλ ∈ RC . ( 2 ) where θ ( λ , x ) ∈ Rm weighs the LF votes on a sample-by-sample basis and the softmax for class c on s is defined as softmax ( s ) c = exp ( θ ( λ , x ) T1 { λ = c } ) ∑C j=1 exp ( θ ( λ , x ) T1 { λ = j } ) . While we do not use the class balance P ( y ) in our experiments for our own model , WeaSEL , it is frequently assumed to be known [ 32 , 19 , 11 ] , and can be estimated from a small validation set , or using LF outputs as described in [ 32 ] . Our formulation can be seen as a reparameterization of the posterior of the pairwise MRFs in [ 31 , 32 , 19 ] , where θ corresponds to the LF accuracies that are fixed across the dataset and are solely learned via LF agreement and disagreement signals , ignoring the informative features . We further motivate this formulation and expand upon this connection in the appendix A .

Summary: The authors propose an "end to end" approach to weakly supervised learning. Instead of obtaining ground truth labels by modeling WS sources via a probabilistic latent variable model which don't consider the downstream performance, the authors consider a neural model that directly learning the downstream model by maximizing WS sources' agreement with probabilistic labels generated by reparameterizing previous probabilistic posteriors. The method outperforms prior approaches on five real-world datasets.