Universalizing Weak Supervision

1 INTRODUCTION . Weak supervision ( WS ) frameworks help overcome the labeling bottleneck : the challenge of building a large dataset for use in training data-hungry deep models . WS approaches replace hand-labeling with synthesizing multiple noisy but cheap sources , called labeling functions , applied to unlabeled data . As these sources may vary in quality and be dependent , a crucial step is to model their accuracies and correlations . Informed by this model , high-quality pseudolabels are produced and used to train a downstream model . This simple yet flexible approach is highly successful in research and industry settings ( Bach et al. , 2019 ; Ré et al. , 2020 ) . WS frameworks offer three advantages : they are ( i ) flexible and subsume many existing ways to integrate side information , ( ii ) computationally efficient , and ( iii ) they offer theoretical guarantees , including estimator consistency . Unfortunately , these benefits come at the cost of being particular to very specific problem settings : categorical , usually binary , labels . Extensions , e.g. , to time-series data ( Safranchik et al. , 2020 ) , or to segmentation masks ( Hooper et al. , 2021 ) , require a new model for source synthesis , a new algorithm , and more . We seek to side-step this expensive one-at-a time process via a universal approach , enabling WS to work in any problem setting while still providing the three advantageous properties above . The main technical challenge for universal WS is the diversity of label settings : classification , structured prediction , regression , rankings , and more . Each of these settings seemingly demands a different approach for learning the source synthesis model , which we refer to as the label model . For example , Ratner et al . ( 2019 ) assumed that the distribution of the sources and latent true label is an Ising model and relied on a property of such distributions : that the inverse covariance matrix is graph-structured . It is not clear how to lift such a property to spaces of permutations or to Riemannian manifolds . We propose a general recipe to handle any type of label . The data generation process for the weak sources is modeled with an exponential family distribution that can represent a label from any metric space ( Y , dY ) . We embed labels from Y into two tractable choices of space : the Boolean hypercube { ±1 } d and Euclidean space Rd . The label model ( used for source synthesis ) is learned with an efficient method-of-moments approach in the embedding space . It only requires solving a small number of scalar linear or quadratic equations . Better yet , for isometric embeddings , we show this estimator is consistent via finite sample bounds . Experimentally , we demonstrate our approach on five choices of problems never before tackled in WS : • Learning rankings : on two real-world rankings tasks , our approach with as few as five sources performs better than supervised learning with a smaller number of true labels . In contrast , an adaptation of the Snorkel ( Ratner et al. , 2018 ) framework can not reach this performance with as many as 18 sources . • Regression : on two real-world regression datasets , when using 6 or more labeling function , the performance of our approach is comparable to fully-supervised models . • Learning in hyperbolic spaces : on a geodesic regression task in hyperbolic space , we consistently outperform fully-supervised learning , even when using only 3 labeling functions ( LFs ) . • Estimation in generic metric spaces : in a synthetic setting of metric spaces induced by random graphs , we demonstrate that our method handles LF heterogeneity better than the majority vote baseline . • Learning parse trees : in semantic dependency parsing , we outperform strong baseline models . In each experiment , we confirm the following desirable behaviors of weak supervision : ( i ) more high-quality and independent sources yield better pseudolabels , ( ii ) more pseudolabels can yield better performance compared to training on fewer clean labels , ( iii ) when source accuracy varies , our approach outperforms generalizations of majority vote . 2 PROBLEM FORMULATION AND LABEL MODEL . We give background on weak supervision frameworks , provide the problem formulation , describe our choice of universal label model , special cases , the embedding technique we use to reduce our problem to two easilyhandled cases , and discuss end model training . Background WS frameworks are a principled way to integrate weak or noisy information to produce label estimates . These weak sources include small pieces of code expressing heuristic principles , crowdworkers , lookups in external knowledge bases , pretrained models , and many more ( Karger et al. , 2011 ; Mintz et al. , 2009 ; Gupta & Manning , 2014 ; Dehghani et al. , 2017 ; Ratner et al. , 2018 ) . Given an unlabeled dataset , users construct a set of labeling functions ( LFs ) based on weak sources and apply them to the data . The estimates produced by each LF are synthesized to produce pseudolabels that can be used to train a downstream model . Problem Formulation Let x1 , x2 , . . . , xn be a dataset of unlabeled datapoints from X . Associated with these are labels from an arbitrary metric space Y with metric dY . In conventional supervised learning , we would have pairs ( x1 , y1 ) , . . . , ( xn , yn ) ; however , in WS , we do not have access to the labels . Instead , we have estimates of y from m labeling functions ( LFs ) . Each such LF s : X → Y produces an estimate of the true label y from a datapoint x . We write λa ( i ) ∈ Y for the output of the ath labeling function sa applied to the ith sample . Our goal is to obtain an estimate of the true label yi using the LF outputs λ1 ( i ) , . . . , λm ( i ) . This estimate ŷ , is used to train a downstream model . To produce it , we learn the label model P ( λ1 , λ2 , . . . , λm|y ) . The main challenge is that we never observe samples of y ; it is latent . We can summarize the weak supervision procedure in two steps : • Learning the label model : use the samples λa ( i ) to learn the label model P ( λ1 , λ2 , . . . , λm|y ) , • Perform inference : compute ŷi , or P ( yi|λ1 ( i ) , . . . , λm ( i ) ) , or a related quantity . Modeling the Sources Previous approaches , e.g. , Ratner et al . ( 2019 ) , select a particular parametric choice to model p ( λ1 , . . . , λm|y ) that balances two goals : ( i ) model richness that captures differing LF accuracies and correlations , and ( ii ) properties that permit efficient learning . Our setting demands greater generality . However , we still wish to exploit the properties of exponential family models . The natural choice is p ( λ1 , . . . , λm|y ) = 1 Z exp ( m∑ a=1 −θadY ( λa , y ) c︸ ︷︷ ︸ Accuracy Potentials + ∑ ( a , b ) ∈E −θa , bdY ( λa , λb ) c︸ ︷︷ ︸ Correlation Potentials ) . ( 1 ) Here , the set E is a set of correlations corresponding to the graphical representation of the model ( Figure 1 , right ) . Observe how source quality is modeled in ( 1 ) . If the value of θa is very large , any disagreement between the estimate λa and y is penalized through the distance dY ( λa , y ) c and so has low probability . If θa is very small , such disagreements will be common ; the source is inaccurate . We also consider a more general version of ( 1 ) . We replace−θadY ( λa , y ) with a per-source distance dθa . For example , for Y = { ±1 } d , dθa ( λa , y ) = −θTa ( λa− y ) abs , with θa ∈ Rd , generalizes the Hamming distance . Similarly , for Y = Rd , we can generalize −θa‖λa − y‖2 with dθa ( λa , y ) = − ( λa − y ) T θa ( λa − y ) , with θa ∈ Rd×d p.d. , so that LF errors are not necessarily isotropic . In the Appendix B , we detail variations and special cases of such models along with relationships to existing weak supervision work . Below , we give a selection of examples , noting that the last three of which can not be tackled with existing methods . • Binary classification : Y = { ±1 } , dY is the Hamming distance , c = 1 : this yields a shifted Ising model for standard binary classification , as in Fu et al . ( 2020 ) . • Sequence learning : Y = { ±1 } d , dY is the Hamming distance , c = 1 : this yields an Ising model for sequences , as in Sala et al . ( 2019 ) and Safranchik et al . ( 2020 ) . • Ranking : Y = Sρ , the permutation group on { 1 , . . . , ρ } , dY is the Kendall tau distance , c = 1 . This is a heterogenous Mallows model , where rankings are produced from varying-quality sources . If m = 1 , we obtain a variant of the conventional Mallows model ( Mallows , 1957 ) . • Regression : Y = R , dY is the ` 2 distance , c = 2 produces sources in Rd with Gaussian errors . • Learning on Riemannian manifolds : Y = M , a Riemannian manifold ( e.g. , hyperbolic space ) , dY is the Riemannian distance dM , c = 1 . Majority Vote and its Relatives A simplifying approach often used as a baseline in weak supervision is the majority vote . Assume that the sources are conditionally independent ( i.e . E is empty in ( 1 ) ) and all accuracies are identical . In this case , there is no need to learn the model ( 1 ) ; instead , the most “ popular ” label is used . For binary labels , this is the majority label . In the universal setting , a natural generalization is ŷMV = arg min z∈Y 1 m ∑m a=1 dY ( λ a , z ) c. ( 2 ) Special cases of ( 2 ) have their own name ; for permutations , it is the Kemeny rule ( Kemeny , 1959 ) used to solve the rank aggregation problem . For Riemannian manifolds , ŷMV is called the Fréchet or Karcher mean . Algorithm 1 : Universal Label Model Learning Input : Output of labeling functions λa ( i ) , correlation set E , prior p for Y , optionally embedding function g. Embedding : If g is not given , use Multidimensional Scaling ( MDS ) to obtain embeddings g ( λa ) ∈ Rd ∀a for a ∈ { 1 , 2 , . . . , m } do For b : ( a , b ) ∈ E Estimate Correlations : ∀i , j , Ê [ g ( λa ) ig ( λ b ) j ] = 1 n ∑n t=1 g ( λ a ( t ) ) ig ( λ b ( t ) ) j Estimate Accuracy : Pick b , c : ( a , b ) 6∈ E , ( a , c ) 6∈ E , ( b , c ) 6∈ E. if Im ( g ) = { ±1 } d ∀i , Estimate Ôa , b = P̂ ( g ( λa ) i = 1 , g ( λb ) i = 1 ) , Ôa , c , Ôb , c , Estimate ` a = P̂ ( g ( λa ) i = 1 ) , ` b , ` c Accuracies← QUADRATICTRIPLETS ( Oa , b , Oa , c , Ob , c , ` a , ` b , ` c , p , i ) else ∀i , Estimate ea , b : = Ê [ g ( λa ) ig ( λ b ) i ] = 1 n ∑n t=1 g ( λ a ( t ) ) ig ( λ b ( t ) ) i , ea , c , eb , c Accuracies← CONTINUOUSTRIPLETS ( ea , b , ea , c , eb , c , p , i ) Recover accuracy signs ( Fu et al. , 2020 ) end for return θ̂a , θ̂a , b by running the backward mapping on accuracies and correlations Majority vote , however , is insufficient in cases where there is variation in the source qualities . We must learn the label model . However , generically learning ( 1 ) is an ambitious goal . Even cases that specialize ( 1 ) in multiple ways have only recently been fully solved , e.g. , the permutation case for identical θa ’ s ) was fully characterized by Mukherjee ( 2016 ) . To overcome the challenge of generality , we opt for an embedding approach that reduces our problem to two tractable cases . Universality via Embeddings To deal with the very high level of generality , we reduce the problem to just two metric spaces : the boolean hypercube { −1 , +1 } d and Euclidean space Rd . To accomplish this , let g : Y → { ±1 } d ( or g : Y → Rd ) be an injective embedding function . The advantage of this approach is that if g is isometric—that is , distance preserving—then probabilities are preserved under g. This is because the sufficient statistics are preserved : for example , for g : Y → Rd , −θadY ( λa , y ) c = −θad ( g ( λa ) , g ( y ) ) c = −θa‖g ( λa ) − g ( y ) ‖c , yielding is a multivariate normal for c = 2 . Even if g is not isometric , there is a rich literature on low-distortion embeddings , with Bourgain ’ s theorem as a cornerstone result ( Bourgain , 1985 ) . This makes it possible to bound the error in recovering the parameters for any label type . End Model Once we have produced pseudolabels—either by applying generalized majority vote or by learning the label model and performing inference—we can use the labels to train an end model . Table 2 summarizes examples of problem types , explaining the underlying label set , the metric , the exponent c , the generalization of majority vote , the embedding space Im ( g ) , and an example of an end model .

Summary: This work studies a weakly supervised learning setup of aggregating multiple weak sources of labels into high-quality pseudo labels for learning.   The proposed approach is based on building graphical models with the true labels as latent variables following existing techniques, but generalizes existing works in 1) supporting more types of task labels such as ranking, regression and in non-Euclidean spaces through approximating the graphical model terms using dot product between embeddings, and 2) algorithms for efficient learning under the embedding approximation and theoretical bounds on the estimation error using embedding-based approximation.   Experiments focus on comparing weakly supervised learning performance of learning a graphical model versus using majority vote on the newly supported tasks and fully supervised learning with varying amount of labels. Results show that learning the graphical model achieved better performance than majority vote and can achieve parity reach a certain level of fully supervision.