Creating Training Sets via Weak Indirect Supervision

1 INTRODUCTION . One of the greatest bottlenecks of using modern machine learning models is the need for substantial amounts of manually-labeled training data . In real-world applications , such manual annotations are typically time-consuming , labor-intensive and static . To reduce the efforts of annotation , researchers have proposed Weak Supervision ( WS ) frameworks ( Ratner et al. , 2016 ; 2018 ; 2019 ; Fu et al. , 2020 ) for synthesizing labels from multiple weak supervision sources , e.g. , heuristics , knowledge bases , or pre-trained classifiers . These frameworks have been widely applied on various machine learning tasks ( Dunnmon et al. , 2020 ; Fries et al. , 2021 ; Safranchik et al. , 2020 ; Lison et al. , 2020 ; Zhou et al. , 2020 ; Hooper et al. , 2021 ; Zhan et al. , 2019 ; Varma et al. , 2019 ) and industrial data ( Bach et al. , 2019 ) . Among them , data programming ( Ratner et al. , 2016 ) , one representative example that generalizes many approaches in the literature , represents weak supervision sources as labeling functions ( LFs ) and synthesizes training labels using Probabilistic Graphical Model ( PGM ) . Given both the increasing popularity of WS and the general increase in open-source availability of machine learning models and tools , there is a rising tide of available supervision sources that WS frameworks and practitioners could potentially leverage , including pre-trained machine learning models or prediction APIs ( Chen et al. , 2020 ; d ’ Andrea & Mintz , 2019 ; Yao et al. , 2017 ) . However , existing WS frameworks only utilize weak supervision sources with the same label space as the target task . This incompatibility largely limits the scope of usable sources , necessitating manual effort from domain experts to provide supervision for unseen labels . For example , consider target task of classifying { “ dog ” , “ wolf ” , “ cat ” , “ lion ” } and a set of three weak supervision sources ( e.g . trained classifiers or expert heuristics ) with disjoint output spaces { “ caninae ” , “ felidae ” } , { “ domestic animals ” , “ wild animals ” } and { “ husky ” , “ bengal cat ” } respectively . We call these types of sources indirect supervision sources . For concreteness , we follow the general convention of data programming ( Ratner et al. , 2016 ) and refer to these sources as indirect labeling functions ( ILFs ) . Despite their apparent utility , existing weak supervision methods could not directly leverage such ILFs , as their output spaces have no overlap with the target one . In this paper , we formulate a novel research problem that aims to leverage such ILFs automatically , minimizing the manual efforts to develop and deploy new models . We refer to this as the Weak Indirect Supervision ( WIS ) setting , a new Weak Supervision paradigm which leverages ILFs , along with the relational structures between individual labels , to automatically create training labels . The key difficulty of leveraging ILFs is due to the mismatched label spaces . To overcome this , we introduce pairwise relations between individual labels to the WIS setup , which are often available in structured sources ( e.g . off-the-shelf Knowledge Bases ( Miller , 1995 ; Sinha et al. , 2015 ; Dong et al. , 2020 ) or large scale label hierarchies ( Murty et al. , 2017 ; The Gene Ontology Consortium , 2018 ; Partalas et al. , 2015 ) for various domains ) , or can be provided by subject matter experts in far less time than generating entirely new sets of weak supervision sources . For example , in the aforementioned example , we could rely on a biological species ontology to see that the unseen labels “ dog ” and “ cat ” are both subsumed by the seen label “ domestic animals ” . Based on the label relations , we can automatically leverage the supervision sources as ILFs . Notably , previous work ( Qu et al. , 2020 ) also leveraged a label relation graph but was focused on relation extraction task in a few-shot learning setting , while You et al . ( 2020 ) proposed to learn label relations given data for each label in a transfer learning scenario . In contrast , we aim to solve the target task directly and without clean labeled data . The remaining questions are ( 1 ) how to synthesize labels based on pair-wise label relations and ILFs ? and ( 2 ) How can we know whether , given a set of ILFs and label relations , the unseen labels are distinguishable or not ? To address the first question , we develop a probabilistic label relation model ( PLRM ) , the first PGM for WIS which aggregates the output of ILFs and models the label relations as dependencies between random variables . In turn , we use the learned PLRM to produce labels for training an end model . Furthermore , we derive the generalization error bound of PLRM based on assumptions similar to previous work ( Ratner et al. , 2016 ) . The second question presents an important stumbling block when dealing with unseen labels , as we may not be able to distinguish the unseen labels given existing label relations and ILFs , resulting in an unsatisfactory synthesized training set . To address this issue , we formally introduce the notion of distinguishability in WIS setting and theoretically establish an equivalence between : ( 1 ) the distinguishability of the label relation structure as well as the ILFs , and ( 2 ) the capability of PLRM to distinguish unseen labels . This result then leads to a simple sanity test for preventing the model from failing to distinguish unseen labels . In preliminary experiments , we observe a significant drop in model performance when the condition is violated . In experiments , we make non-trivial adaptations for baselines from related settings to the new WIS problem . On both text and image classification tasks , we demonstrate the advantages of PLRM over adapted baselines . Finally , in a commercial advertising system where developers need to collect annotations for new ads tags , we illustrate how to formulate the training label collection as a WIS problem and apply PLRM to achieve an effective performance . Summary of Contributions . Our contributions are summarized as follows : • We formulate Weak Indirect Supervision ( WIS ) , a new research problem which synthesizes training labels based on indirect supervision sources and label relations , minimizing human efforts of both data annotation and weak supervision sources construction ; • We develop the first model for WIS , the Probabilistic Label Relation Model ( PLRM ) with comparable statistical efficiency to previous WS frameworks and standard supervised learning ; • We introduce a new notion of distinguishability in WIS setting , and provide a simple test of the distinguishability of PLRM for unseen labels by theoretically establishing the connection between the label relation structures and distinguishability ; • We showcase the potential of the WIS formulation and the effectiveness of PLRM in a commercial advertising system for synthesizing training labels of new ads tags . On academic image and text classification tasks , we demonstrate the advantages of PLRM over baselines by quantitative experiments . Overall , PLRM outperforms baselines by a margin of 2 % -9 % . 2 RELATED WORK . We briefly review related settings . The comparison between WIS and related tasks is in Table 1 . Weak Supervision : We draw motivation from recent work which model and integrate weak supervision sources using PGMs ( Ratner et al. , 2016 ; 2018 ; 2019 ; Fu et al. , 2020 ) and other methods ( Guan et al. , 2018 ; Khetan et al. , 2018 ) to create training sets . While they assume supervision sources share the same label space as the new tasks , we aim to leverage indirect supervision sources with mismatched label spaces in a labor-free way . Indirect Supervision : Indirect supervision arises more generally in latent-variable models for various domains ( Brown et al. , 1993 ; Liang et al. , 2013 ; Quattoni et al. , 2004 ; Chang et al. , 2010 ; Zhang et al. , 2019 ) . Very recently , Raghunathan et al . ( 2016 ) proposed to use the linear moment method for indirect supervision , wherein the transition between desired label space Y and indirect supervision space O is known , as well as the ground truth of indirect supervisions for training . In contrast , both are unavailable in WIS. Theoretically , Wang et al . ( 2020 ) developed a unified framework for analyzing the learnability of indirect supervision with shared or superset label spaces , while we focus on disjoint label spaces and the consequent unique challenge of distinguishability of unseen classes . Zero-Shot Learning : Zero-Shot Learning ( ZSL ) ( Lampert et al. , 2009 ; Wang et al. , 2019 ) aims to learn a classifier that is able to generalize to unseen classes . The WIS problem differentiates from ZSL by ( 1 ) in ZSL setting , the training and test data belong to seen and unseen classes , respectively , and training data is labeled , while for WIS , both training and test data belong to unseen classes and unlabeled ; ( 2 ) ZSL tends to render a classifier that could predict unseen classes given certain label information , e.g. , label attributes ( Romera-Paredes & Torr , 2015 ) , label descriptions ( Srivastava et al. , 2018 ) or label similarities ( Frome et al. , 2013 ) , while WIS aims to provide training labels for unlabeled training data , allowing users to train any machine learning models , and the label relations are used only in synthesizing training labels . 3 PRELIMINARY : WEAK SUPERVISION . We first describe the Weak Supervision ( WS ) setting . A glossary of notations used is in App . A. Definitions and notations . We assume a k-way classification task , and have an unlabeled dataset D consisting of m data points . Denote by Xi ∈ X the individual data point and Yi ∈ Y = { y1 , . . . , yk } the unobserved interested label of Xi . We also have n sources , each represented by a labeling function ( LF ) and denoted by λj . Each λj outputs a label Ŷ j i ∈ Yλj = { ŷj1 , . . . , ŷjkλj } on Xi , where Yλj is the label space associated with λj and |Yλj | = kλj . We denote the concatenation of LFs ’ output as Ŷi = [ Ŷ 1i , Ŷ 2 i , . . . , Ŷ n i ] , and the union set of LFs ’ label spaces as Ŷ with |Ŷ| = k̂ . Note that k̂ is not necessarily equal to the sum over kλj , since LFs may have overlapping label spaces . We call ŷ ∈ Ŷ seen label and y ∈ Y desired labels . In WS settings , we have Y ⊂ Ŷ . Notably , we assume all the involved labels come from the same semantic space . The goal of WS . The goal is to infer the training labels for the dataset D based on LFs , and to use them to train an end discriminative classifier fW : X → Y , all without ground truth training labels .

Summary: The paper addresses a novel research problem of using "indirect" label sources in the weak supervision framework to create labelled datasets. The indirect label sources are similar to the labeling functions in prior works in weak supervision ( data programming) with one caveat that these sources produce labels from different space than that of the original ( target) label spaces. It can be useful, when there are good indirect LFs and there is some relationships between the labels in the LF's label space and target label space. This paper, gives a probabilistic label model (PRML) which utilizes these indirect LFs and label relationships to produce desired labels for the given unlabeled data. The methodology is backed by theoretical analysis and real-world experiments. In analysis, a generalization error bound (for the end model learned using estimated labels) is provided which turns out to be similar to the work in Data programming (Ratner et al. 2016). One needs to be careful with the issue of indistinguishability between labels, in such setup. This issue has been studied in detail and they provide definition, conditions for distinguishability. Experiments on real-world data shows that the proposed method works well in comparison to several competing baselines.