Creating Training Sets via Weak Indirect Supervision

1 INTRODUCTION . One of the greatest bottlenecks of using modern machine learning models is the need for substantial amounts of manually-labeled training data . In real-world applications , such manual annotations are typically time-consuming , labor-intensive and static . To reduce the efforts of annotation , researchers have proposed Weak Supervision ( WS ) frameworks ( Ratner et al. , 2016 ; 2018 ; 2019 ; Fu et al. , 2020 ) for synthesizing labels from multiple weak supervision sources , e.g. , heuristics , knowledge bases , or pre-trained classifiers . These frameworks have been widely applied on various machine learning tasks ( Dunnmon et al. , 2020 ; Fries et al. , 2021 ; Safranchik et al. , 2020 ; Lison et al. , 2020 ; Zhou et al. , 2020 ; Hooper et al. , 2021 ; Zhan et al. , 2019 ; Varma et al. , 2019 ) and industrial data ( Bach et al. , 2019 ) . Among them , data programming ( Ratner et al. , 2016 ) , one representative example that generalizes many approaches in the literature , represents weak supervision sources as labeling functions ( LFs ) and synthesizes training labels using Probabilistic Graphical Model ( PGM ) . Given both the increasing popularity of WS and the general increase in open-source availability of machine learning models and tools , there is a rising tide of available supervision sources that WS frameworks and practitioners could potentially leverage , including pre-trained machine learning models or prediction APIs ( Chen et al. , 2020 ; d ’ Andrea & Mintz , 2019 ; Yao et al. , 2017 ) . However , existing WS frameworks only utilize weak supervision sources with the same label space as the target task . This incompatibility largely limits the scope of usable sources , necessitating manual effort from domain experts to provide supervision for unseen labels . For example , consider target task of classifying { “ dog ” , “ wolf ” , “ cat ” , “ lion ” } and a set of three weak supervision sources ( e.g . trained classifiers or expert heuristics ) with disjoint output spaces { “ caninae ” , “ felidae ” } , { “ domestic animals ” , “ wild animals ” } and { “ husky ” , “ bengal cat ” } respectively . We call these types of sources indirect supervision sources . For concreteness , we follow the general convention of data programming ( Ratner et al. , 2016 ) and refer to these sources as indirect labeling functions ( ILFs ) . Despite their apparent utility , existing weak supervision methods could not directly leverage such ILFs , as their output spaces have no overlap with the target one . In this paper , we formulate a novel research problem that aims to leverage such ILFs automatically , minimizing the manual efforts to develop and deploy new models . We refer to this as the Weak Indirect Supervision ( WIS ) setting , a new Weak Supervision paradigm which leverages ILFs , along with the relational structures between individual labels , to automatically create training labels . The key difficulty of leveraging ILFs is due to the mismatched label spaces . To overcome this , we introduce pairwise relations between individual labels to the WIS setup , which are often available in structured sources ( e.g . off-the-shelf Knowledge Bases ( Miller , 1995 ; Sinha et al. , 2015 ; Dong et al. , 2020 ) or large scale label hierarchies ( Murty et al. , 2017 ; The Gene Ontology Consortium , 2018 ; Partalas et al. , 2015 ) for various domains ) , or can be provided by subject matter experts in far less time than generating entirely new sets of weak supervision sources . For example , in the aforementioned example , we could rely on a biological species ontology to see that the unseen labels “ dog ” and “ cat ” are both subsumed by the seen label “ domestic animals ” . Based on the label relations , we can automatically leverage the supervision sources as ILFs . Notably , previous work ( Qu et al. , 2020 ) also leveraged a label relation graph but was focused on relation extraction task in a few-shot learning setting , while You et al . ( 2020 ) proposed to learn label relations given data for each label in a transfer learning scenario . In contrast , we aim to solve the target task directly and without clean labeled data . The remaining questions are ( 1 ) how to synthesize labels based on pair-wise label relations and ILFs ? and ( 2 ) How can we know whether , given a set of ILFs and label relations , the unseen labels are distinguishable or not ? To address the first question , we develop a probabilistic label relation model ( PLRM ) , the first PGM for WIS which aggregates the output of ILFs and models the label relations as dependencies between random variables . In turn , we use the learned PLRM to produce labels for training an end model . Furthermore , we derive the generalization error bound of PLRM based on assumptions similar to previous work ( Ratner et al. , 2016 ) . The second question presents an important stumbling block when dealing with unseen labels , as we may not be able to distinguish the unseen labels given existing label relations and ILFs , resulting in an unsatisfactory synthesized training set . To address this issue , we formally introduce the notion of distinguishability in WIS setting and theoretically establish an equivalence between : ( 1 ) the distinguishability of the label relation structure as well as the ILFs , and ( 2 ) the capability of PLRM to distinguish unseen labels . This result then leads to a simple sanity test for preventing the model from failing to distinguish unseen labels . In preliminary experiments , we observe a significant drop in model performance when the condition is violated . In experiments , we make non-trivial adaptations for baselines from related settings to the new WIS problem . On both text and image classification tasks , we demonstrate the advantages of PLRM over adapted baselines . Finally , in a commercial advertising system where developers need to collect annotations for new ads tags , we illustrate how to formulate the training label collection as a WIS problem and apply PLRM to achieve an effective performance . Summary of Contributions . Our contributions are summarized as follows : • We formulate Weak Indirect Supervision ( WIS ) , a new research problem which synthesizes training labels based on indirect supervision sources and label relations , minimizing human efforts of both data annotation and weak supervision sources construction ; • We develop the first model for WIS , the Probabilistic Label Relation Model ( PLRM ) with comparable statistical efficiency to previous WS frameworks and standard supervised learning ; • We introduce a new notion of distinguishability in WIS setting , and provide a simple test of the distinguishability of PLRM for unseen labels by theoretically establishing the connection between the label relation structures and distinguishability ; • We showcase the potential of the WIS formulation and the effectiveness of PLRM in a commercial advertising system for synthesizing training labels of new ads tags . On academic image and text classification tasks , we demonstrate the advantages of PLRM over baselines by quantitative experiments . Overall , PLRM outperforms baselines by a margin of 2 % -9 % . 2 RELATED WORK . We briefly review related settings . The comparison between WIS and related tasks is in Table 1 . Weak Supervision : We draw motivation from recent work which model and integrate weak supervision sources using PGMs ( Ratner et al. , 2016 ; 2018 ; 2019 ; Fu et al. , 2020 ) and other methods ( Guan et al. , 2018 ; Khetan et al. , 2018 ) to create training sets . While they assume supervision sources share the same label space as the new tasks , we aim to leverage indirect supervision sources with mismatched label spaces in a labor-free way . Indirect Supervision : Indirect supervision arises more generally in latent-variable models for various domains ( Brown et al. , 1993 ; Liang et al. , 2013 ; Quattoni et al. , 2004 ; Chang et al. , 2010 ; Zhang et al. , 2019 ) . Very recently , Raghunathan et al . ( 2016 ) proposed to use the linear moment method for indirect supervision , wherein the transition between desired label space Y and indirect supervision space O is known , as well as the ground truth of indirect supervisions for training . In contrast , both are unavailable in WIS. Theoretically , Wang et al . ( 2020 ) developed a unified framework for analyzing the learnability of indirect supervision with shared or superset label spaces , while we focus on disjoint label spaces and the consequent unique challenge of distinguishability of unseen classes . Zero-Shot Learning : Zero-Shot Learning ( ZSL ) ( Lampert et al. , 2009 ; Wang et al. , 2019 ) aims to learn a classifier that is able to generalize to unseen classes . The WIS problem differentiates from ZSL by ( 1 ) in ZSL setting , the training and test data belong to seen and unseen classes , respectively , and training data is labeled , while for WIS , both training and test data belong to unseen classes and unlabeled ; ( 2 ) ZSL tends to render a classifier that could predict unseen classes given certain label information , e.g. , label attributes ( Romera-Paredes & Torr , 2015 ) , label descriptions ( Srivastava et al. , 2018 ) or label similarities ( Frome et al. , 2013 ) , while WIS aims to provide training labels for unlabeled training data , allowing users to train any machine learning models , and the label relations are used only in synthesizing training labels . 3 PRELIMINARY : WEAK SUPERVISION . We first describe the Weak Supervision ( WS ) setting . A glossary of notations used is in App . A. Definitions and notations . We assume a k-way classification task , and have an unlabeled dataset D consisting of m data points . Denote by Xi ∈ X the individual data point and Yi ∈ Y = { y1 , . . . , yk } the unobserved interested label of Xi . We also have n sources , each represented by a labeling function ( LF ) and denoted by λj . Each λj outputs a label Ŷ j i ∈ Yλj = { ŷj1 , . . . , ŷjkλj } on Xi , where Yλj is the label space associated with λj and |Yλj | = kλj . We denote the concatenation of LFs ’ output as Ŷi = [ Ŷ 1i , Ŷ 2 i , . . . , Ŷ n i ] , and the union set of LFs ’ label spaces as Ŷ with |Ŷ| = k̂ . Note that k̂ is not necessarily equal to the sum over kλj , since LFs may have overlapping label spaces . We call ŷ ∈ Ŷ seen label and y ∈ Y desired labels . In WS settings , we have Y ⊂ Ŷ . Notably , we assume all the involved labels come from the same semantic space . The goal of WS . The goal is to infer the training labels for the dataset D based on LFs , and to use them to train an end discriminative classifier fW : X → Y , all without ground truth training labels .

Summary: This paper studied a weakly supervised classification problem, called **weak indirect supervision**, where the supervision signals are from labels that are different from but still informative of the classes. The author proposed a new two-step method that first creates probabilistic labels using an **exponential family graphical model** based on (1) a set of **indirect labeling functions** (pretrained classifier, heuristic rules, etc.) that output deterministic labels and (2) a given **label relation graph** (ontology graph, knowledge base, etc.) that captures the relations between the observed labels and the target classes, then uses the generated labels to train a classifier for the target classes. The author provided a theoretical analysis on the requirements of the labeling functions and derived a generalization error bound. The proposed method was evaluated on semi-synthetic datasets based on the ImageNet dataset for image classification and the LSHTC dataset for text classification.