PABI: A Unified PAC-Bayesian Informativeness Measure for Incidental Supervision Signals

1 INTRODUCTION . The supervised learning paradigm , where direct supervision signals are assumed to be available in high-quality and large amounts , has been struggling to fulfill the needs in many real-world AI applications . As a result , researchers and practitioners often resort to datasets that are not collected directly for the target task but , hopefully , capture some phenomena useful for it ( Pan & Yang , 2009 ; Vapnik & Vashist , 2009 ; Roth , 2017 ; Kolesnikov et al. , 2019 ) . However , it remains unclear how to predict the benefits of these incidental signals on our target task beforehand , so the common practice is often trial-and-error : do experiments with different combinations of datasets and learning protocols , often exhaustively , to achieve improvement on a target task ( Liu et al. , 2019 ; Khashabi et al. , 2020 ) . Not only this is very costly , this trial-and-error approach can also be hard to interpret : if we don ’ t see improvements , is it because the incidental signals themselves are not useful for our target task , or is it because the learning protocols we have tried are inappropriate ? The difficulties of foreshadowing the benefits of various incidental supervision signals are two-fold . First , it is hard to provide a unified measure because of the intrinsic differences among different signals ( e.g. , how do we predict and compare the benefit of learning from noisy data and the benefit of knowing some constraints for the target task ? ) . Second , it is hard to provide a practical measure supported by theory . Previous attempts are either not practical or too heuristic ( Baxter , 1998 ; BenDavid et al. , 2010 ; Thrun & O ’ Sullivan , 1998 ; Gururangan et al. , 2020 ) . In this paper , we propose a unified PAC-Bayesian motivated informativeness measure ( PABI ) to quantify the value of incidental signals . We suggest that the informativeness of various incidental signals can be uniformly characterized by the reduction in the original concept class uncertainty they provide . Specifically , in the PAC-Bayesian framework1 , the informativeness is based on the Kullback–Leibler ( KL ) divergence between the prior and the posterior , where incidental signals are used to estimate a better prior ( closer to the gold posterior ) to achieve better generalization performance . Furthermore , we provide a more practical entropy-based approximation of PABI . In practice , PABI first computes the entropy of the prior estimated from incidental signals , and then computes the relative decrease to the entropy of the prior without any information , as the informativeness of incidental signals . 1We choose the PAC-Bayes framework here because it allows us to link PABI to the performance measure . 6 We have been in need of a unified informativeness measure like PABI . For instance , it might be obvious that we can expect better learning performance if the training data are less noisy and more completely annotated , but what if we want to compare the benefits of a noisy dataset and that of a partial dataset ? PABI enables this kind of comparisons beforehand , on a wide range of incidental signals such as partial labels , noisy labels , constraints2 , auxiliary signals , cross-domain signals , and some combinations of them , for sequence tagging tasks in NLP . A specific example of NER is shown in Fig . 1 , and the advantages of PABI are in Table 1 . Finally , our experiments on two NLP tasks , NER and QA , show that there is a strong positive correlation between PABI and the relative improvement for various incidental signals . This strong positive correlation indicates that the proposed unified , theory-motivated measure PABI can serve as a good indicator of the final learning performance , providing a promising way to know which signals are helpful for a target task beforehand . Organization . We start with related work in Section 1.1 . Then we derive informativeness measure PABI in Section 2 . We show examples on how to compute PABI using various incidental signals in Section 3 . We verify the effectiveness of PABI in Section 4 . Section 5 concludes this paper . 1.1 RELATED WORK . There are lots of practical measures proposed to quantify the benefits of specific types of signals . For example , a widely used measure for partial signals in structured learning is the partial rate ( Cour et al. , 2011 ; Hovy & Hovy , 2012 ; Liu & Dietterich , 2014 ; Van Rooyen & Williamson , 2017 ; Ning et al. , 2019 ) ; a widely used measure for noisy signals is the noise ratio ( Angluin & Laird , 1988 ; Natarajan et al. , 2013 ; Rolnick et al. , 2017 ; Van Rooyen & Williamson , 2017 ) ; Ning et al . ( 2019 ) propose to use the concaveness of the mutual information with different percentage of annotations to quantify the strength of constraints in the structured learning ; others , in NLP , have quantified the contribution of constraints experimentally ( Chang et al. , 2012 ; 2008 ) . Bjerva ( 2017 ) proposes to use conditional entropy or mutual information to quantify the value for auxiliary signals . As for domain adaptation , domain similarity can be measured by the performance gap between domains ( Wang et al. , 2019 ) or measures based on the language model in NLP , such as the vocabulary overlap ( Gururangan et al. , 2020 ) . Among them , the most relevant work is ( Bjerva , 2017 ) . However , their conditional entropy or mutual information is based on token-level label distribution , which can not be used for incidental signals involving multiple tokens or inputs , such as constraints and cross-domain signals . At the same time , for the cases where both PABI and mutual information can handle , PABI works similar to the mutual information as shown in Fig . 2 , and PABI can further be shown to be a strictly increasing function of the mutual information . The key advantage of our proposed measure PABI is that PABI is a unified measure motivated by the PAC-Bayesian theory for a broader range of incidental signals compared to these practical measures for specific types of incidental signals . There also has been a line of theoretical work that attempts to exploit incidental supervision signals . Among them , the most relevant part is task relatedness . Ben-David & Borbely ( 2008 ) define the 2Constraints are used to model the dependency among words and sentences , which are considered in a lot of work , such as CRF ( Lafferty et al. , 2001 ) and ILP ( Roth & Yih , 2004 ) . task relatedness based on the richness of transformations between inputs for different tasks , but their analysis is limited to cases where data is from the same classification problem but the inputs are in different subspace . Juba ( 2006 ) proposes to use the joint Kolmogorov complexity ( Li et al. , 2008 ) to characterize relatedness , but it is still unclear how to compute the joint Kolmogorov complexity in real-world applications . Mahmud & Ray ( 2008 ) further propose to use conditional Kolmogorov complexity to measure the task relatedness and provide empirical analysis for decision trees , but it is unclear how to use their relatedness for other models , such as deep neural networks . Thrun & O ’ Sullivan ( 1998 ) propose to cluster tasks based on the similarity between the task-optimal distance metric of k-nearest neighbors ( KNN ) , but their analysis is based on KNN and it is unclear how to use their relatedness for other models . A lot of other works provide quite good qualitative analysis to show various incidental signals are helpful but they did not provide quantitative analysis to quantify to what extent these types of incidental signals can help ( Balcan & Blum , 2010 ; Abu-Mostafa , 1993 ; Natarajan et al. , 2013 ; Van Rooyen & Williamson , 2017 ; Baxter , 1998 ; London et al. , 2016 ; Ciliberto et al. , 2019 ; Ben-David et al. , 2010 ; Wang et al. , 2020 ) . Compared to these theoretical analyses , PABI can be easily used in practice to quantify the benefits of a broader range of incidental signals . 2 PABI : A UNIFIED PAC-BAYESIAN INFORMATIVENESS MEASURE We start with notations and preliminaries . Let X be the input space , Y be the label space , and Ŷ be the prediction space . LetD denote the underlying distribution onX ×Y . Let ` : Y×Ŷ → R+ be the loss function that we use to evaluate learning algorithms . A set of training samples S = { xi , yi } mi=1 is generated i.i.d . from D. In the common supervised learning setting , we usually assume the concept that generates data comes from the concept class C. In this paper , we assume C is finite , and its size is | C | , which is common in NLP . We want to choose a predictor c : X → Ŷ from C such that it generalizes well to unseen data with respect to ` , measured by the generalization error RD ( c ) = Ex , y∼D [ ` ( y , c ( x ) ) ] . The training error over S is RS ( c ) = 1m ∑m i=1 ` ( yi , c ( xi ) ) . More generally , instead of predicting a concept , we can specify a distribution over the concept class . Let P denote the space of probability distributions over C. General Bayesian learning algorithms ( Zhang et al. , 2006 ) in the PAC-Bayesian framework ( McAllester , 1999a ; b ; Seeger , 2002 ; McAllester , 2003b ; a ; Maurer , 2004 ; Guedj , 2019 ) aim to choose a posterior πλ ∈ P over the concept class C based on a prior π0 ∈ P and training data S , where λ is a hyper parameter that controls the tradeoff between the prior and the data likelihood . In this setting , the training error and the generalization error need to be generalized , to take the distribution into account , as LS ( πλ ) = Ec∼πλ [ RS ( c ) ] and LD ( πλ ) = Ec∼πλ [ RD ( c ) ] respectively . One can easily see that when the posterior is one-hot ( exactly one entry of the distribution is 1 ) , we have the original definitions of training error and generalization error , as in the PAC framework ( Valiant , 1984 ) . 4Papers are denoted by abbreviations of author names as : CST ’ 11 ( Cour et al. , 2011 ) , HH ’ 12 ( Hovy & Hovy , 2012 ) , LD ’ 14 ( Liu & Dietterich , 2014 ) , AL ’ 88 ( Angluin & Laird , 1988 ) , NDRT ’ 13 ( Natarajan et al. , 2013 ) , RVBS ’ 17 ( Rolnick et al. , 2017 ) , VW ’ 17 ( Van Rooyen & Williamson , 2017 ) , WNR ’ 20 ( Wang et al. , 2020 ) , NHFR ’ 19 ( Ning et al. , 2019 ) , B ’ 17 ( Bjerva , 2017 ) , GMSLBDS ’ 20 ( Gururangan et al. , 2020 ) . 2.1 INFORMATIVENESS MEASURES IN THE PAC-BAYESIAN FRAMEWORK . We are now ready to derive the proposed informativeness measure PABI motivated by PAC-Bayes . The generalization error bound in the PAC-Bayesian framework ( Guedj , 2019 ; Catoni , 2007 ) says that with probability 1 − δ over S , LD ( πλ∗ ) ≤ LD ( π∗ ) + √ 8B ( DKL ( π∗||π0 ) +ln 2 ln ( mC ) δ ) m , where πλ∗ is the posterior distribution with the optimal λ∗ = √ 2m ( DKL ( π∗||π0 ) +ln 2δ ) B , π ∗ ∈ P is the gold posterior that generates the data , DKL ( π∗||π0 ) denotes the KL divergence from π0 to π∗ , B and C are two constants . This is based on the Theorem 2 in Guedj ( 2019 ) . As shown in the generalization bound , the generalization error is bounded by the KL divergence DKL ( π ∗||π0 ) from the prior distribution to the gold posterior distribution . Therefore , we propose to utilize incidental signals to improve the prior distribution from π0 to π̃0 so that it is closer to the gold posterior distribution π∗ . Correspondingly , we can define PABI , the informativeness measure for incidental supervision signals , by measuring the improvement with regard to the gold posterior , in KL divergence sense . Definition 2.1 ( PABI ) . Suppose we use incidental signals to improve the prior distribution from π0 to π̃0 . The informativeness measure for incidental signals , PABI , is defined as S ( π0 , π̃0 ) , √ 1− DKL ( π ∗||π̃0 ) DKL ( π∗||π0 ) ( 1 ) Remark . Note that S ( π0 , π̃0 ) = 0 if π̃0 = π0 , while if π̃0 = π∗ , then S ( π0 , π̃0 ) = 1 . This result is consistent with our intuition that the closer π̃0 is to π∗ , the more benefits we can gain from incidental signals . The square root function is used in PABI for two reasons : first , the generalization bounds in both PAC-Bayesian and PAC ( see Sec . 2.2 ) frameworks have the square root function ; second , in our later experiments , we find that square root function can significantly improve the Pearson correlation between the relative performance improvement and PABI . It is worthwhile to note that the square root is not crucial for our framework , because our goal is to compare the benefits among different incidental supervision signals , where the relative values are expressive enough . In this sense , any strictly increasing function in [ 0 , 1 ] over the current formulation would be acceptable . In our paper , we focus on the setting that the gold posterior π∗ is one-hot , which means π∗ concentrates on the true concept c∗ ∈ C , though the definition of PABI can handle general gold posterior . However , π∗ is unknown in practice , which makes Eq . ( 1 ) hard to be computed in reality . In the following , we provide an approximation Ŝ of PABI . Definition 2.2 ( Approximation of PABI ) . Assume that the original prior π0 is uniform , and the gold posterior π∗ is one-hot concentrated on the true concept c∗ in C , as we have assumed that C is finite . Let H ( · ) be the entropy function . The approximation Ŝ of PABI is defined as Ŝ ( π0 , π̃0 ) , √ 1− H ( π̃0 ) H ( π0 ) = √ 1− H ( π̃0 ) ln | C | ( 2 ) The uniform prior π0 is usually used when we do not have information about the prior on which concept in the class that generates data . The intuition behind Ŝ is that , it measures how much entropy incidental signals reduce , compared with non-informative prior π0 . Ŝ can be computed through data and thus is practical . To see how this approximation works , first note DKL ( π∗||π0 ) = ln | C | because π∗ is one-hot and π0 is uniform over the finite concept class C. Let πc be the one-hot distribution concentrated on concept c for each c ∈ C. The approximation is that we estimate π∗ by πc , where c follows π̃0 : DKL ( π∗||π̃0 ) ≈ Ec∼π̃0DKL ( πc||π̃0 ) . It turns out Ec∼π̃0DKL ( πc||π̃0 ) = H ( π̃0 ) . Therefore , Ŝ ( π0 , π̃0 ) = √ 1− H ( π̃0 ) ln | C | = √ 1− Ec∼π̃0DKL ( πc||π̃0 ) ln | C | ≈ √ 1− DKL ( π ∗||π̃0 ) DKL ( π∗||π0 ) = S ( π0 , π̃0 ) . We later show that the approximation of PABI and PABI is equivalent in the non-probabilistic cases with the finite concept class , indicating the quality of this approximation . Furthermore , the effectiveness of this approximation in NLP applications also indicates the quality of this approximation . 2.2 PABI IN THE PAC FRAMEWORK We have derived PABI in the PAC-Bayesian framework . Here , we discuss briefly on what PABI reduces to in the PAC framework and what limitations are when PABI is restricted to the PAC framework . The generalization bound in the PAC framework ( Mohri et al. , 2018 ) says with probability 1− δ over S , RD ( c ) ≤ RS ( c ) + √ ln | C |+ln 2δ 2m . We propose to reduce the concept class from C to C̃ by using incidental signals . Then PABI in the PAC framework can be written as S ( C , C̃ ) = √ 1− ln |C̃| ln | C | ( 3 ) It turns out that Eq . ( 3 ) is a special case of Eq . ( 1 ) when π∗ is one-hot over C , π0 is uniform over C and π̃0 is uniform over C̃ ( See Appx . A.1 for the derivation ) . As shown in Appx . A.1 , we can see that the three informativeness measures , PABI in Eq . ( 1 ) , the approximation of PABI in Eq . ( 2 ) , and PABI in the PAC framework in Eq . ( 3 ) , are equivalent , i.e . S ( π0 , π̃0 ) = Ŝ ( π0 , π̃0 ) = S ( C , C̃ ) , in the non-probabilistic cases with the finite concept class . The equivalence among three measures further indicates that both PABI and the approximation of PABI are reasonable . However , PABI restricted to the PAC framework can not handle the probabilistic cases . For example , incidental signals can reduce the probability of some concepts , though the concept class is not reduced . In this example , S ( C , C̃ ) is zero , but we actually benefit from incidental signals . Some analysis on the extensions and general limitations of PABI can be found in Appx . A.2 and A.3 . We need to notice that the size of concept class also plays an important role in the lower bound on the generalization error ( more details in Appx . A.4 ) , indicating that PABI based on the reduction of the concept class is a reasonable measure . 3 EXAMPLES FOR PABI In this section , we show some examples of sequence tagging tasks5 in NLP for PABI . Similar to the categorization of transfer learning ( Pan & Yang , 2009 ) , we use inductive signals to denote the signals with a different conditional probability distribution ( P ( y |x ) ) from gold signals , such as noisy and auxiliary signals , and transductive signals to denote the signals with the same task ( P ( y |x ) ) as gold signals but a different marginal distribution of x ( P ( x ) ) from gold signals , such as cross-domain and cross-lingual signals . In our following analysis , we study the tasks with finite concept class which is quite common in NLP . For simplicity , we focus on simple cases where the number of incidental signals is large enough . How different factors ( including base model performance , size of incidental signals , data distributions , algorithms , cost-sensitive losses ) affect PABI are discussed in Appx . A.5 . We derive the PABI for partial labels in detail and the derivations for others are similar . More examples and details can be found in Appx . A.6 .

Summary: This paper proposes PABI (PAC-Bayesian Informativeness?), a way of measuring and predicting the usefulness of “incidental supervision signal” for a downstream classification task. In particular, when labeled data is only available in noisy or partial form, or over a different domain than the target test domain, this data may still be used to improve a classifier, but it’s unclear how to tell which forms of incidental supervision will be most useful. Having a measure which allows us to compare different types of such supervision enables us to make intelligent tradeoffs.