Counterfactual Thinking for Long-tailed Information Extraction

1 INTRODUCTION . The goal of Information Extraction ( IE ) ( Sarawagi , 2008 ; Chiticariu et al. , 2013 ) is to detect the structured information from unstructured texts . IE tasks , such as named entity recognition ( NER ) ( Lample et al. , 2016 ) , relation extraction ( RE ) ( Zeng et al. , 2014 ; Peng et al. , 2017 ) and event detection ( ED ) ( Nguyen & Grishman , 2015 ) have developed rapidly with the data-hungry deep learning models trained on a large amount of data . However , in real-world settings , unstructured texts follow a long-tailed distribution ( Doddington et al. , 2004 ) , leading to a significant performance drop on the instance-scarce ( or tail ) classes which have very few instances available . For example , in the ACE2005 ( Doddington et al. , 2004 ) dataset , nearly 70 % of event triggers are long-tailed while they only take up 20 % of training data . On a strong baseline ( Jie & Lu , 2019 ) , the macro F1 score of instance-rich ( or head ) classes can be 71.6 , while the score of tail classes sharply drops to 41.7 . The underlying causes for the above issues are the biased statistical dependencies and spurious correlations between feature representations and classes learned from an imbalanced dataset . For example , an entity Gardens appears 13 times in the training set of OntoNotes5.0 ( Pradhan et al. , 2013 ) , with the NER tag LOC , and only 2 times as organization ORG . A classifier trained on this dataset will build a spurious correlations between Gardens and LOC . As a result , an organization that contains the entity Gardens may be wrongly predicted as a location LOC . There are only a few studies ( Zhang et al. , 2019 ; Han et al. , 2018 ) in the Natural Language Processing ( NLP ) field to address such long-tailed issues . These works mostly rely on external and pre-constructed knowledge graphs , providing useful data-specific prior information which may not be available for other datasets . On the other hand , there are plenty of works from the computer vision society , where the bias is also quite straightforward . Current solutions include re-balanced training ( Lin et al. , 2017 ) that re-balances the contribution of each class in the training stage , transfer learning ( Liu et al. , 2019b ) that takes advantage of the knowledge in data-rich class to boost the performance of instance-scarce classes , decoupling ( Kang et al. , 2019 ) strategy that learns the representations and classifiers separately , and causal inference ( Tang et al. , 2020a ; b ; Abbasnejad et al. , 2020 ) that relies on structured causal models for unbiased scene graph generation , image classification and visual question answering . The aforementioned studies from the computer vision community may not achieve good performance on the textual datasets in the NLP area due to a significant difference between the two fields . For example , unlike images , texts involve complex language structures such as dependency tree and constituent tree that describe the syntactic or semantic level relations between tokens . For the longtailed IE , how to explore the rich relational information as well as complex long-distance interactions among words as conveyed by such linguistic structures remains an open challenge . Furthermore , to capture a more informative context , the way of utilizing the syntax tree for three IE tasks varies : the RE task relies more on the context and entity type rather than entities themselves , while classifications in NER and ED tasks count more on entities than the context . Hence , it is challenging to decide properly on how to utilize language structures for the above three different IE tasks . One may also think that the prevalent pre-trained models such as BERT ( Devlin et al. , 2019 ) may address the long-tailed issues . However , we empirically show that such models still suffer from bias issues . In this paper , we propose CFIE , a novel framework that combines the language structure and counterfactual analysis in causal inference ( Pearl et al. , 2016 ) to alleviate the spurious correlations of the IE tasks including NER , RE and ED . From a causal perspective , counterfactuals ( Bottou et al. , 2013 ; Abbasnejad et al. , 2020 ) state the results of the outcome if certain factors had been different . This concept entails a hypothetical scenario where the values in the causal graph can be altered to study the effect of the factor . Intuitively , the factor that yields the most significant changes in model predictions have the greatest impact and is therefore considered as main effect . Other factors with minor changes are categorized as side effects . In the context of IE with complex language structures , counterfactual analysis answers the question on “ which tokens in the text would be the key clues for RE , NER or ED that could change the prediction result ? ” . With that in mind , our CFIE is proposed to explore the language structure to eliminate the bias caused by the side effect and maintain the main effect for the classification . We evaluate our model on five public datasets across three IE tasks , and achieve significant performance gain on instance-scarce classes . We will release our code to contribute the community . Our major contributions are summarized as : • To the best of our knowledge , our CFIE is the first attempt that marries the counterfactual analysis and language structure to address the long-tailed IE issues . We build different structured causal models ( SCMs ) ( Pearl et al. , 2016 ) for the IE tasks and fuse the dependency structure to the models to better capture the main causality for the classification . • We generate counterfactuals based on syntax structure , where the counterfactuals can be used as interventions to alleviate spurious corrections on models . In doing so , the main effect can be better estimated by the intervention methodology . • We also propose flexible classification debiasing approaches inspired by Total Direct Effect ( TDE ) in causal inference . Our proposed approach is able to make a good balance between the direct effect and counterfactuals representation to achieve more robust predictions . 2 RELATED WORK . Long-tailed Information Extraction : Information extraction tasks , such as relation extraction ( Zeng et al. , 2014 ; Peng et al. , 2017 ; Quirk & Poon , 2017 ) , named entity recognition ( Lample et al. , 2016 ; Chiu & Nichols , 2016 ) , and event extraction ( Nguyen & Grishman , 2015 ; Huang et al. , 2018 ) are fundamental NLP tasks and have been extensively studied in recent years , For the long-tailed IE , recent models ( Lei et al. , 2018 ; Zhang et al. , 2019 ) leverage external rules or transfer knowledge from data-rich classes to the tail classes . Few-shot leaning ( Gao et al. , 2019 ; Obamuyide & Vlachos , 2019 ) has been also applied to IE tasks , although this task focuses more on new classification tasks with only a handful of training instances . Re-balancing/Decoupling Models : Re-balancing approaches include re-sampling strategies ( Mahajan et al. , 2018 ; Wang et al. , 2020a ) that aim to alleviate statistical bias from head classes , and re-weighting approaches ( Milletari et al. , 2016 ; Lin et al. , 2017 ) which assign balanced weights to the losses of training samples from each class to boost the discriminability via robust classifier decision boundaries . These techniques may inevitably suffer the under-fitting/over-fitting issue to head/tail classes ( Tang et al. , 2020a ) . There are also recent studies ( Kang et al. , 2019 ) that decouple the representation learning and the classifier , which effectively mitigate the performance loss caused by direct re-sampling . Casual Inference : Causal inference ( Pearl et al. , 2016 ; Rubin , 2019 ) and counterfactuals have been widely used in psychology , politics and epidemiology for years . There are many studies in computer vision society ( Tang et al. , 2020b ; Abbasnejad et al. , 2020 ; Tang et al. , 2020a ; Niu et al. , 2020 ; Yang et al. , 2020 ; Zhang et al. , 2020 ; Yue et al. , 2020 ) , which use Total Direct Effect ( TDE ) analysis framework and counterfactuals for Scene Graph Generation ( SGG ) , visual question answering , and image classifications . There is also a recent work ( Zeng et al. , 2020 ) that generates counterfactuals for weakly-supervised NER by replacing the target entity with another entity . Our methods differ from the previous works in three aspects : 1 ) We explore the syntax structures of texts for building different causal graphs , 2 ) Counterfactuals are generated based on a task-specific pruned dependency tree . 3 ) Our proposed inference method yields robust predictions for the NER and ED tasks . Model Interpretation : Besides causal inference , there have been plenty of studies ( Molnar , 2020 ) about traditional model interpretation applied in various applications , such as text and image classification ( Ribeiro et al. , 2016 ; Ebrahimi et al. , 2018 ) , question answering ( Feng et al. , 2018 ; Ribeiro et al. , 2018 ) , and machine translation ( Doshi-Velez & Kim , 2017 ) . LIME ( Ribeiro et al. , 2016 ) was proposed to select a set of instances to explain the predictions . The input reduction method ( Feng et al. , 2018 ) is able to find out the most important features and use very few words to obtain the same prediction . Unlike the LIME and input reduction method , the word selections in our CFIE are based on the syntax structure . SEARs ( Ribeiro et al. , 2018 ) induces adversaries by data augmentation during the training phase . Along this line , a recent study ( Kaushik et al. , 2019 ) also uses data augmentation technqiue to provide extra training signal . Our CFIE is orthogonal to data augmenation as it generates counterfactuals during the inference stage , where the counterfactuals are used to mitigate the spurious correlations rather than training the network parameters . 3 MODEL . Figure 1 shows the work flow of our proposed CFIE . We detail these components as follows . 3.1 STEP1 : CAUSAL REPRESENTATION LEARNING . In this step , we train a causal graph on an imbalanced dataset . Our goal here is to teach the model to identify the main cause ( main effect ) and the spurious correlations ( side effect ) for the classification . Structural Causal Models ( SCMs ) : The two well-known causal inference frameworks are SCMs and potential outcomes ( Rubin , 2019 ) which are complementary and theoretically connected . We choose SCMs in our case due to their advantages in expressing and reasoning about the effects of causal relationships among variables . An SCM can be represented as a directed acyclic graph ( DAG ) G = { V , F , U } , where we denote the set of observables ( vertices ) as V = { V1 , ... , Vn } , the set of functions ( directed edges ) as F = { f1 , ... , fn } , and the set of exogenous variables ( e.g . noise ) as U = { U1 , ... , Un } . Note that in the deterministic case where U is given , the value of all variables in the SCM are uniquely determined ( Pearl , 2009 ) . Each observable Vi can be derived from : Vi : = fi ( PAi , Ui ) , ( i = 1 , ... , n ) , ( 1 ) ∀i , PAi ⊆ V\Vi is the set of parents of Vi . Directed edges , such as PAi → Vi in the graph G , i.e. , fi , refers to the direct causation from the parental variables PAi to the child variable Vi . Our Proposed SCMs : Figure 2 ( a ) demonstrates our unified SCMs for IE tasks , which are built based on our prior knowledge for the tasks . The variable S indicates the contextualized representations of an unstructured input sentence , where the representations are the output from a BiLSTM ( Schuster & Paliwal , 1997 ) or pre-trained BERT encoder ( Devlin et al. , 2019 ) . Zi ( i ∈ [ 1 , m ] ) represents features such as the NER tags and part-of-speech ( POS ) tagging . The variable X is the representation of a target relation for RE , entity representation for NER , or trigger representation for ED , and Y indicates the output logits for classification . S X P Y S N X P Y ( a ) ( b ) Let E = { S , X , Z1 , ... , Zm } denotes the parents of Y . The direct causal effects towards Y including X → Y , S → Y , Z1 → Y , .... , Zm → Y are linear transformations . For each edge i → Y , its transformation is denoted as WiY ∈ Rc×d , where i ∈ E and c is the number of classes . We let Hi ∈ Rd×h denote h1 representations with d dimensions for node i ∈ E . Then , the prediction can be obtained by summation Yx = ∑ i∈EWiY Hi or gated mechanism Yx = WgHX σ ( ∑ i∈EWiY Hi ) , where refers to element-wise product , Wg ∈ Rc×d is the linear transformation , and σ ( · ) in- dicates the sigmoid function . To avoid any single edge , such as S → Y , dominating the generation of the logits Yx , we add a cross-entropy loss LiY , i ∈ E for each branch , where i indicates the parent of the node Y . Let LY denote the loss for Yx , the total loss L can be computed by : L = LY + ∑ i∈E LiY ( 2 ) Note that the proposed SCM is encoder neutral . The SCM can be equipped with various encoders , such as BiLSTM , BERT and Roberta ( Liu et al. , 2019a ) . For simplicity , we omit exogenous variables U from the graph as its only useful for the derivations in the following sections . Fusing Syntax Structures Into SCMs : So far we have built basic SCMs for IE tasks . On the edge S → X , we adopt different neural networks architectures for RE , NER and ED . For RE , we use dependency trees to aggregate long-range relations with graph convolution networks ( GCN ) ( Kipf & Welling , 2017 ) . Assume the length of the sentence is h. For the GCN , we generate a matrix A ∈ Rh×h from a dependency tree . The convolution computation for the node i at the l-th layer takes the representation xl−1i from previous layer as input and outputs the updated representations xli . The formulation is given as : xli = σ ( l∑ j=1 AijW lxl−1i + b l ) , i ∈ [ 1 , h ] ( 3 ) where Wl and bl are the weight matrix and bias vector of the l-th layer respectively , and σ ( · ) is the sigmoid function . Here x0 takes value from HS and HX takes value from the output of the last GCN layer xlmax . For NER and ED , we adopt the dependency-guided concatenation approach ( Jie & Lu , 2019 ) . Given a dependency edge ( th , ti , r ) with th as a head ( parent ) , ti as a dependent ( child ) and r is the dependency relation between them , the representations of the dependent ( assume at the 1h is the sequence length for NER and ED , and h = 1 for relation extraction . i-th position of a sentence ) can be denoted as : xi = [ H ( i ) S ; H ( h ) S ; vr ] , th = parent ( ti ) HX = LSTM ( x ) ( 4 ) where H ( i ) S and H ( h ) S are the word representations of the word ti and its parent th , vr denotes the learnable embedding of dependency relation r .

Summary: The novelty of the paper seems to be in application of the counterfactual analysis to address the long-tailed IE issues, which might be interesting to the IE researchers. Overall, more theory about the counterfactual generation for IE task should be added, for this is what the novelty of the paper; also, for the rebalancing learning for slide effect and counterfactual, the theory appears to be not enough. The weak of this work is the theoretical and conceptual underpinnings of the proposed methodology.