Interpretable Semantic Role Relation Table for Supporting Facts Recognition of Reading Comprehension

1 INTRODUCTION . There has been an increasing interest in the explainability of Machine Reading Comprehension ( MRC ) in recent years . For enhancing the explainability in MRC , some researchers ( Qiu et al. , 2019 ; Tu et al. , 2020 ; Fang et al. , 2020 ) utilize Graph Networks . The relational inductive bias encoded in Graph Networks ( Battaglia et al. , 2018 ) provides viable support for reasoning and learning over structured representations . Some researchers ( Feng et al. , 2020 ) utilize explicit inference chains for multi-hop reasoning . Yang et al . ( 2018 ) provide sentence-level supporting facts required for reasoning , allowing QA systems to reason with strong supervision and explain the predictions . This paper focuses on establishing an interpretable model sentence-level supporting facts recognition . A system capable of delivering explanations is generally more interpretable , meeting some of the requirements for real-world applications , such as user trust , acceptance , and confidence ( Thayaparan et al. , 2020 ) . An example from HotpotQA ( Yang et al. , 2018 ) is illustrated in Figure1 . To correctly answer the question ( ” The director of the romantic comedy Big Stone Gap is based in what New York City ” ) the model is required to first identify ParagraphB as a relevant paragraph , whose title contains the keywords that appear in the question ( ” Big Stone Gap ” ) . S3 , the first sentence of ParagraphB , is then chosen by the model as a supporting fact that leads to the next-hop paragraph ParagraphA . Lastly , from ParagraphA , the span Greenwich Village , New York City is selected as the predicted answer . In this example , s1 and s3 contain the critical information needed to reason the answer . When we judge whether S3 is a supporting fact , our model needs to understand the semantic relation between the S3 , question and the paragraphs . People can identify the supporting factors in the paragraph and give a detailed explanation of the judgment result . We argue that the more specific the semantic interpretation , the more helpful the model imitates the human reasoning process . The input of the most recent model is Pre-trained embeddings , which have the advantage of capturing semantic similarity , but it is hard to explain in detail . We believe that the model uses interpretable features for reasoning , which contributes to enhanced interpretability . Recently , the attention mechanism has achieved remarkable performance on many natural language processing tasks . The model of attention mechanism learns the relevance between words through training ( Figure 2 ) . Inspired by the attention mechanism , for establishing rich and interpretable semantic features , we propose the semantic relational table ( Figure 3 ) . Semantic role labeling ( SRL ) is a shallow semantic parsing task aiming to discover who did what to whom , when and why ( He et al. , 2018 ; Li et al. , 2018 ) , providing explicit contextual semantics , which naturally matches the task target of text comprehension and It is easy to explain for people . Recently , Ribeiro et al . ( 2020 ) show that although measuring held-out accuracy has been the primary approach to evaluate generalization , it often overestimates the performance of NLP models . Moreover , the model does not seem to resolve basic Coreferences and grasp simple subject/object or active/passive distinctions ( SRL ) , all of which are critical to comprehension . Zhang et al . ( 2018 ) regard the semantic signals as SRL embeddings and employ a lookup table to map each label to vectors , similar to the implementation of word embedding . For each word , a joint embedding is obtained by the concatenation of word embedding and SRL embedding . Extensive experiments on benchmark machine reading comprehension and inference datasets verify that the proposed semantic learning helps for the model . The previous work indicates that SRL may hopefully help to understand contextual semantics relation . Formal semantics generally presents the semantic relation as ” predicate argument ” structure . For example , given the following sentence with target verb ( predicate ) sold , all the arguments are labeled as follows , [ ARG0 : Tom ] [ V : sold ] [ ARG1 : a book ] [ ARG2 : to jerry ] [ ARGM-TMP : last week ] . Where ARG0 represents the seller ( agent ) , ARG1 represents the thing sold ( theme ) , ARG2 represents the buyer ( recipient ) , ARGM-TMP is an adjunct indicating the timing of the action , and V represents the predicate . A question sentence example : [ ARG0 : Who ] [ V : bought ] [ ARG1 : flowers ] [ ARG2 : for jerry ] [ ARGM-LOC : in the park ] [ ARGM-TMP : last week ] . In the reference text , the sentences highly related to the semantic roles of the question are essential reasoning information . In this paper , we use the name of entities to integrate into the semantic role relation table to establish the semantic relation between sentences . Since many features are difficult to process efficiently , we simplified the semantic role relation table between two sentences . In particular , the contributions of this work are : ( 1 ) We believe that the model uses interpretable features for reasoning , which contributes to enhanced interpretability . So we propose an interpretable form of semantic relations to enhance the interpretability of the model ’ s input data . We use the enti- ty ’ s name to integrate into the semantic role relation table to establish the semantic relation between question , article , and the target sentence . ( 2 ) With few training data sets , the model ’ s performance based on the semantic role relation table is still stable . 2 METHOD . 2.1 A SIMPLIFIED EXAMPLE . For each sentence , we extract predicate-argument tuples via SRL toolkits . We employ the BERTbased model ( Shi & Lin , 2019 ) in the AllenNLP toolkit to perform SRL . We arrange the different semantic roles labels in a fixed order . A simplified example of supporting sentence prediction task : • question : who eats bread at night ? • sent1 : Tom eats bread . • sent2 : Jerry eats an apple . • sent3 : Tom eats something at night . • sent4 : Jerry drinks milk . We need to recognize sent1 and sent3 as the supporting facts . First step : we use semantic role label tools to parse sentences , set the position that the semantic role label contained in the sentence to 1 , and set the position that the semantic role label is not contained in the sentence to 0 . Second step : We match the entities between the two sentences , the position of the semantic role tag corresponding to the same named entities set to 1 , and the matching result is regarded as the semantic relation between the sentences . The two steps can build the following semantic role relation tables : The ” q ” stands for ” question ” , the ” a ” stands for ” article ” , the ” t ” stands for ” table ” . The question feature table ( Figure 4 ) shows the distribution of the semantic structure information of the question sentence . ” Who ” does not appear in the figure due to it is regarded as a stop word . The semantic role label feature of the question sentence is an essential clue for judging whether the sentence is a piece of evidence . The sent1 is the target sentence for which the model makes evidence prediction . sent1 feature table ( Figure 5 ) , show the distribution of the semantic structure information of the sentence . The article feature table ( Figure 6 ) show the distribution of the global semantic structure information of the article . The question feature sent1 table ( Figure 7 ) and The sent1 feature question table ( Figure 8 ) denotes the semantic relationship between the problem and the target sentence . The sent1 feature article table ( Figure 9 ) and article feature sent1 table ( Figure 10 ) denotes the semantic relationship between the article and the target sentence . The question feature article table ( Figure 11 ) and article feature question table ( Figure 12 ) denotes the semantic relationship between the article and the question . 2.2 MODEL . The model architecture , shown in ( Figure13 ) , the input data of the model is semantic role relation tables ( Figure 3∼11 ) , each figure undergoes the same convolution operation . The k is the set of types of semantic role tags . Let i k , the i-th represents a specific semantic role tag in a sentence . Let the v ( padded where necessary ) means the maximum number of verbs in a sentence . Let the s ( padded where necessary ) means the maximum number of sentences in the article . A convolution operation involves a filter w1 Rvs , which is applied to a window of each semantic role relation table to produce a new feature . ci = f ( w1 · xi : i+1 + b ) ( 1 ) For example , a feature ci is generated from a window of a table . Here b R is a bias term , and f is a non-linear function such as the hyperbolic tangent . This filter is applied to each possible semantic role relation table window to produce a feature map . c1 = [ c1 , c2 , · · · , c25 ] ( 2 ) Another convolution operation involves a filter w2 Rkvs , which is applied to a window of each semantic role relation table to produce a global feature . c2 = f ( w2 · x1 :25 + b ) ( 3 ) g = c1 ⊕ c2 ( 4 ) Where ⊕ is the concatenation operator to contact local features with global features . g = g1 ⊕ g2 · · · ⊕g9 ( 5 ) These features extracted from each semantic role relation table are spliced . Then These features are passed to a fully connected softmax layer whose output is the probability distribution over labels . 3 EXPERIMENT . 3.1 RESULT ANALYSIS . We conduct experiments on the HotpotQA data set . HotpotQA encourages explainable QA models by providing supporting sentences for the answer , which usually come from several documents ( a document is called ” gold doc ” if it contains supporting facts ) . Each case includes ten paragraphs , two of which are ” gold doc. ” We only use ” gold doc ” as an article . Since the test set of HotpotQA is not publicly available , our evaluations are based on the dev set . We use exact match ( EM ) , F1 , Accuracy as three evaluation metrics . The number of sentences and the number of verbs in sentences affects the semantic role feature map size . Therefore , we conducted experiments on the data of the maximum number of sentences ( 5∼10 ) in an article and the maximum number of verbs ( 3∼4 ) in a sentence on the proposed model , and the experimental results are shown in Table 1 . The ( article sent num ) stands for the maximum number of sentences in an article . The ( verb num ) stands for the maximum number of verbs in a sentence The ( case num ) stands for the number of all the articles in the data sets that satisfy the constraint condition . The ( sent num ) stands for the number of all the sentences in the data sets that satisfy the constraint condition . From ( table 1 ) , as the number of sentences in articles increases , EM decreases , F1 decreases , and Accuracy does not change much . As the number of verbs in the sentence increases , EM , F1 , and Accuracy do not change much . There are some experiment result in table 2 . From ( table 2 ) , within a certain range , EM does not Table 2 : Some experimental results in the number of sentences not more than 10 , the number of verbs not more than 4 . The case num = 5219 , the sent num = 30976. article sent num ( ≤ ) verb num ( ≤ ) Accuracy EM F1 10 4 0.7559 0.2433 0.7132 10 4 0.7621 0.2603 0.6939 10 4 0.7628 0.2638 0.6778 increase with the increase of F1 . Sometimes , the EM decreases with the increase of F1 . From ( table 3 ) , Although we did not use all the data in the dev set , we used most of the dev set . so experimental results can be compared to the Baseline . Our model performance should be close to the Baseline model . However , our model structure is simpler than Bert and Baseline . The Baseline model reimplemented the architecture described in Clark & Gardner ( 2018 ) The Baseline model subsumes the technical advances on question answering , including word embedding , character-level models , bi-attention Seo et al . ( 2017 ) and self-attention Wang et al . ( 2017 ) . Recently proposed models such as SAE Tu et al . ( 2020 ) include BERT module , so the performance is excellent . This shows that the semantic role relation table is a critical feature in machine reading comprehension .

Summary: The authors propose a method for adding interpretable semantic features for a Machine Reading Comprehension model  based on semantic relations across words and sentences.  The goal is to use semantic role labeling annotation to produce features used on a neural model for a language downstream task.  The main contributions are:  semantic features based on linguistic annotation, and interpretable semantic features for training a neural model. The study shows that the semantic features used to train Question Answering model performs comparable to related work.