Interpretable Semantic Role Relation Table for Supporting Facts Recognition of Reading Comprehension

1 INTRODUCTION . There has been an increasing interest in the explainability of Machine Reading Comprehension ( MRC ) in recent years . For enhancing the explainability in MRC , some researchers ( Qiu et al. , 2019 ; Tu et al. , 2020 ; Fang et al. , 2020 ) utilize Graph Networks . The relational inductive bias encoded in Graph Networks ( Battaglia et al. , 2018 ) provides viable support for reasoning and learning over structured representations . Some researchers ( Feng et al. , 2020 ) utilize explicit inference chains for multi-hop reasoning . Yang et al . ( 2018 ) provide sentence-level supporting facts required for reasoning , allowing QA systems to reason with strong supervision and explain the predictions . This paper focuses on establishing an interpretable model sentence-level supporting facts recognition . A system capable of delivering explanations is generally more interpretable , meeting some of the requirements for real-world applications , such as user trust , acceptance , and confidence ( Thayaparan et al. , 2020 ) . An example from HotpotQA ( Yang et al. , 2018 ) is illustrated in Figure1 . To correctly answer the question ( ” The director of the romantic comedy Big Stone Gap is based in what New York City ” ) the model is required to first identify ParagraphB as a relevant paragraph , whose title contains the keywords that appear in the question ( ” Big Stone Gap ” ) . S3 , the first sentence of ParagraphB , is then chosen by the model as a supporting fact that leads to the next-hop paragraph ParagraphA . Lastly , from ParagraphA , the span Greenwich Village , New York City is selected as the predicted answer . In this example , s1 and s3 contain the critical information needed to reason the answer . When we judge whether S3 is a supporting fact , our model needs to understand the semantic relation between the S3 , question and the paragraphs . People can identify the supporting factors in the paragraph and give a detailed explanation of the judgment result . We argue that the more specific the semantic interpretation , the more helpful the model imitates the human reasoning process . The input of the most recent model is Pre-trained embeddings , which have the advantage of capturing semantic similarity , but it is hard to explain in detail . We believe that the model uses interpretable features for reasoning , which contributes to enhanced interpretability . Recently , the attention mechanism has achieved remarkable performance on many natural language processing tasks . The model of attention mechanism learns the relevance between words through training ( Figure 2 ) . Inspired by the attention mechanism , for establishing rich and interpretable semantic features , we propose the semantic relational table ( Figure 3 ) . Semantic role labeling ( SRL ) is a shallow semantic parsing task aiming to discover who did what to whom , when and why ( He et al. , 2018 ; Li et al. , 2018 ) , providing explicit contextual semantics , which naturally matches the task target of text comprehension and It is easy to explain for people . Recently , Ribeiro et al . ( 2020 ) show that although measuring held-out accuracy has been the primary approach to evaluate generalization , it often overestimates the performance of NLP models . Moreover , the model does not seem to resolve basic Coreferences and grasp simple subject/object or active/passive distinctions ( SRL ) , all of which are critical to comprehension . Zhang et al . ( 2018 ) regard the semantic signals as SRL embeddings and employ a lookup table to map each label to vectors , similar to the implementation of word embedding . For each word , a joint embedding is obtained by the concatenation of word embedding and SRL embedding . Extensive experiments on benchmark machine reading comprehension and inference datasets verify that the proposed semantic learning helps for the model . The previous work indicates that SRL may hopefully help to understand contextual semantics relation . Formal semantics generally presents the semantic relation as ” predicate argument ” structure . For example , given the following sentence with target verb ( predicate ) sold , all the arguments are labeled as follows , [ ARG0 : Tom ] [ V : sold ] [ ARG1 : a book ] [ ARG2 : to jerry ] [ ARGM-TMP : last week ] . Where ARG0 represents the seller ( agent ) , ARG1 represents the thing sold ( theme ) , ARG2 represents the buyer ( recipient ) , ARGM-TMP is an adjunct indicating the timing of the action , and V represents the predicate . A question sentence example : [ ARG0 : Who ] [ V : bought ] [ ARG1 : flowers ] [ ARG2 : for jerry ] [ ARGM-LOC : in the park ] [ ARGM-TMP : last week ] . In the reference text , the sentences highly related to the semantic roles of the question are essential reasoning information . In this paper , we use the name of entities to integrate into the semantic role relation table to establish the semantic relation between sentences . Since many features are difficult to process efficiently , we simplified the semantic role relation table between two sentences . In particular , the contributions of this work are : ( 1 ) We believe that the model uses interpretable features for reasoning , which contributes to enhanced interpretability . So we propose an interpretable form of semantic relations to enhance the interpretability of the model ’ s input data . We use the enti- ty ’ s name to integrate into the semantic role relation table to establish the semantic relation between question , article , and the target sentence . ( 2 ) With few training data sets , the model ’ s performance based on the semantic role relation table is still stable . 2 METHOD . 2.1 A SIMPLIFIED EXAMPLE . For each sentence , we extract predicate-argument tuples via SRL toolkits . We employ the BERTbased model ( Shi & Lin , 2019 ) in the AllenNLP toolkit to perform SRL . We arrange the different semantic roles labels in a fixed order . A simplified example of supporting sentence prediction task : • question : who eats bread at night ? • sent1 : Tom eats bread . • sent2 : Jerry eats an apple . • sent3 : Tom eats something at night . • sent4 : Jerry drinks milk . We need to recognize sent1 and sent3 as the supporting facts . First step : we use semantic role label tools to parse sentences , set the position that the semantic role label contained in the sentence to 1 , and set the position that the semantic role label is not contained in the sentence to 0 . Second step : We match the entities between the two sentences , the position of the semantic role tag corresponding to the same named entities set to 1 , and the matching result is regarded as the semantic relation between the sentences . The two steps can build the following semantic role relation tables : The ” q ” stands for ” question ” , the ” a ” stands for ” article ” , the ” t ” stands for ” table ” . The question feature table ( Figure 4 ) shows the distribution of the semantic structure information of the question sentence . ” Who ” does not appear in the figure due to it is regarded as a stop word . The semantic role label feature of the question sentence is an essential clue for judging whether the sentence is a piece of evidence . The sent1 is the target sentence for which the model makes evidence prediction . sent1 feature table ( Figure 5 ) , show the distribution of the semantic structure information of the sentence . The article feature table ( Figure 6 ) show the distribution of the global semantic structure information of the article . The question feature sent1 table ( Figure 7 ) and The sent1 feature question table ( Figure 8 ) denotes the semantic relationship between the problem and the target sentence . The sent1 feature article table ( Figure 9 ) and article feature sent1 table ( Figure 10 ) denotes the semantic relationship between the article and the target sentence . The question feature article table ( Figure 11 ) and article feature question table ( Figure 12 ) denotes the semantic relationship between the article and the question . 2.2 MODEL . The model architecture , shown in ( Figure13 ) , the input data of the model is semantic role relation tables ( Figure 3∼11 ) , each figure undergoes the same convolution operation . The k is the set of types of semantic role tags . Let i k , the i-th represents a specific semantic role tag in a sentence . Let the v ( padded where necessary ) means the maximum number of verbs in a sentence . Let the s ( padded where necessary ) means the maximum number of sentences in the article . A convolution operation involves a filter w1 Rvs , which is applied to a window of each semantic role relation table to produce a new feature . ci = f ( w1 · xi : i+1 + b ) ( 1 ) For example , a feature ci is generated from a window of a table . Here b R is a bias term , and f is a non-linear function such as the hyperbolic tangent . This filter is applied to each possible semantic role relation table window to produce a feature map . c1 = [ c1 , c2 , · · · , c25 ] ( 2 ) Another convolution operation involves a filter w2 Rkvs , which is applied to a window of each semantic role relation table to produce a global feature . c2 = f ( w2 · x1 :25 + b ) ( 3 ) g = c1 ⊕ c2 ( 4 ) Where ⊕ is the concatenation operator to contact local features with global features . g = g1 ⊕ g2 · · · ⊕g9 ( 5 ) These features extracted from each semantic role relation table are spliced . Then These features are passed to a fully connected softmax layer whose output is the probability distribution over labels . 3 EXPERIMENT . 3.1 RESULT ANALYSIS . We conduct experiments on the HotpotQA data set . HotpotQA encourages explainable QA models by providing supporting sentences for the answer , which usually come from several documents ( a document is called ” gold doc ” if it contains supporting facts ) . Each case includes ten paragraphs , two of which are ” gold doc. ” We only use ” gold doc ” as an article . Since the test set of HotpotQA is not publicly available , our evaluations are based on the dev set . We use exact match ( EM ) , F1 , Accuracy as three evaluation metrics . The number of sentences and the number of verbs in sentences affects the semantic role feature map size . Therefore , we conducted experiments on the data of the maximum number of sentences ( 5∼10 ) in an article and the maximum number of verbs ( 3∼4 ) in a sentence on the proposed model , and the experimental results are shown in Table 1 . The ( article sent num ) stands for the maximum number of sentences in an article . The ( verb num ) stands for the maximum number of verbs in a sentence The ( case num ) stands for the number of all the articles in the data sets that satisfy the constraint condition . The ( sent num ) stands for the number of all the sentences in the data sets that satisfy the constraint condition . From ( table 1 ) , as the number of sentences in articles increases , EM decreases , F1 decreases , and Accuracy does not change much . As the number of verbs in the sentence increases , EM , F1 , and Accuracy do not change much . There are some experiment result in table 2 . From ( table 2 ) , within a certain range , EM does not Table 2 : Some experimental results in the number of sentences not more than 10 , the number of verbs not more than 4 . The case num = 5219 , the sent num = 30976. article sent num ( ≤ ) verb num ( ≤ ) Accuracy EM F1 10 4 0.7559 0.2433 0.7132 10 4 0.7621 0.2603 0.6939 10 4 0.7628 0.2638 0.6778 increase with the increase of F1 . Sometimes , the EM decreases with the increase of F1 . From ( table 3 ) , Although we did not use all the data in the dev set , we used most of the dev set . so experimental results can be compared to the Baseline . Our model performance should be close to the Baseline model . However , our model structure is simpler than Bert and Baseline . The Baseline model reimplemented the architecture described in Clark & Gardner ( 2018 ) The Baseline model subsumes the technical advances on question answering , including word embedding , character-level models , bi-attention Seo et al . ( 2017 ) and self-attention Wang et al . ( 2017 ) . Recently proposed models such as SAE Tu et al . ( 2020 ) include BERT module , so the performance is excellent . This shows that the semantic role relation table is a critical feature in machine reading comprehension .

Summary: The paper presents a model for the supporting sentence prediction task in machine reading comprehension. To enhance interpretability of the model, the paper proposes to use the semantic relations among a question, target sentence, and sentences in an article as features of the model. The semantic relations are obtained from semantic role labeling and named entities. The paper shows the prediction accuracy of the proposed model compared against some prior models, and conducts ablation experiments and case studies.