Year: 2021

Abstract:
The current Machine Reading Comprehension (MRC) model has poor interpretability. Interpretable semantic features can enhance the interpretability of the model. Semantic role labeling (SRL) captures predicate-argument relations, such as ”who did what to whom,” which are critical to comprehension and interpretation. To enhance the interpretability of the model, we propose the semantic role relation table, which represents the semantic relation of the sentence itself and the semantic relations among sentences. We use the name of entities to integrate into the semantic role relation table to establish the semantic relation between sentences. This paper makes the first attempt to utilize contextual semantic’s explicit relation to the recognition supporting fact of reading comprehension. We have established nine semantic relation tables between target sentence, question, and article. Then we take each semantic relationship table’s overall semantic role relevance and each semantic role relevance as important judgment information. Detailed experiments on HotpotQA, a challenging multi-hop MRC data set, our method achieves better performance. With few training data sets, the model performance is still stable.

heading: 1 INTRODUCTION
text: There has been an increasing interest in the explainability of Machine Reading Comprehension (MRC) in recent years. For enhancing the explainability in MRC, some researchers (Qiu et al., 2019; Tu et al., 2020; Fang et al., 2020) utilize Graph Networks. The relational inductive bias encoded in Graph Networks (Battaglia et al., 2018) provides viable support for reasoning and learning over structured representations. Some researchers (Feng et al., 2020) utilize explicit inference chains for multi-hop reasoning. Yang et al. (2018) provide sentence-level supporting facts required for reasoning, allowing QA systems to reason with strong supervision and explain the predictions. This paper focuses on establishing an interpretable model sentence-level supporting facts recognition. A system capable of delivering explanations is generally more interpretable, meeting some of the requirements for real-world applications, such as user trust, acceptance, and confidence (Thayaparan et al., 2020).
An example from HotpotQA (Yang et al., 2018) is illustrated in Figure1. To correctly answer the question (”The director of the romantic comedy Big Stone Gap is based in what New York City”) the model is required to first identify ParagraphB as a relevant paragraph, whose title contains the keywords that appear in the question (”Big Stone Gap”). S3, the first sentence of ParagraphB, is then chosen by the model as a supporting fact that leads to the next-hop paragraph ParagraphA. Lastly, from ParagraphA, the span Greenwich Village, New York City is selected as the predicted answer. In this example, s1 and s3 contain the critical information needed to reason the answer. When we judge whether S3 is a supporting fact, our model needs to understand the semantic relation between the S3,question and the paragraphs.
People can identify the supporting factors in the paragraph and give a detailed explanation of the judgment result. We argue that the more specific the semantic interpretation, the more helpful the model imitates the human reasoning process. The input of the most recent model is Pre-trained embeddings, which have the advantage of capturing semantic similarity, but it is hard to explain in detail. We believe that the model uses interpretable features for reasoning, which contributes to enhanced interpretability. Recently, the attention mechanism has achieved remarkable performance on many natural language processing tasks. The model of attention mechanism learns the relevance between words through training(Figure 2). Inspired by the attention mechanism, for establishing rich and interpretable semantic features, we propose the semantic relational table(Figure 3).
Semantic role labeling (SRL) is a shallow semantic parsing task aiming to discover who did what to whom, when and why(He et al., 2018; Li et al., 2018), providing explicit contextual semantics, which naturally matches the task target of text comprehension and It is easy to explain for people.
Recently, Ribeiro et al. (2020) show that although measuring held-out accuracy has been the primary approach to evaluate generalization, it often overestimates the performance of NLP models. Moreover, the model does not seem to resolve basic Coreferences and grasp simple subject/object or active/passive distinctions(SRL), all of which are critical to comprehension. Zhang et al. (2018) regard the semantic signals as SRL embeddings and employ a lookup table to map each label to vectors, similar to the implementation of word embedding. For each word, a joint embedding is obtained by the concatenation of word embedding and SRL embedding. Extensive experiments on benchmark machine reading comprehension and inference datasets verify that the proposed semantic learning helps for the model. The previous work indicates that SRL may hopefully help to understand contextual semantics relation.
Formal semantics generally presents the semantic relation as ”predicate argument” structure. For example, given the following sentence with target verb (predicate) sold, all the arguments are labeled as follows, [ARG0: Tom] [V: sold] [ARG1: a book] [ARG2: to jerry] [ARGM-TMP: last week]. Where ARG0 represents the seller (agent), ARG1 represents the thing sold (theme), ARG2 represents the buyer (recipient), ARGM-TMP is an adjunct indicating the timing of the action, and V represents the predicate.
A question sentence example: [ARG0: Who] [V: bought] [ARG1: flowers] [ARG2: for jerry] [ARGM-LOC:in the park] [ARGM-TMP: last week]. In the reference text, the sentences highly related to the semantic roles of the question are essential reasoning information. In this paper, we use the name of entities to integrate into the semantic role relation table to establish the semantic relation between sentences. Since many features are difficult to process efficiently, we simplified the semantic role relation table between two sentences.
In particular, the contributions of this work are: (1)We believe that the model uses interpretable features for reasoning, which contributes to enhanced interpretability. So we propose an interpretable form of semantic relations to enhance the interpretability of the model’s input data. We use the enti-
ty’s name to integrate into the semantic role relation table to establish the semantic relation between question, article, and the target sentence. (2)With few training data sets, the model’s performance based on the semantic role relation table is still stable.

heading: 2 METHOD

heading: 2.1 A SIMPLIFIED EXAMPLE
text: For each sentence, we extract predicate-argument tuples via SRL toolkits. We employ the BERTbased model (Shi & Lin, 2019) in the AllenNLP toolkit to perform SRL. We arrange the different semantic roles labels in a fixed order. A simplified example of supporting sentence prediction task:
• question: who eats bread at night? • sent1: Tom eats bread. • sent2: Jerry eats an apple. • sent3: Tom eats something at night. • sent4: Jerry drinks milk.
We need to recognize sent1 and sent3 as the supporting facts. First step: we use semantic role label tools to parse sentences, set the position that the semantic role label contained in the sentence to 1, and set the position that the semantic role label is not contained in the sentence to 0. Second step: We match the entities between the two sentences, the position of the semantic role tag corresponding to the same named entities set to 1, and the matching result is regarded as the semantic relation between the sentences. The two steps can build the following semantic role relation tables: The ”q” stands for ”question”,the ”a” stands for ”article”,the ”t” stands for ”table”.
The question feature table (Figure 4) shows the distribution of the semantic structure information of the question sentence. ”Who” does not appear in the figure due to it is regarded as a stop word. The semantic role label feature of the question sentence is an essential clue for judging whether the sentence is a piece of evidence.
The sent1 is the target sentence for which the model makes evidence prediction. sent1 feature table (Figure 5), show the distribution of the semantic structure information of the sentence. The article feature table(Figure 6) show the distribution of the global semantic structure information of the article.
The question feature sent1 table (Figure 7) and The sent1 feature question table (Figure 8) denotes the semantic relationship between the problem and the target sentence.
The sent1 feature article table(Figure 9) and article feature sent1 table(Figure 10) denotes the semantic relationship between the article and the target sentence.
The question feature article table(Figure 11) and article feature question table(Figure 12) denotes the semantic relationship between the article and the question.

heading: 2.2 MODEL
text: The model architecture, shown in (Figure13) ,the input data of the model is semantic role relation tables(Figure 3∼11), each figure undergoes the same convolution operation. The k is the set of types of semantic role tags. Let i k, the i-th represents a specific semantic role tag in a sentence. Let the v(padded where necessary) means the maximum number of verbs in a sentence. Let the s(padded where necessary) means the maximum number of sentences in the article.
A convolution operation involves a filter w1 Rvs, which is applied to a window of each semantic role relation table to produce a new feature.
ci = f (w1 · xi:i+1 + b) (1)
For example, a feature ci is generated from a window of a table. Here b R is a bias term, and f is a non-linear function such as the hyperbolic tangent. This filter is applied to each possible semantic role relation table window to produce a feature map.
c1 = [c1, c2, · · ·, c25] (2)
Another convolution operation involves a filter w2 Rkvs, which is applied to a window of each semantic role relation table to produce a global feature.
c2 = f (w2 · x1 :25 + b) (3)
g = c1 ⊕ c2 (4) Where ⊕ is the concatenation operator to contact local features with global features.
g = g1 ⊕ g2 · · · ⊕g9 (5)
These features extracted from each semantic role relation table are spliced. Then These features are passed to a fully connected softmax layer whose output is the probability distribution over labels.

heading: 3 EXPERIMENT

heading: 3.1 RESULT ANALYSIS
text: We conduct experiments on the HotpotQA data set. HotpotQA encourages explainable QA models by providing supporting sentences for the answer, which usually come from several documents (a document is called ”gold doc” if it contains supporting facts). Each case includes ten paragraphs, two of which are ”gold doc.” We only use ”gold doc” as an article. Since the test set of HotpotQA is not publicly available, our evaluations are based on the dev set. We use exact match (EM), F1, Accuracy as three evaluation metrics.
The number of sentences and the number of verbs in sentences affects the semantic role feature map size. Therefore, we conducted experiments on the data of the maximum number of sentences (5∼10) in an article and the maximum number of verbs (3∼4) in a sentence on the proposed model, and the experimental results are shown in Table 1. The (article sent num) stands for the maximum number of sentences in an article. The (verb num) stands for the maximum number of verbs in a sentence The (case num) stands for the number of all the articles in the data sets that satisfy the constraint condition. The (sent num) stands for the number of all the sentences in the data sets that satisfy the constraint condition.
From (table 1), as the number of sentences in articles increases, EM decreases, F1 decreases, and Accuracy does not change much. As the number of verbs in the sentence increases, EM, F1, and Accuracy do not change much.
There are some experiment result in table 2. From (table 2), within a certain range, EM does not
Table 2: Some experimental results in the number of sentences not more than 10, the number of verbs not more than 4. The case num = 5219, the sent num = 30976.
article sent num(≤) verb num(≤) Accuracy EM F1 10 4 0.7559 0.2433 0.7132 10 4 0.7621 0.2603 0.6939 10 4 0.7628 0.2638 0.6778
increase with the increase of F1. Sometimes, the EM decreases with the increase of F1.
From (table 3), Although we did not use all the data in the dev set, we used most of the dev set. so experimental results can be compared to the Baseline. Our model performance should be close to the Baseline model. However, our model structure is simpler than Bert and Baseline. The Baseline
model reimplemented the architecture described in Clark & Gardner (2018) The Baseline model subsumes the technical advances on question answering, including word embedding, character-level models, bi-attention Seo et al. (2017) and self-attention Wang et al. (2017). Recently proposed models such as SAE Tu et al. (2020) include BERT module, so the performance is excellent. This shows that the semantic role relation table is a critical feature in machine reading comprehension.

heading: 3.2 LOW-RESOURCE EXPERIMENT
text: Table 4: The result in dev set. The case num = 5219, the sent num = 30976.
article sent num(≤) verb num(≤) train case num Accuracy EM F1
10 4 250 0.6686 0.133 0.6682 10 4 500 0.7251 0.1941 0.6949 10 4 2500 0.7567 0.2381 0.6884 10 4 5000 0.7592 0.2507 0.6726 10 4 50000 0.7542 0.2476 0.7119
We do experiment on the training set and dev set(the number of sentences not more than 10, the number of verbs not more than 4). From (table 4), when the training set is lower than 2500 (5%), the performance(EM, Accuracy, and f1 ) is lower than that of the model using all training set. When the training set in 2500∼5000 (5%∼10%),the performance(EM, Accuracy) is close to the model using all training set. This shows that the semantic role relation table is a convenient feature, and the model’s performance based on the semantic role relation table is stable.

heading: 3.3 ABLATION STUDIES
text: We do ablation experiments on the dev set(The number of sentences (s) not more than 10, the number of verbs (v) not more than 4. And The number of sentences (s) equal 5,the number of verbs (v) equal 3). Local(L): The model only uses the features, which are produced by the filter w1 Rvs.
Global(G): The model only uses the features, which are produced by the filter w2 Rkvs.
From (table 5), when the training set is small, the performance(Accuracy, EM, F1) of the model(L) is significantly higher than the model(G). Moreover, the performance(Accuracy, EM, F1) of the model(L+G) is higher than the model(L) or the model(G). When the training set and the dev set increase, the performance(Accuracy, EM ) of the model(L+G) may be lower than than the model(L) or the model(G). This result shows that our model may not fully use local features, which are critical for EM.

heading: 3.4 CASE STUDIES
text: We do case studies experiments on the dev set(The number of sentences (s) not more than 5, the number of verbs (v) not more than 3. (G,L) is the same as (G,L) in ablation studies. A:all finegrained features of semantic relational tables are input into the fully connected neural network. epoch:refers to the number of training. From (figure 14), the model(G+L) works best, and it is the model chosen for this paper. Model(G) and model(L) are similar in performance. Model(A) has poor performance,large fluctuation range. Because The model(A+G,A+L,A+G+L) contain too many fine-grained features, it is difficult to use effectively, and the performance lower than the model(L) and model(G).

heading: 4 RELATED WORK

heading: 4.1 SUPPORTING FACT
text: Supporting factors is useful and becomes an important component in multiple-choice reading comprehension (Wang et al., 2019), natural language inference (Chen et al., 2017), open-domain question answering (Lin et al., 2018). Following HotpotQA , several benchmarks on open-domain tasks have gradually refined the supporting facts annotation, whose benefits have been demonstrated in terms of interpretability, bias, and performance (Dua et al., 2020; Inoue et al., 2020).

heading: 4.2 EXPLAINABLE MODEL
text: The study in Zhou et al. (2018) proposed an Interpretable Reasoning Network for QA on a knowledge base. The baseline model provided in the HotpotQA paper (Yang et al., 2018) and the QFE model Computational Linguistics (Nishida et al., 2019) are based on a single document MRC system proposed in Clark & Gardner (2018), with interpretable answer prediction. However, multi-hop reasoning was not explicitly dealt with in this work. Recent studies on multi-hop QA also build graphs based on entities and reasoning over the constructed graph using graph neural networks. DFGN Qiu et al. (2019) considered the model explainability by locating supporting entities and then leading to support sentences. The Hierarchical Graph Network (Fang et al., 2020) leverages a hierarchical graph representation of the background knowledge (question, paragraphs, sentences, and entities). SAE Tu et al. (2020) defines three edge types in the sentence graph based on the named entities and noun phrases appearing in the question and sentences. The Semantic table incorporates the semantic features of the entity graph (Battaglia et al., 2018)that uses sentence vectors as nodes and edges connecting sentences that share the same named entities and reflect the semantic role relation.
CONCLUSION AND FUTURE WORK
In this paper, we propose the semantic role relation table form to enhance the interpretability of the processing process. We use the name of entities to integrate into the semantic role relation table. The model based on a semantic relational table achieves good performance in the experiments despite using a simple neural network structure. The model’s performance based on the semantic role relation table is still stable with few training data sets. Unlike most recent works focusing on heuristically stacking complex mechanisms for performance improvement, this work is to shed some light on how even after fusing different kinds of semantic signals, the representation of semantic relations maintains good interpretability.
There are at least four potential future directions. First, The name of entities contains a few semantic features. However, the pre-trained embeddings have the advantage of capturing semantic similarity. Explore how to integrate the semantic signals of Pre-trained embeddings in a sematic role relation table to enhance the performance further. Second, according to our experimental results, to make full use of the information of the semantic relational table, it is necessary to propose a more effective neural network model. Third, SRL contains the shallow semantic information of all the words in a sentence. Therefore, it is convenient to integrate the finer-grained interpretable
semantic features(eg:the part-of-speech(POS)) of the text corresponding to SRL into the semantic relational table, to further enhance the interpretability of semantic relational tables. Forth, in some specific situations, the pronunciation of words and the image information related to words are also essential features.

heading: A APPENDIX
text: You may include other additional sections here.

Review summaries:
The paper presents a model for the supporting sentence prediction task in machine reading comprehension. To enhance interpretability of the model, the paper proposes to use the semantic relations among a question, target sentence, and sentences in an article as features of the model. The semantic relations are obtained from semantic role labeling and named entities. The paper shows the prediction accuracy of the proposed model compared against some prior models, and conducts ablation experiments and case studies. 

The authors propose a method for adding interpretable semantic features for a Machine Reading Comprehension model  based on semantic relations across words and sentences.  The goal is to use semantic role labeling annotation to produce features used on a neural model for a language downstream task.  The main contributions are:  semantic features based on linguistic annotation, and interpretable semantic features for training a neural model. The study shows that the semantic features used to train Question Answering model performs comparable to related work.