Interpretable Semantic Role Relation Table for Supporting Facts Recognition of Reading Comprehension
Authors: YanQing Bai
Zhichang Zhang
HaoYuan Chen
Xiaohui Qin
Yanglong Qiu
Venue: ICLR 2022 Submitted (2022)
Keywords: interpretability
semantic role relation table
supporting facts

Abstract: The current Machine Reading Comprehension (MRC) model has poor interpretability. 
Interpretable semantic features can enhancethe interpretability of the model. 
Semantic role labeling (SRL) captures predicate-argument relations, such as "who did what to whom," which
are critical to comprehension and interpretation. 
To enhance the interpretability of the model, we propose the semantic role relation table,
which represents the semantic relation of the sentence itself and the semantic relations among sentences. 
We use the name of entities to integrate into the semantic role relation table to establish the semantic relation between sentences. 
This paper makes the first attempt to utilize contextual semantic's explicit relation to the recognition supporting fact of reading 
comprehension. 
We have established nine semantic relationtables between target sentence, question, and article. 
Then we take each semantic relationship table's overall semantic role relevance and each
semantic role relevance as important judgment information. 
Detailed experiments on HotpotQA, a challenging multi-hop MRC data set, our method achieves better performance. 
With few training data sets, the model performance is still stable.

Under review as a conference paper at ICLR 2022
FORMATTING INSTRUCTIONS FOR ICLR 2022 CONFERENCE SUBMISSIONS
Anonymous authors
Paper under double-blind review
ABSTRACT
The current Machine Reading Comprehension (MRC) model has poor inter- pretability. Interpretable semantic features can enhance the interpretability of the model. Semantic role labeling (SRI.) captures medicate-argumeen relations, such as "who did what to whom," which are critical to comprehension and interpreta- tion. To enhance the interpretability of the model, propose the semantic role relation table, which represents the semantic relation of the sentence itself and the semantic relations among sentences. We use the name of entities to integrate into the semantic role relation table to establish the semantic relation between sentences. This paper makes the first attempt utilize contextual semantic's ex- plicit relation to the recognition supporting fact of reading comprehension. We have established nine semantic relation tables between target sentence, question, and article. Then we take each semantic relationship table's overall semantic role relevance and each semantic role relevance as important judgment information. Detailed experiments on HotpotQA, a challenging multi-hop MRC data set, our method achieves better performance. With few training data sets, the model per- formance is still stable.
INTRODUCTION
There has been increasing interest in the explainability of Machine Reading Comprehension (MRC) in recent years. For enhancing the explainability in MRC, some rescarchers (Qiu et al. 2019 Tu et al. 2020 Fang et al. 2020) utilize Graph Networks. The relational inductive bias encoded in Graph Networks (Battaglia et al., 2018) provides viable support for reasoning and learning over structured representations. Some researchers (Feng et al.] (2020) utilize explicit inference chains for multi-hop reasoning. Yangg et al. (2018) provide sentence-leve supporting facts required for reasoning, allowing QA systems reason with strong supervision and explain the predictions. This paper focuses on establishing an interpretable model sentence-level supporting facts recognition. system capable of delivering explanations generally more interpretable, meeting some of the requirements for real-world applications, such as user trust, acceptance, and confidence (Thayaparan al. 2020).
Paragh A "Adriana Trigiani"
[]]Adriana Trigiani is Italion American best-selling author of sixteen Books, television writer, film director, and entreprenzur hased in Greenwich Village, New York City.
12]Tria published novel a year 2000.
Parapgh B: "Big Stone Gap (film)"
[3] Stone Gap is 2014 American drama romantic comedy film written and directed by Adriana Trigiani
and produced by Donna Giglioti for Altar Identity Studios, a subsidiary of Media Society.
[4]Based on Trigiani's 7000 best-selling nevel same name, the story set in the actual Virginia town
Big Stone Gupicirca 1970s.
[S]ehe film had its world premiere at the Virginia Film Fesival on November 6, 2014.
Parapgh C....
question "The director of the romantic comedy "Big Stone Gap" is based in what New Yark city?"
answer : "Greenwich Village, New York City"
supporting facts: [1][3]
Figure 1: An example of the multi-hop questions in HOTPOTQA. The the supporting facts in blue. The red area is an entity that is highly relevant to the question.

---

Under review as a conference paper at ICLR 2022
An example from HotpotQA (Yang et al., 2018) is illustrated in Figure]] To correctly answer the question ("The director of the romantic comedy Big Stone Gap is based in what New York City") the model is required to first identify ParagraphB as a relevant paragraph, whose title contains the keywords that appear in the question ("Big Stone Gap"). S3, the first sentence of ParagraphB, is then chosen by the model from ParagraphA, the span Greenwich Village, New York City is selected as the predicted answer. In this example, sl and s3 contain the critical information needed reason the answer. When we judge whether S3 is supporting fact, our model needs to understand the semantic relation between the S3,question and the paragraphs.
supporting fact that leads the next-hop paragraph ParagraphA. Lastly,
51
S1
$2 Tom
who
eat bread at night
S2
ARGO V
ARG1
TMP
ARGO
eat bread
V
in the mommng
ARG1
TMP
Figure 2: example of attention mechanism Figure 3: example of semantic relational table
People identify the supporting factors in the paragraph and give detailed explanation of the judgment result. We argue that the more specific the semantic interpretation, the more helpful the model imitates the human reasoning process. The input of the most recent model is Pre-trained embeddings, which have the advantage of capturing semantic similarity, but it is hard to explain in detail. We believe that the model uses interpretable features for reasoning, which contributes to enhanced interpretability. Recently, the attention mechanism has achieved remarkable performance on many natural language processing tasks. The model of attention mechanism learns the relevance between words through training(Figure 2 Inspired by the attention mechanism, for establishing rich and interpretable semantic features, we propose the semantic relational table(Figure)) 3
Semantic role labeling (SRL) is shallow semantic parsing task aiming to discover who did what to whom, when and Why et al., 2018 Li et al. 2018). providing explicit contextual semantics, which naturally matches the task target of text comprehension and It is easy to explain for people.
Recently, Ribeiro al. (2020) show that although measuring held-out accuracy has been the pri- mary approach to evaluate generalization, it often overestimates the performance of NLP models. Moreover, the model does not seem to resolve basic Coreferences and grasp simple subject/object active/passive distinctions(SRL), all of which are critical to comprehension. Zhang et al. (2018) regard the semantic signals as SRL embeddings and employ a lookup table to map each label to vectors, similar to the implementation of word embedding. For each word, a joint embedding is obtained by the concatenation of word embedding and SRL embedding. Extensive experiments on benchmark machine reading comprehension and inference datasets verify that the proposed se- mantic learning helps for the model. The previous work indicates that SRL may hopefully help to understand contextual semantics relation.
Formal semantics generally presents the semantic relation as "predicate argument" structure. For example, given the following sentence with target verb (predicate) sold, all the arguments are la- beled as follows, [ARGO: Tom] [V: sold] [ARGI: a book] [ARG2: to jerry] [ARGM-TMP: last week]. Where ARGO represents the seller (agent), ARGI represents the thing sold (theme), ARG2 represents the buyer (recipient), ARGM-TMP is an adjunct indicating the timing of the action, and represents the predicate.
A question sentence example: [ARGO: Who] [V: bought] [ARG1: flowers] [ARG2: for jerry] [ARGM-LLO:in the park] [ARGM-TMP: last week]. In the reference text, the sentences highly related to the semantic roles of the question essential reasoning information. In this paper, use the name of entities to integrate into the semantic role relation table to establish the semantic relation between sentences. Since many features are difficult to process efficiently, simplified the semantic role relation table between two sentences.
In particular, the contributions of this work are: (1)We believe that the model uses interpretable fea- tures for reasoning, which contributes to enhanced interpretability. So we propose an interpretable form of semantic relations enhance the interpretability of the model's input data. We use the enti-
2

---

Under review as a conference paper at ICLR 2022
ty's name to integrate into the semantic role relation table to establish the semantic relation between question, article, and the target sentence. (2)With few training data sets, the model's performance based on the semantic role relation table is still stable.
METHOD
2.1 A SIMPLIFIED EXAMPLE
For each sentence, we extract predicate-argumeet tuples via SRL toolkits. We employ the BERT- based model (Shi & Lin 2019) in the AllenNLP toolkit to perform SRL. We arrange the different semantic roles labels in a fixed order. A simplified example of supporting sentence prediction task:
question: who eats bread at night?
sentl: Tom eats bread.
sent2: Jerry eats an apple.
sent3: Tom eats something night.
sent4: Jerry drinks milk.
We need recognize sentl and sent3 as the supporting facts. First step: we use semantic role label tools to parse sentences, set the position that the semantic role label contained in the sentence to 1, and set the position that the semantic role label not contained in the sentence to O. Second step: We match the entities between the two sentences, the position of the semantic role tag corresponding to the same named entities set to 1, and the matching result regarded as the semantic relation between the sentences. The two steps build the following semantic role relation tables: The "q" stands for "question" the "a" stands for "article", the "t" stands for "table".
eat
bread
night
Tom
eat
bread
ARGO
ARGI
TMP
fromure
ARGO
ARG1
TMP
ARGO V
ARGI
TMP
sext fisturs
question
Figure 4: f_featurett
Figure 5:
Figure 6: a.feature.t
The question_feature.. treature_tblle (Figure shows the distribution of the semantic structure information of the question sentence. "Who" does not appear in the figure due to it is regarded as a stop word. The semantic role label feature of the question sentence is essential clue for judging whether the sentence is a piece of evidence.
The sentl is the target sentence for which the model makes evidence prediction. sent lffattrreeeeecc _table (Figure 5). show the distribution of the semantic structure information of the sentence. The arti- cle feature table(Figure 6) show the distribution of the global semantic structure information of the article.
eat
bread
night
Tom
eat
bread
Tom
cul
bread
ARGO
V
ARGI
TMP
ARGO
ARGI
TMP
wenet,
sent
1
0
question
0
sonn,
Figure 7: f_eeature.sent_.tt
Figure 8: sent .feature..
Figure 9: .entturcatueeaat
The question feature sentl table (Figure 7 and The sentl feature question table (Figure g denotes the semantic relationship between the problem and the target sentence.
3

---

Under review as a conference paper at ICLR 2022
ARGO
V
ARGI
TMP
cat
breud
night
ARGO
ARGI
TMP
1
ARGI
soy, Jaan _anor
0
sas;
Saakee
1
send,
0
faave
0
0
1
ast,
2012
Figure 10: a.feature.sent
Figure 11: q.feature_a.t
Figure 12: a_feature.qu
The sent wwwtlleeeaeeartaee wwtttrrttteeeteee www.arclle.aaeeeFiiee...e. table(Figuree 2 and fettcle.feature.eeme sentl Ltable(Figure 10) denotes the se- mantic relationship between the article and the target sentence.
The
I and the semantic relationship between the article and the question.
12
denotes
2.2
MODEL.
Verb-1
Verb-2
Verb-3
Figure 13: There 9 semantic role relation tables. this figure, use two tables of sentence size. Each sentence contains three verbs, and each verb has 25 corresponding semantic role tags. We do the same convolution operation on the tables of the same size.
The model architecture, shown in (Figure 13) ,the input data of the model is semantic role relation tables(Figure 3~11)), 11), each figure undergoes the same convolution operation. The k is the set of types of semantic role tags. Let ick; the i-th represents specific semantic role tag in sentence. Let the v(padded where necessary) means the maximum number of verbs in sentence. Let the s(padded where necessary) means maximum number of sentences the article.
A convolution operation involves filter R which is applied to window of each semantic role relation table to produce a new feature.
C; (w) Tmits +b)
For example, a feature is generated from window of table. Here be R is bias term, and is non-linear function such as the hyperbolic tangent. This filter is applied to each possible semantic role relation table window to produce feature map.
C1 = [cc C2, C25]
Another convolution operation involves filter war Rkue, which is applied to a window of each semantic role relation table to produce global feature.
C2 f (w2 1:255 + 8)
(3)
g = C1 ① C2
Where @ is the concatenation operator contact local features with global features.
3g:
(5)
These features extracted from each semantic role relation table are spliced. Then These features are passed to fully connected softmax layer whose output is the probability distribution over labels.
4

---

Under review as a conference paper at ICLR 2022
3 EXPERIMENT
3.1 RESULT ANALYSIS
article sent num! Table 1:
Experiments with different maximum numbers of verbs in sentence, different maximum
numbers of sentences in a article.
verb num S
case nmm
sent mum
Accuracy
EM
F1
3
]692
6848
0.7627
0.3965
0.7944
4
2279
9408
0.7600
0.3865
0.7825
4
3262
15296
0.7533
0.3258
0.7617
4
4075
20392
0.748
0.2743
0.7364
3225
4
4673
25792
0.7527
0.2673
0.7271
3
4
3024
28928
0.7555
0.2555
0.7156
0.7615
0.274
10
5219
30976
0.7539
0.2433
0.7132
We conduct experiments the HotpotQA data set. HotpotQA encourages explainable QA models by providing supporting sentences for the answer, which usually come from several documents (a paragraphs,
document is called "gold doc" if it contains supporting facts). Each case includes two of which are "gold doe." We only use "gold doc" as an article. Since the test set of HotpotQA not publicly available, our evaluations are based on the dev set. We use exact match (EM), FI, Accuracy as three evaluation metrics.
The number of sentences and the number of verbs in sentences affects the semantic role feature map size. Therefore, we conducted experiments on the data of the maximum number of sentences (5~10) an article and the maximum number of verbs (3~44 in a sentence on the proposed model, and the experimental results are shown in Table The (article aerricle_eent.une sent_num) stands for the maximum number of sentences an article. The (verb num) stands for the maximum number verbs sentence The (case_num) stands for the number of all the articles in the data sets that satisfy the constraint condition. The (sent_num) stands for the number all the sentences in the data sets that satisfy the constraint condition.
From (table [. as the number of sentences in articles increases, EM decreases, Fl decreases, and Accuracy does not change much. As the number of verbs in the sentence increases, EM, F1, and Accuracy do not change much.
There are experiment result in table 2 From (table 2). within a certain range, EM does not
Table 2:
Some experimental results in the number of sentences not more than 10, the number of verbs not more than 4. The case_numt 5219, the sent_num 30976.
article sent num(s)
verb num(s) <
Accuracy
EM
F1
10
4
0.7559
0.2433
0.7132
10
4
0.7621
0.2603
0.6939
10
0.7628
0.2638
0.6778
increase with the increase of F1. Sometimes, the EM decreases with the increase of F1.
Table 3:
the result with Baseline and Bert.
Model
case num
Accuracy
EM
F1
SAE-large 7405
0,6330
0.8738
Bert 5219
0.8753
0,5214
0.8434
Baseline 7405
0.2195
0.6666
Our model 5219
0.7559
0.2433
0.7132
From (table 3). Although did not all the data in the dev set, used most of the dev set. so experimental results can be compared to the Baseline. Our model performance should be close to the Baseline model. However, model structure is simpler than Bert and Baseline. The Baseline
5

---

Under review as a conference paper at ICLR 2022
model reimplemented the architecture described in Clark & Gardner (2018) The Baseline model subsumes the technical advances on question answering, including word embedding, character- models, bi-attention Seo et al. (2017) and self-attention [Wang al. (2017p. Recently proposed models such as SAE TTu et al. (020)) include BERT module, so the performance is excellent, This shows that the semantic role relation table critical feature in machine reading comprehension.
3.2 LOW-RESOURCE EXPERIMENT
Table The result in dev The case_numm 5219, the sent.num 30976.
article sent num(E) verb num(
train case num
Accuracy
EM
FI
10
250)
0.6686
0.133
0.6682
10
500
0.7251
0.1941
0.6949
10
2500
0.7567
0.2381
0.6884
5000
0.7592
0.2507
0.6726
50000
0.7542
0.2476
0.7119
We do experiment the training set and dev set(the number of sentences not than 10, the number verbs not more than 4). From (table (4) when the training set is lower than 2500 (5%), the performance(EM, Accuracy, and f1 is lower than that of the model using all training set. When the training set in 2500~~000 (5%~~10%)tthe performance(EM, Accuracy) is close to the model using all training set. This shows that the semantic role relation table convenient feature, and the model's performance based on the semantic role relation table is stable.
3.3 ABLATION STUDIES
We do ablation experiments on the dev set(The number of sentences (s) not more than 10, the number of verbs (v) not more than 4. And The number of sentences (s) equal 5,the number of verbs (v) equal 3). Local(L): The model only uses the features, which are produced by filter wi w
Table 5: Ablation experiment
model
case num
Accuracy
EM
F1
$ 55v=3(LGG)
461
0.7382
0.2689
0.7275
3=
461
0.7335
0.2472
0.7037
5=5
461
0.7339
0.2516
0.7201
$ 10,v5 4(I+G)
5219
0.7559
0.2433
0.7132
$5 10,v5 4(G)
5219
0.7592
0,2496
0.7012
2x 10,v5 4(L)
5219
0.7613
0,2525
0.6835
GGobal(G): The model only uses the features, which are produced by the filter Wze WoeRkus
From (table when the training set is small, the performanceeAccuurayyy EM, F1) of the model(L) significantly higher than the model(G). Moreover, the Copormanne(Accuraag) EM, F1) of the model(L++G) is higher than the model(L.) the model(G). When the training and the dev set increase, the Cerformance(Accuraa or the model(G). This result shows that our model may not fully use local features, which are critical for EM.
EM of the model(L+G) may be lower than than the model(L)
3.4
CASE STUDIES
We do case studies experiments on the dev set(The number of sentences (s) not more than 5, the number of verbs (v) not more than 3. (G,L) is the same as (G,L. in ablation studies. A:all fine- grained features of semantic relational tables are input into the fully connected neural network. epoch:refers to the number of training. From (figure 14p. the model(G+L) works best, and it is the model chosen for this paper. Model(G) and mL are similar in performance. Model(A) contain too
has poor performance, many fine- grainer features, it is difficult to use effectively, and the performance lower than the model((.) and mo)e
fluctuation range. Because The

---

Under review as a conference paper at ICLR 2022
0.74
0.72
acultt
G+L
L4A
0.66
epoch
Figure 14: case studies
RELATED WORK
4.1 SUPPORTING FACT
Supporting factors is useful and becomes an important component in multiple-choice reading com- prehension (Wang et al., (2019), natural language inference (Chen et al. (017)) open-domain ques- tion answering (Lin et al. 2018). Following HotpotQA, several benchmarks on open-domain tasks have gradually refined the supporting facts annotation, whose benefits have been demonstrated in
terms of interpretability, bias, and performance (Dua et 2020 Inoue et al. 2020
4.2 EXPLAINABLE MODEL
The study in Zhou et al. (2018) proposed an Interpretable Reasoning Network for QA on a knowl- edge base. The baseline model provided in the HotpotQA paper (Yang et 2018) and the QFE model Computational Linguistics (Nishida et al. (2019) are based on a single document MRC system proposed in Clark & Gardner (2018), with interpretable answer prediction. However, multi-hop rea- soning was not explicitly dealt with in this work. Recent studies on multi-hop QA also build graphs based on entities and reasoning over the constructed graph using graph neural networks. DFGN Qiu et al. (2019) considered the model explainability by locating supporting entities and then leading to support sentences. The Hicrarchical Graph Network (Fang et al. 2020) leverages a hicrarchical graph representation of the background knowledge (question, paragraphs, sentences, and entities). SAE et al. (2020) defines three edge types in the sentence graph based on the named entities and noun phrases appearing in the question and sentences. The Semantic table incorporates the semantic features of the entity graph (Battaglia et al., 2018)that uses sentence vectors as nodes and edges connecting sentences that share the same named entities and reflect the semantic role relation.
CONCLUSION AND FUTURE WORK
In this paper, we propose the semantic role relation table form to enhance the interpretability of the processing process. We use the name of entities to integrate into the semantic role relation table. The model based on semantic relational table achieves good performance in the experiments despite using simple neural network structure. The model's performance based the semantic role relation table is still stable with few training data sets. Unlike most recent works focusing on heuristically stacking complex mechanisms for performance improvement, this work is to shed some light on how even after fusing different kinds of semantic signals, the representation of semantic relations maintains good interpretability.
There are at least four potential future directions. First, The name of entities contains a few se- mantic features. However, the pre-trained embeddings have the advantage of capturing semantic similarity. Explore how to integrate the semantic signals of Pre-trained embeddings in sematic role relation table to enhance the performance further. Second, according to our experimental re- sults, to make full use of the information of the semantic relational table, it is necessary to propose a more effective neural network model. Third, SRL contains the shallow semantic information of all the words in sentence. Therefore, it is convenient to integrate the finer-grained interpretable

---

Under review as a conference paper at ICLR 2022
semantic features(eg:the patt-of-peeech(POOS)) of the text corresponding to SRL into the semantic relational table, to further enhance the interpretability of semantic relational tables. Forth, in some specific situations, the pronunciation of words and the image information related to words are also essential features.
REFERENCES
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinícius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. CoRR, 0tss/806.011200 2018.
URL http::ttt//xxi//e org/ abs/ www//8/6..... 111//8866000 01261
Yoshua Bengio and Yann LeCun. Scaling learning algorithms towards AI. In Large Scale Kernel Machines. MIT Press, 2007.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced LSTM for natural language inference. In Regina Barzilay and Min-Yer Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACI. 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pp. 1657-1668, Association for Computa- tional Linguistics, 2017. doi: 10.1-653/11/11--115222 URL
http:aoi..ggggg// /t. 18653
v1/P17-1152
Christopher Clark and Matt Gardner. Simple and effective multi-paragraph reading comprehen- sion. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 845-855, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10.18653/v1/P18- 1078. (tR.:ttss:/ wwwwww/www.../////////////./// acl.eb.
org/
Dheeru Dua, Sameer Singh, and Matt Gardner. Benefits of intermediate annotations in reading comprehension. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5627-5634, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18663/1/2202..almmimml
URL httts:://wwwt ac 1 web www.eee.rrr//anooooo////0000
acl mhain.997
Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. Hierarchical graph network for multi-hop question answering. Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pp. 8823-8838, Online, November 2020. Association for Computational Linguistics, doi: 10. 11553/11/2020..cooo - main.710. URL https: wwtp///wwww../. acw.eaclleeee www.:ebbbrggge.. www.oo/ggganootgggoeooeeoo
emn main. main.710
Yufei Feng, Mo Yu, Wenhan Xiong, Xiaoxiao Guo, Junjie Huang, Shiyu Chang, Murray Campbell, Michael A. Greenspan, and Xiaodan Zhu. Learning to recover reasoning chains for multi-hop
question answering via cooperative games. CoRR, abs/2004.02393, 2020. URL https://
arxiv. org/ abtp///s..000.... 02393
Shexia He, Zuchao Li, Hai Zhao, and Hongxiao Bai. Syntax for semantic role labeling, to be, not to be. In Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 2061-2071, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 1...1853/11/P1-1111 www. ac www.wwwww.bb........ www.aa..web.org/mtttooooooo wtt.oo///mmooooogg//1-111113
URL https:
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18: 18:1527-1554, 2006.
Naoya Inoue, Pontus Stenetorp, and Kentaro Inui. R4C: A benchmark for evaluating RC sys- tems to get the right answer for the right reason. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pp. 6740-6755 6750, Online, July 2020. As- sociation for Computational Linguistics. doi: 10. 10.11863/vvll 1/201//22222 acl-2a0.clmmmi..66 602. URL hetps: 7/www. a web. www..//b....//atatoooooo////0000 a ma n.. 602
8

---

Under review as a conference paper at ICLR 2022
Zuchao Li, Shexia He, Jiaxun Cai, Zhuosheng Zhang, Hai Zhao, Gongshen Liu, Linlin Li, and Luo Si. A unified syntax awnte aware framework for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2401-2411, Brussels, Belgium, Octoher-November 2018. Association for Computational Linguistics. doi: 10.18653/ v1/D18-120 12622. URL URI.https. www. ac web . wtw.bb.rggg/toooooggooooooooo//e
1262
Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. Denoising distantly supervised open-
domain question answering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (
(Volume Long Papers), pp. 1736-1745, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.186533v1/P18..1 1161.
URL https:
7/ //www. www.waclweb..o orge httgg//athooooyyP1e.1 1161
Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata, Atsushi Otsuka, Itsumi Saito. Hisako Asano, and Junji Tomita. Answering while summarizing: Multi-task learning for multi-hop QA with evidence extraction. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pp. 2335-2345, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10.18653/11//9-11 P19-1225
utthttps:/ htop:://wwww..
Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, and Yong Yu. Dy- namically fused graph network for multi-hop reasoning. In Proceedings of the 57th Annu- al Meeting of the Association for Computational Linguistics, pp. (1500 Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10. 10.18653/1//199-117,, URL https:/ httpp::/wwww...//m wttp://wwwwww....c/// wwwpee...gg..aooooooo///1------.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy:
Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pp. 4902-4912, Online, July 2020. As- sociation for Computational Linguisties. doi: http://athoooooyooooaoe acl -a-maain 442)
URL https:
7/ 7/www.
Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URLLhttps hwt./openvvviiieee
//tpttss://peeerriiee.... www. wtte://oorrmmmmoo-o0.00 9ge
Peng Shi and Jimmy Lin. Simple BERT models for relation extraction and semantic role labeling. CoRR, abs/1994.00255, 2019. URL wrt. 1904. 05255
Mokanarangan Thayaparan, Marco Valentino, and André Freitas. survey on explainability in machine reading comprehension. CoRR, abs/2010.00330 2020. URL htthttss://axi...coo
abs/2010. 00389
Ming Tu, Kevin Huang, Guangtao Wang, Jing Huang. Xiaodong He, and Bowen Zhou. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents. In The Thirty-Fourthy AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty- Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Sym- posium on Educational Advances in Artificial Intelligence EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 9073 9080. AAAI Press, 2020. URL https: http::///aaaa.rr////....
index. php/
6441
Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, Dong Yu, David McAllester, and Dan Roth. Evidence sentence extraction for machine reading comprehension. In Proceedings of the 23rd Confer- ence on Computational Natural Language Learning (CoNLL), pp. 696-707, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/K19-1100 1065. URL https: htt:::///wwww... ac web. wwtt/att/oooooooooooooo ogyp 11019--1 065)
Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 189-198, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 10.18663/v 1/P 1018. URL https: / www. , a www.wrr..wae....... www.orrg/nttholleeeeeeemo hgy

---

Under review as a conference paper at ICLR 2022
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question an- swering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2369-2380, Brussels, Belgium, October-November 2018. Association for Com- putational Linguistics. doi: 10.18653/1//D18. 1/591-12999
URL https: / www. ac web. org.
anthology/DI8 1259.
Zhuosheng Zhang, Yuwei Wu, Zuchao Li, and Hai Zhao. Explicit contextuul semantics for text com- prehension. CoRR, abs/1809 (111809000299 02794, 2018. URL http:://aaax....rrr//////0110..
Mantong Zhou, Minlie Huang, and Xiaoyan Zhu. An interpretable reasoning network for multi- relation question answering. In Proceedings of the 27th International Conference on Computa- tional Linguistics, pp. 2010-2022, Santa Fe, New Mexico, USA, August 2018. Association for Computational Linguistics. Lingussts URL [RIhhttps: //wp://wwww.w////// ac web. wwt.://wwww......aaaaaoooooolaaa
A
APPENDIX
You may include other additional sections here.
10


Reviews:
[{"review_id": "mNUtj_XeGw", "reviewer": "Reviewer_GpZ5", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Strengths\n\n* The table-based featurization is interesting. It seems like a potentially nice way of characterizing propositional overlap, where different sentences have semantic substructures in common.\n\n* It's nice that ablations were done and a study of how the model performs with less supervision\n\n## Weaknesses\n\nUnfortunately, I think this paper is a bit underdeveloped and there are several issues with the presentation.\n\n* First, the model is not fully described. The paper gives a \"simplified example\" but never elaborates on it to describe how the full model works. I can't figure out, for example, how it is determined whether two arguments in different sentences should be considered to overlap — in the example they share the word \"Tom\", but how is it done? Just by token identity? If so, isn't this pretty problematic?\n\n* Second, not enough background information is given about the task, dataset, and I/O spec of the model. Does this only apply to extractive QA? How are the metrics on HotpotQA calculated? Why are some of them missing? (Accuracy for SAE-large and Baseline in Table 3)\n\n* Third, I think there are a lot of other issues which make the paper overall hard to understand.\n  * Table 1 is an overload of numbers. Fewer significant figures are needed for the metrics and the trends in the table should be conveyed in a graph (or graphs).\n  * snake_case variable names and other abbreviations are used throughout the paper where simple language would have sufficed and been easier to understand. It feels almost like directly describing the code — I think it's best to keep the reader isolated from unnecessary details. I have a hard time understanding Sections 3.3 and 3.4 for this reason. It's hard for me to remember what $w_2$ refers to, and I actually still don't intuitively understand it from its original description in Sec. 2.2.\n\n* Fourth, I don't see what is interpretable or explainable about this model. While it is featurized in a way that depends on SRL, explainability of the model decisions would require a usable theory of how these features relate to the final decisions.\n\nFinally, in light of all of these issues I don't know what to make of the results. They are apparently better than a different pre-BERT baseline but I don't understand the model description well enough to know what's better; and, it's not clear what insights we might take away from this work that could be applied to future models. Overall I think this paper presents an intriguing and promising approach, but needs a lot of organizational and presentation work (plus maybe more experimental work) before it is ready for scholarly publication.\n\n## Minor comments\n\n* I think I get what Figure 2 is getting at, but I don't think it effectively illustrates what an attention mechanism is.\n\n* I do not understand how the conclusion at the end of Sec. 3.1 was drawn: it says \"this shows that semantic role relation table is a critical feature in machine reading comprehension\" just after describing how the baseline models perform well (Table 3).\n\n* The low-resource study in Sec. 3.2 seems to me to show that performance levels off early, meaning semantic role relation tables are limited in what they can express alone.\n\n* It seems to me that the binary featurization is basically just constructing a graph of nodes (for the tokens) and edges (for SRL relations and identity of roles and tokens). Isn't doing convolutions over this table something like just using a GCN? In that case, has this not been done before?", "summary_of_the_paper": "This work proposes a model architecture for reading comprehension based on a large number of binary features corresponding to propositional overlap (in terms of SRL and, as far as I can tell, token surface forms or lemmas) between sentences in a passage and a reading comprehension question. These features are arranged into tables which are passed into a few convolutional filters to yield representations that are passed into a final classification layer.", "summary_of_the_review": "Intriguing idea, but needs a lot of organizational work, clarity/presentation improvements, etc., and it's unclear what we learn from the results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LCglX6h_UJH", "reviewer": "Reviewer_uT9c", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n- Clear description of background knowledge, and clear exposition of the proposed model with examples.\n- The authors perform an evaluation on  Question Answering and an ablation study.\n- The findings show that the proposed model has comparable performance with related work.\n\nWeaknesses\n- It is not clearly discussed the evaluation of the model interpretability.\n- It is not clear how the selection of hyper-parameters would affect the baseline compared to the addition of the semantic features.  \n\nQuestions to the Authors. Please address the following questions during the rebuttal:\n\n- Which evaluation can be used to highlight the model decisions given the input semantic features? Elaborate on the interpretability of the proposed features. \n- Elaborate on the dependency of the proposed model on linguistic annotation. How does a SRL parser affect performance?\n- Could you elaborate on the baseline set-up and the selection of the hyper-parameters? \n- How sensitive the model is to a given set of hyper-parameters? \n- How multiple  random runs could affect the final results?\n- How the semantic features could be used on other downstream tasks? such as, Summarization or language inference.\n-  The authors could observe more benefits with  using a graph neural network model than the current baseline. For example, De Cao et al. Question Answering by Reasoning Across Documents with Graph Convolutional Networks.\n\n Extra: please double check for typos and the title of the paper.\n", "summary_of_the_paper": "The authors propose a method for adding interpretable semantic features for a Machine Reading Comprehension model  based on semantic relations across words and sentences.  The goal is to use semantic role labeling annotation to produce features used on a neural model for a language downstream task.  The main contributions are:  semantic features based on linguistic annotation, and interpretable semantic features for training a neural model. The study shows that the semantic features used to train Question Answering model performs comparable to related work.\n", "summary_of_the_review": "The authors proposed the addition of linguistic annotation into a neural model for Question Answering. The paper clearly describes related work, and proposed model. The proposed model incorporates interpretability, however, it relies on manual or automatic semantic annotation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EVzqH4ayCKx", "reviewer": "Reviewer_dcWb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Strengths: I think using semantic roles to enhance interpretability of a model is a good idea, since semantic roles provide important relation information.\n\n- Weaknesses:\n1) The writing needs to be improved a lot. Currently, it is hard to understand many parts and justifications are often missing. \n2) The paper claims that using SRL can improve interpretability of a model. However, there is no supporting experiment showing that.\n\n- Detailed feedback\n1) I realized this work addresses the supporting sentence prediction task only after reading the beginning of 2.1. Before that, it was not clear to me, whether this work was proposing a module for interpretation that can be plugged in MRC models, proposing an MRC model that strengthens interpretability, or something else.\n2) abstract: \"better performance\" than what?\n3) Figure 1: \"Paragh\"? \"Parapgh\"? I guess they are typos of \"paragraph\".\n4) Introduction: It is not clear to me how attention mechanisms are relevant to this work, even after reading the whole paper\n5) Below Figure 2: references are missing for \"the recent model\", \"We believe that the model ...\"?\nsynonyms? \n6) I see lots of typos. e.g., \"and It is easy\", wrong styles of citation,.. \n7) \"the semantic signals\"? What are they?\n8) Why did you show the experiment with using smaller number of training examples? Does this say that the proposed model uses the semantic role relation even for improving the accuracy in small datasets?\n9) \"We arrange the different semantic roles labels in a fixed order\" <= not sure what this means.\n10) \"set the position ... to 1\" \"set the position .. to 0\" <= these descriptions are confusing. I think the authors set the value at a position in a table/matrix to 0/1.\n11) \"Who\" does not appear in the figure due to it is regarded as a stop word? <= I think it should not appear in the table because it is what the question asks? I understand using stop words could be a heuristic way of getting it, though.\n12)  the caption of Figure 13 didn't make sense to me. \n13) \"types\"? What types\" examples would be helpful.\n14) typos: \"let v means\" \"let s means\" => \"mean\"\n15) need to specify what x_i is.\n16) global? local? how do you define them?\n17) Why is the different maximum number of verbs in a sentence important? The paper says because it affects the feature map size. Why is it important?\n18) How is \"Exact Match\" computed? How is it different from accuracy? \n19) \"target sentence\" is not defined early enough. \n20) what kind of classifier is used?\n21) descriptions of baselines are too late.\n22) \"Although we did not use all the data in the dev set, we used most of the dev set. so\nexperimental results can be compared to the Baseline\" <= this doesn't make sense to me. How can you compare the results from different algorithms when the data are different.\n23) The last sentence before 3.2. What is this? The previous sentence talks about related work.\n24) Why did you do the low-resource experiment?\n25) Did you use data that only satisfy the condition of the maximum # of sentences and # of verbs? What is the average number of sentences and verbs in a sentence? Also, what happens if an article or a sentence includes more sentences or verbs? \n26) I didn't understand what the case studies do and why. \n27) How would synonyms be treated when matching the SRL tables among a question, target sentence, and articles? \n", "summary_of_the_paper": "The paper presents a model for the supporting sentence prediction task in machine reading comprehension. To enhance interpretability of the model, the paper proposes to use the semantic relations among a question, target sentence, and sentences in an article as features of the model. The semantic relations are obtained from semantic role labeling and named entities. The paper shows the prediction accuracy of the proposed model compared against some prior models, and conducts ablation experiments and case studies. ", "summary_of_the_review": "I think the paper has too many unclear parts. In addition, the paper proposes using semantic role labeling to enhance interpretability, but I don't see how SRL enhances interpretability in the paper -- no experiments or studies about that. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6O4txYUBGps", "reviewer": "Reviewer_aAdr", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The hypothesis that performing some kind of inference based on matching\n  predicate argument structures of supporting sentences and question is intuitively appealing.\n\n- The limited experiments presented seem to indicate that the method has\n  some potential.\n\n\nWeaknesses:\n\n- The experiments are very preliminary, as they compare systems on different\n  subsets of examples, use a baseline which is not clear whether it is\n  comparable head-to-head to the proposed system, and the target system\n  performs well below competing systems.\n\n- The paper should be much more clear in the description of the motivating\n  example and the method. In the current writing it cannot be understood\n  in its entirety. There are many details missing and the explanations are\n  very convoluted and vague.\n\n- In order to show that the proposed technique captures information which is\n  complementary to current state-of-the-art systems, the authors should try\n  to incorporate their method in one of those systems.\n\nIn addition, the authors should study and review the latest related work,\nclearly write the problem setting, model description, experimental setup and\nresults, as well as give an analysis of the results to provide an\nunderstanding of why it is working.\n", "summary_of_the_paper": "The paper proposes to use the predicate argument structure of sentences in\norder to improve the results of a question answering system on the HotpotQA\ndataset, where the answer depends on a number of supporting sentences.\n\nThe main contribution is a method encode whether the fillers of predicate arguments match\nacross supporting sentences and the question. \n\n", "summary_of_the_review": "\nThe lack of clarity and the weak empirical evidence are well below ICLR\nstandards, and thus I recommend a strong rejection.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that this paper does not meet the bar for ICLR. The reviewers provide detailed feedback to the authors on how to improve the writing as well as the overall content of the paper.
Decision: Reject