Multi-Perspective Fusion Network for Commonsense Reading Comprehension

Abstract: Commonsense Reading Comprehension (CRC) is a significantly challenging task, aiming at choosing the right answer for the question referring to a narrative passage, which may require commonsense knowledge inference. Most of the existing approaches only fuse the interaction information of choice, passage, and question in a simple combination manner from a \emph{union} perspective, which lacks the comparison information on a deeper level. Instead, we propose a Multi-Perspective Fusion Network (MPFN), extending the single fusion method with multiple perspectives by introducing the \emph{difference} and \emph{similarity} fusion\deleted{along with the \emph{union}}. More comprehensive and accurate information can be captured through the three types of fusion. We design several groups of experiments on MCScript dataset \cite{Ostermann:LREC18:MCScript} to evaluate the effectiveness of the three types of fusion respectively. From the experimental results, we can conclude that the difference fusion is comparable with union fusion, and the similarity fusion needs to be activated by the union fusion. The experimental result also shows that our MPFN model achieves the state-of-the-art with an accuracy of 83.52\% on the official test set.

Content: Task Definition
1. Describe the task of commonsense reading comprehension(CRC) belongs to which filed and how important it is.
2. Define the task of CRC
3. Data feature of CRC
4. Figure 1 shows an example.
Machine Reading Comprehension (MRC) is an extremely challenging topic in natural language processing field. It requires a system to answer the question referring to a given passage.no matter whether the answer is mentioned in the passage. MRC consists of several sub-tasks, such as cloze-style reading comprehension, span-extraction reading comprehension, and open-domain reading comprehension. Most of existing datasets emphasize the question whose answer is mentioned in the passage since it does not need any commonsense. In real reading comprehension, the human reader can fully understand the passage with the prior knowledge to answer the question. To directly relate commonsense knowledge to reading comprehension, SemEval2018 Task 11 defines a new sub-task called Commonsense Reading Comprehension, aiming at answering the questions that requires both commonsense knowledge and the understanding of the passage. The challenge of this task is how tolies in answer questions requires a system to draw inferences from multiple sentences from the passage and requireswith the commonsense knowledge that does not appear in the passage explicitly. Table 1 shows an example of CRC.
Content: Previous Research
1. Category the methods in SemEval2018 task 11
2. Describe the first method
3. Describe the second method
4. State that your work is belong to which method
Most studies on CRC task are neural network based (NN-based) models, which typically have the following characteristics. Firstly, word representations are augmented by additional lexical information. , such as pre-trained embedding, POS and NER embedding, Relation embedding and some other handcraft features. Secondly, the interaction process is usually implemented by the attention mechanism, which can provide the interaction representations like choice-aware passage, choice-aware question, and question-aware passage. Thirdly, the original representations and interaction representations are fused together and then aggregated by a Bidirectional Long Short-Term Memory Network (BiLSTM) BIBREF1 to get high-order semantic information. Fourthly, the final output based on their bilinear interactions. is the sum scores of passage to choice and question to choice.
The NN-based models have shown powerfulness on this task. However, there are still some limitations. Firstly, the two fusion processes of passage and question to choice are implemented separately, until producing the final output. Secondly, the existing fusion method used in reading comprehension task is usually implemented by concatenation BIBREF2 , BIBREF3 , which is monotonous and cannot capture the partial comparison information between two parts. Studies on Natural Language Inference (NLI) have explored more functions BIBREF4 , BIBREF5 , such as element-wise subtraction and element-wise multiplication, to capture more comparison information, which have been proved to be effective.
In this paper, we introduce a Muti-Perspective Fusion Network (MPFN) to tackle these limitations. The model can fuse the choice with passage and question simultaneously to get a multi-perspective fusion representation. Furthermore, inspired by the element-wise subtraction and element-wise multiplication function used in BIBREF5 , we define three kinds of fusion functions from multiple perspectives to fuse choice, choice-aware passage, and choice-aware question. The three fusions are union fusion, difference fusion, and similarity fusion. Note that, we name the concatenation fusion method as union fusion in this paper, which collects the global information. The difference fusion and the similarity fusion can discover the different parts and similar parts among choice, choice-aware passage, and choice-aware question respectively.
MPFN comprises an encoding layer, a context fusion layer, and an output layer. In the encoding layer, we employ a BiLSTM as the encoder to obtain context representations. to convert the embeddings of passage, question, and choice to their corresponding context embeddings. To acquire better semantic representations, we apply union fusion in the word level. to choice, choice-aware passage embedding, and choice-aware question embedding. In the context fusion layer, we apply union fusion, difference fusion, and similarity fusion to obtain a multi-perspective fusion representation. In the output layer, a self-attention and a feed-forward neural network are used to make the final prediction.
We conduct experiments on MRScript dataset released by BIBREF0 . Our single and ensemble model achieve the accuracy of 83.52% and 84.84% on the official test set respectively. Our main contributions are as follows:
We propose a general fusion framework with two-layer fusion, which can fuse the passage, question, and choice simultaneously.
To collect multi-perspective fusion representations, we define three types of fusions, consisting of union fusion, difference fusion, and similarity fusion.
We extend the fusion method to multi-perspective to obtain deeper understanding of the passage, question, and choice.
We design several groups of experiments to evaluate the effectiveness of the three types of fusion and prove that our MPFN model outperforms all the other models. with an accuracy of 83.52%.
MRC has gained significant popularity over the past few years. Several datasets have been constructed for testing the comprehension ability of a system, such as MCTest BIBREF6 , SQuAD BIBREF7 , BAbI BIBREF8 , TriviaQA BIBREF9 , RACE BIBREF10 , and NewsQA BIBREF11 . The types of passage, question and answer of these datasets are various. Each dataset focuses on one specific aspect of reading comprehension. Particularly, the MCScript BIBREF0 dataset concerns answering the question which requires using commonsense knowledge.
including Wikipedia articles, examinations, narrative stories, news articles. Answering questions in these datasets. Meanwhile, the question types and answer types vary differently. The answer type multiple choice, span-answer, exact match
Many architectures on MRC follow the process of representation, attention, fusion, and aggregation BIBREF12 , BIBREF2 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 . BiDAF BIBREF12 fuses the passage-aware question, the question-aware passage, and the original passage in context layer by concatenation, and then uses a BiLSTM for aggregation. The fusion levels in current advanced models are categorized into three types by BIBREF14 , including word-level fusion, high-level fusion, and self-boosted fusion. They further propose a FusionNet to fuse the attention information from bottom to top to obtain a fully-aware representation for answer span prediction.
 BIBREF16 present a DFN model to fuse the passage, question, and choice by dynamically determine the attention strategy.
On SemEval2018 Task 11, most of the models use the attention mechanism to build interactions among the passage, the question, and the choice BIBREF17 , BIBREF3 , BIBREF18 , BIBREF19 . The most competitive models are BIBREF17 , BIBREF3 , and both of them employ concatenation fusion to integrate the information. BIBREF17 utilizes choice-aware passage and choice-aware question to fuse the choice in word level. In addition, they apply the question-aware passage to fuse the passage in context level. Different from BIBREF17 , both the choice-aware passage and choice-aware question are fused into choice in the context level in BIBREF3 , which is the current state-of-the-art result on the MCSript dataset.
On NLI task, fusing the premise-aware hypothesis into the hypothesis is an effective and commonly-used method. BIBREF20 , BIBREF21 leverage the concatenation of the hypothesis and the hypothesis-aware premise to help improve the performance of their model. The element-wise subtraction and element-wise multiplication between the hypothesis and the hypothesis-aware premise are employed in BIBREF5 to enhance the concatenation. and further achieved the state-of-the-art results on Stanford Natural Language Inference BIBREF22 benchmark.
Almost all the models on CRC only use the union fusion. In our MPFN model, we design another two fusion methods to extend the perspective of fusion. We evaluate the MPFN model on MRC task and achieve the state-of-the-art result.
The overview of our Multi-Perspective Fusion Network (MPFN) is shown in Fig. 1 . Given a narrative passage about a series of daily activities and several corresponding questions, a system requires to select a correct choice from two options for each question. In this paper, we denote $\bf {p=\lbrace p_1,p_2,...,p_{|p|}\rbrace }$ as the passage, $\bf {q=\lbrace q_1,q_2,...,q_{|q|}\rbrace }$ as a question, $\bf {c=\lbrace c_1,c_2,...,c_{|c|}\rbrace }$ as one of the candidate choice, and a true label $y^{*} \in \lbrace 0,1\rbrace $ . Our model aims to compute a probability for each choice and take the one with higher probability as the prediction label. Our model consists of three layers: an encoding layer, a context fusion layer, and an output layer. The details of each layer are described in the following subsections.
This layer aims to encode the passage embedding $p$ , the question embedding $q$ , and the choice embedding $c$ into context embeddings. Specially, we use a one-layer BiLSTM as the context encoder. 
$$&\bar{c}_i = \text{BiLSTM}(c, i) , & i \in [1,2, \cdots ,|c|] \\
&\bar{p}_j = \text{BiLSTM}(p, j) , & j \in [1,2, \cdots ,|p|]  \\
&\bar{q}_k = \text{BiLSTM}(q, k) , & k \in [1,2, \cdots ,|q|] $$   (Eq. 18) 
The embeddings of $p$ , $q$ and $c$ are semantically rich word representations consisting of several kinds of embeddings. Specifically, the embeddings of passage and question are the concatenation of the Golve word embedding, POS embedding, NER embedding, Relation embedding and Term Frequency feature. And the embeddings of choice comprise the Golve word embedding, the choice-aware passage embedding, $c^p$ and choice-aware question embedding $c^q$ . The details about each embedding are follows:
Glove word embedding We use the 300-dimensional Glove word embeddings trained from 840B Web crawl data BIBREF23 . The out-of-vocabulary words are initialized randomly. The embedding matrix are fixed during training.
POS&NER embedding We leverage the Part-of-Speech (POS) embeddings and Named-Entity Recognition(NER) embeddings. The two embeddings $c_i^{pos} \text{and} c_i^{ner}$ are randomly initialized to 12d and 8d respectively, and updated during training.
Relation embedding Relations are extracted form ConceptNet. For each word in the choice, if it satisfies any relation with another word in the passage or the question, the corresponding relation will be taken out. If the relations between two words are multiple, we just randomly choose one. The relation embeddings $c_i^{rel}$ are generated in the similar way of POS embeddings. randomly initialized and updated during training as well.
Term Frequency Following BIBREF17 , we introduce the term frequency feature to enrich the embedding of each word. The calculation is based on English Wikipedia.
Choice-aware passage embedding The information in the passage that is relevant to the choice can help encode the choice BIBREF24 . To acquire the choice-aware passage embedding $c_i^p$ , we utilize dot product between non-linear mappings of word embeddings to compute the attention scores for the passage BIBREF25 . 
$$& c_i^p = Attn(c_i,\lbrace p_j\rbrace _1^{|p|}) = \sum _{j=1}^{|p|} {\alpha }_{ij} p_j \\
& {\alpha }_{ij} \propto exp(S(c_i, p_j)), \quad S(c_i, p_j) = {ReLU(W{c_i})}^{T} ReLU(W {p_j})$$   (Eq. 19) 
Choice-aware question embedding The choice relevant question information is also important for the choice. Therefore, we adopt the similar attention way as above to get the choice-aware question embedding $c_i^q=Attn(c_i, \lbrace q_k\rbrace _{1}^{|q|})$ .
The embeddings delivered to the BiLSTM are the concatenation the above components, where $p_j = [p_j^{glove}, p_j^{pos},p_j^{ner},p_j^{rel}, p_j^{tf} ]$ , $c_i = [c_i^{glove}, c_i^{p},c_i^{q}]$ , and $q_k = [q_k^{glove}, q_k^{pos}, q_k^{ner}, q_k^{rel},q_k^{tf} ]$ .
This is the core layer of our MPFN model. To take the union, different and similar information of the choice, passage, and question into consideration, three fusion functions are defined in this layer. In this layer, we define three fusion functions, which consider the union information, the different information, and the similar information of the choice, passage, and question.
Since we have obtained the choice context $\bar{c}_i$ , the passage context $\bar{p}_j$ , and the question context $\bar{q}_k$ in the encoding layer, we can calculate the choice-aware passage contexts $\tilde{c}^p_i$ and choice-aware question contexts $\tilde{c}^q_i$ . Then we deliver them together with the choice contexts $\bar{c}_i$ to the three fusion functions.
In this layer, we define three fusion functions to fuse the $\bar{c}_i$ , $\tilde{c}^p_j$ , and $\bar{c}^q_k$ simultaneously and multi-perspectively. The three fusion functions take the union information, the different information, and the similar information of the choice, passage, and question into consideration. To better integrate this information, we feed the three fusion outputs to FNN for aggregation.
Choice-aware passage context In this part, we calculate the choice-aware passage representations $\tilde{c}_i^p= \sum _{j}{\beta }_{ij} \bar{p}_j$ . For model simplification, here we use dot product between choice contexts and passage contexts to compute the attention scores ${\beta }_{ij}$ : 
$$&{\beta }_{ij}= \frac{exp({\bar{c}_i^T \bar{p}_j)}}{\sum \limits {_{j^\prime =1}^{|p|}exp(\bar{c}_i^T \bar{p}_{j^\prime })}}$$   (Eq. 21) 
Choice-aware question context In a similar way as above, we get the choice-aware question context $\tilde{c}_i^q= \sum _{j}{\beta }_{ik} \bar{q}_k$ . The ${\beta }_{ik}$ is the dot product of the choice context $\bar{c}_i$ and question context $\bar{q}_k$ .
Multi-perspective Fusion This is the key module in our MPFN model. The goal of this part is to produce multi-perspective fusion representation for the choice $\bar{c}_i$ , the choice-aware passage $\tilde{c}^p_i$ , and the choice-aware question $\tilde{c}^q_i$ . In this paper, we define fusion in three perspectives: union, difference, and similarity. Accordingly, we define three fusion functions to describe the three perspectives. The outputs and calculation of the three functions are as follows: : concatenation $;$ , element-wise dot product and element-wise subtraction. $f^u$ , $f^d$ , and $f^s$ All of the three fusion functions take the choice context, the choice-aware passage, and the choice-aware question as input. 
$$&u_i = [\bar{c}_i \, ; \tilde{c}_i^p \,; \tilde{c}^q_i] ,\\
&d_i = ( \bar{c}_i - \tilde{c}_i^p)\odot (\bar{c_i} - \tilde{c}_i^q) ,\\
&s_i = \bar{c}_i \odot \tilde{c}_i^p \odot \tilde{c}_i^q ,$$   (Eq. 22) 
 where $; \,$ , $-$ , and $\odot $ represent concatenation, element-wise subtraction, and element-wise multiplication respectively. And $u_i$ , $d_i$ , and $s_i$ are the representations from the union, difference and similarity perspective respectively.
The union perspective is commonly used in a large bulk of tasks BIBREF21 , BIBREF14 , BIBREF2 . It can see the whole picture of the passage, the question, and the choice by concatenating the $\tilde{c}^p_i$ and $\tilde{c}^q_i$ together with $c_i$ . While the difference perspective captures the different parts between choice and passage, and the difference parts between choice and question by $\bar{c_i} - \tilde{c}_i^p$ and $\bar{c_i} - \tilde{c}_i^q$ respectively. The $\odot $ in difference perspective can detect the two different parts at the same time and emphasize them. In addition, the similarity perspective is capable of discovering the similar parts among the passage, the question, and the choice.
To map the three fusion representations to lower and same dimension, we apply three different FNNs with the ReLU activation to $u_i$ , $d_i$ , and $s_i$ . The final output $g_i$ is the concatenation of the results of the three FNNs, which represents a global perspective representation. 
$$g_i=[f^u(u_i),f^d(d_i),f^s(s_i)] $$   (Eq. 23) 
 The output layer includes a self-attention layer and a prediction layer. Following BIBREF26 , we summarize the global perspective representation $\lbrace g_i\rbrace _1^{|c|}$ to a fixed length vector $r$ . We compute the $r= \sum _{i=1}^{|c|} b_i g_i$ , where $b_j$ is the self-weighted attention score : 
$$&b_i = \frac{exp(W{g}_i)}{\sum \limits {_{i^\prime =1}^{|c|}exp(W {g}_{i^\prime })}}$$   (Eq. 25) 
In the prediction layer, we utilize the output of self-attention $r$ to make the final prediction.
The final output y is obtained by transforming the $\mathbf {v}$ to a scalar and then apply a sigmoid activation to map it to a probability.
Data We conduct experiments on the MCScript BIBREF0 , which is used as the official dataset of SemEval2018 Task11. This dataset constructs a collection of text passages about daily life activities and a series of questions referring to each passage, and each question is equipped with two answer choices. The MCScript comprises 9731, 1411, and 2797 questions in training, development, and test set respectively. For data preprocessing, we use spaCy for sentence tokenization, Part-of-Speech tagging, and Name Entity Recognization. The relations between two words are generated by ConceptNet. The MCScript is a recently released dataset, which collects 2,119 narrative texts about daily events along with 13,939 questions. In this dataset, 27.4% questions require commonsense inference.
Parameters We use the standard cross-entropy function as the loss function. We choose Adam BIBREF27 with initial momentums for parameter optimization. As for hyper-parameters, we set the batch size as 32, the learning rate as 0.001, the dimension of BiLSTM and the hidden layer of FNN as 123. The embedding size of Glove, NER, POS, Relation are 300, 8, 12, 10 respectively. The dropout rate of the word embedding and BiLSTM output are 0.386 and 0.40 respectively.
Table 2 shows the results of our MPFN model along with the competitive models on the MCScript dataset. The TriAN achieves 81.94% in terms of test accuracy, which is the best result of the single model. The best performing ensemble result is 84.13%, provided by HMA, which is the voting results of 7 single systems.
Our single MPFN model achieves 83.52% in terms of accuracy, outperforming all the previous models. The model exceeds the HMA and TriAN by approximately 2.58% and 1.58% absolute respectively. Our ensemble model surpasses the current state-of-the-art model with an accuracy of 84.84%. We got the final ensemble result by voting on 4 single models. Every single model uses the same architecture but different parameters.
To study the effectiveness of each perspective, we conduct several experiments on the three single perspectives and their combination perspective. Table 3 presents their comparison results. The first group of models are based on the three single perspectives, and we can observe that the union perspective performs best compared with the difference and similarity perspective. Moreover, the union perspective achieves 82.73% in accuracy, exceeding the TriAN by 0.79% absolute. We can also see that the similarity perspective is inferior to the other two perspectives.
The second group of models in the Table 3 are formed from two perspectives. Compared with the single union perspective, combining the difference perspective with the union perspective can improve 0.11%. Composing union and similarity fusion together doesn't help the training. To our surprise, the combination of similarity perspective and difference perspective obtains 83.09% accuracy score.
The last model is our MPFN model, which performing best. The final result indicates that composing the union perspective, difference perspective, and similarity perspective together to train is helpful.
Many advanced models employ a BiLSTM to further aggregate the fusion results. To investigate whether a BiLSTM can assist the model, we apply another BiLSTM to the three fusion representations in Formula 23 respectively and then put them together. The results are shown in the second column in Table 3 , which indicate that the BiLSTM does not help improve the performance of the models.
In the section, we conduct ablation study on the encoding inputs to examine the effectiveness each component. The experiment results are listed in Table 3 . In Section "Encoding Layer" , we describe that our encoding inputs comprise six components: POS embedding, NER embedding, Relation embedding, Term Frequency, choice-aware passage embedding $C^p$ and choice-aware question embedding $C^q$ .
From the best model, if we remove the POS embedding and NER embedding, the accuracy drops by 0.82% and 0.9%. Without Relation embedding, the accuracy drops to 81.98%, revealing that the external relations are helpful to the context fusions. Without Term Frequency, the accuracy drops by approximately 1.61%. This behavior suggests that the Term Frequency feature has a powerful capability to guide the model.
After removing the $C^p$ , we find the performance degrades to 81.62%. This demonstrates that information in the passage is significantly important to final performance. If we remove $C^q$ from the MPFN, the accuracy drops to 82.16%. If we remove the word level fusion completely, we will obtain an 81.66% accuracy score. These results demonstrate that each component is indispensable and the bottom embeddings are the basic foundations of the top layer fusions.
In this section, we explore the influence of word-level interaction to each perspective. Fig 2 reports the overall results of how each perspective can be affected by the lower level interaction. The $C^p$ and the $C^q$ represent the choice-aware passage embedding and the choice-aware question embedding respectively. We can observe that the results of $[C;C^p]$ , $[C;C^q]$ , and $[C;C^p;C^q]$ are all higher than the result of $C$ alone, indicating the effectiveness of word embedding interaction.
Both the union fusion and difference fusion can achieve more than 80% accuracy, while the similarity fusion is very unstable. We also observe that the difference fusion is comparable with the union fusion, which even works better than the union fusion when the information of $C^p$ is not introduced into the input of encoding. The similarity fusion performs poorly in $C$ and $[C;C^q]$ , while yielding a huge increase in the remaining two groups of experiments, which is an interesting phenomenon. We infer that the similarity fusion needs to be activated by the union fusion.
In summary, we can conclude that integrate the information of $C^p$ into $C$ can greatly improve the performance of the model. Combining $C^q$ together with $C^p$ can further increase the accuracy. The information in the passage is richer than the question The overall conclusion
In this section, we visualize the union and difference fusion representations and show them in Fig 3 . And, we try to analyze their characteristics and compare them to discover some connections. The values of similarity fusion are too small to observe useful information intuitively, so we do not show it here. We use the example presented in Table 1 for visualization, where the question is Why didn't the child go to bed by themselves? and the corresponding True choice is The child wanted to continue playing.
The left region in Fig 3 is the union fusion. The most intuitive observation is that it captures comprehensive information. The values of child, wanted, playing are obvious higher than other words. This is consistent with our prior cognition, because the concatenation operation adopted in union fusion does not lose any content. While the difference union shows in the right region in Fig 3 focuses on some specific words. By further comparison, we find that the difference fusion can pay attention to the content ignored by the union fusion. What's more, the content acquired by the union would not be focused by the difference again. In other words, the union fusion and difference fusion indeed can emphasize information from the different perspective. Due to space limitation and
In this paper, we propose the Multi-Perspective Fusion Network (MPFN) for the Commonsense Reading Comprehension (CMC) task. We propose a more general framework for CRC by designing the difference and similarity fusion to assist the union fusion. Our MPFN model achieves an accuracy of 83.52% on MCScript, outperforming the previous models. The experimental results show that union fusion based on the choice-aware passage, the choice-aware question, and the choice can surpass the TriAN and HMA model. The difference fusion performs stably, which is comparable with the union fusion. We find that the word-level union fusion can significantly influence the context-level fusion. The choice-aware passage word embedding can activate the similarity fusion. We find that combining the similar parts and the difference parts together can obtain the best performance among the two-perspective models. By taking the three types of fusion methods into consideration, our MPFN model achieves a state-of-the-art result.
This work is funded by Beijing Advanced Innovation for Language Resources of BLCU, the Fundamental Research Funds for the Central Universities in BLCU (17PT05), the Natural Science Foundation of China (61300081), and the Graduate Innovation Fund of BLCU (No.18YCX010).

Figure and table captions:
Fig. 1: Architecture of our MPFN Model.
Table 2: Experimental Results of Models
Table 3: Test Accuracy of Multi-Perspective
Fig. 2: Influence of Word-level Interaction.
Fig. 3: Visualization of Fusions

Questions about this paper:
What baseline models do they compare against?

Answers:
SLQA, Rusalka, HMA Model (single), TriAN (single), jiangnan (ensemble), MITRE (ensemble), TriAN (ensemble), HMA Model (ensemble)