Introduction
Many machine learning models in question answering tasks often involve matching mechanism. For example, in factoid question answering such as SQuAD BIBREF1 , one needs to match between query and corpus in order to find out the most possible fragment as answer. In multiple choice question answering, such as MC Test BIBREF2 , matching mechanism can also help make the correct decision.
The easiest way of matching is to calculate the cosine similarity between two vectors. It is generally done by two step: First, encode text into word vectors, sentence vectors or paragraph vectors. Second, simply calculate the cosine similarity between target vectors. This method performs well when applied to word-level matching. However, as for matching between sentences or paragraphs, a single vector is not sufficient to encode all the important information. In order to solve this problem, Wang and Jiang proposed a “compare-aggregate” BIBREF3 framework that performs word-level matching using multiple techniques followed by aggregation with convolutional neural network. In their work, they show that compare-aggregate framework can effectively match two sequences through a wide range.
Although "compare-aggregate" matching mechanism performs well on multiple question answering tasks, it has two deficiencies. First, it tends to aggregate passively through the sequence rather than take the importance of each element into account. That is, "compare aggregate" model considers all the sequential contents equally. Second, "compare aggregate" can only take few neighboring elements into account at the same time because of the limitation of CNN kernel size.
In this paper, we propose Query-based Attention CNN (QACNN) to deal with the deficiencies above. First, we add query-based attention mechanism into original "compare aggregate" model. Moreover, We re-design the aggregation mechanism in "compare aggregate" to a two-staged CNN architecture which comprises word-level aggregation and sentence-level aggregation. In this way, QACNN can efficiently extract features cross sentences.
Our model consists of three components: 1) The similarity mapping layer which converts the input passage, query and choice into feature representation and perform a similarity operation to each other. 2) The attention-based CNN matching network composed of a two-staged CNN focusing on word-level and sentence-level matching respectively. 3) The prediction layer which makes the final decision.
The main contributions of this work are three-fold. First, we introduce a two-staged CNN architecture which integrates information from word-level to sentence-level, and then from sentence-level to passage-level. Second, we introduce attention mechanism into this net. We use specially designed CNN structure and attention mechanism to recognize the pattern of similarity map and eventually identify specific syntactic structure of queries. By transforming passage-query feature into attention maps and applying it to passage-choice matching result, we reasonably give weight to every word in the passage. Lastly, our model reaches 79.99% accuracy on the MovieQA dataset which yields top 1 result on this dataset.
QACNN
In this question answering task, a reading passage , a query and several answer choices are given. P denotes the passage, Q denotes query and C denotes one of the multiple choices. The target of the model is to choose a correct answer A from multiple choices based on informations of P and Q.
Fig. FIGREF1 is the pipeline overview of QACNN. First, we use embedding layer to transform P, Q, and C into word embedding. Then the compare layer generates passage-query similarity map INLINEFORM0 and passage-choice similarity map INLINEFORM1 . The following part is the main component of QACNN. It consists of two-staged CNN architecture. The first stage projects word-level feature into sentence-level, and the second stage projects sentence-level feature into passage-level. Moreover, we apply query-based attention mechanism to each stage on the basis of INLINEFORM2 feature at word level and sentence level respectively. After QACNN Layer, we obtain each choice answer feature. Finally, a prediction layer collects output information from every choice feature and returns the most possible answer.
Similarity Mapping Layer
Similarity Mapping Layer is composed of two part: embedding layer and compare layer. Given a passage P with INLINEFORM0 sentences, a query Q, and a choice C, the embedding layer transforms every words in P, Q and C into word embedding: DISPLAYFORM0
INLINEFORM0 is the length of a sentence in passage, and INLINEFORM1 and INLINEFORM2 are the length of query and length of one single choice respectively. INLINEFORM5 , INLINEFORM6 and INLINEFORM7 are word embeddings. Word embedding can be obtained by any type of embedding technique, such as recurrent neural network BIBREF4 , Sequence-to-sequence model BIBREF5 , Word2vec BIBREF6 , etc. In our work, we simply use pre-trained GloVe word vectors BIBREF7 as the embedding without any further modification or training.
After word embedding step, We want to acquire similarity map which tells us location relationship between passage and query, passage and choices. We use compare layer to compare each passage sentence INLINEFORM0 to INLINEFORM1 and INLINEFORM2 at word level separately as Fig. FIGREF7 and Fig. FIGREF8 show. DISPLAYFORM0
That is, we compare each word in sentences of passage to each word in query and choice. We use cosine similarity as the comparing method here. This step creates two similarity map, passage-query similarity map INLINEFORM0 and passage-choice similarity map INLINEFORM1 .
QACNN Layer
We propose an attention convolutional matching layer to integrate two similarity maps given above. That is, QACNN Layer is used to learn the location relationship pattern. It contains a two-staged CNN combined with query-based attention mechanism. Each stage comprises two major part: attention map and output representation.
Fig. FIGREF10 shows the architecture of the attention map in first stage CNN. We choose INLINEFORM0 sentence slice INLINEFORM1 in INLINEFORM2 , and apply CNN to it using the convolution kernel INLINEFORM3 , where superscript INLINEFORM4 denotes attention map, subscript INLINEFORM5 denotes the first stage CNN. Symbol INLINEFORM6 and INLINEFORM7 represent width of kernel and number of kernel respectively. The generated feature INLINEFORM8 is as follow: DISPLAYFORM0
where INLINEFORM0 is the bias. With INLINEFORM1 covering whole query and several words in the passage, convolution kernels would learn the query syntactic structure and give weight to each passage's location. That's why we use sigmoid function as activation function in this stage. Furthermore, we perform maxpooling to INLINEFORM2 perpendicularly in order to find the largest weight between different kernels in the same location, using maxpool kernel shaped INLINEFORM3 , and then generate word-level attention map INLINEFORM4 for each sentence.
In this stage, we want to acquire passage's sentence features based on the query and the choice respectively. We apply CNN to INLINEFORM0 to aggregate pattern of location relationship and acquire choice-based sentence features. Also, we apply CNN to INLINEFORM1 to acquire query-based sentence features. CNN architecture of output representation part Fig. FIGREF11 is similar to which of attention map part, but we use different kernels INLINEFORM2 and different bias INLINEFORM3 : DISPLAYFORM0
where the superscript INLINEFORM0 denotes output representation. We apply INLINEFORM3 on INLINEFORM4 and INLINEFORM5 then finally generate INLINEFORM6 and INLINEFORM7 using eq.4. We then multiply INLINEFORM8 by the word-level attention map INLINEFORM9 which comes from stage 2.2.1 element-wise through the first dimension. At last, we maxpool INLINEFORM10 and INLINEFORM11 horizontally with kernel shape INLINEFORM12 to get the query-based sentence features INLINEFORM13 and choice-based sentence features INLINEFORM14 .
Fig. FIGREF28 is the architecture of attention map in the second stage CNN. Based on first stage query-based sentence features from section UID15 , we want to acquire sentence-level attention map. The input of this stage is INLINEFORM0 , which will be further refined by CNN with kernel INLINEFORM1 and generates intermediate features INLINEFORM2 . DISPLAYFORM0
Then, same as attention map of first stage, we maxpool INLINEFORM0 with kernel shaped INLINEFORM1 , and obtain sentence-level attention map INLINEFORM2 .
Output representation part of the second stage in Fig. FIGREF29 has two input, sentence-level attention map INLINEFORM0 and sentence-level features INLINEFORM1 . The equations here are similar to those previously mentioned. As follow: DISPLAYFORM0
where INLINEFORM0 , INLINEFORM1 , and INLINEFORM2 . Output representation of certain choice INLINEFORM3 is the final output of QACNN Layer.
Prediction Layer
Prediction Layer is the final part of QACNN. We use INLINEFORM0 to represent the final output representation of the INLINEFORM1 choice . In order to find out the most correct choice, we simply pass INLINEFORM2 to two fully-connected layer and compute probability for each choice using softmax as follows: DISPLAYFORM0 DISPLAYFORM1
where INLINEFORM0 , INLINEFORM1 , INLINEFORM2 , and INLINEFORM3 .
Implementation details
In the preprocessing step, we used pre-trained GloVe vectors for word embeddings, and they would not be updated during training; We padded sentence number in each passage to 101, all word number in each sentence to 100. Word number of queries and choices were padded to 50. For all kernels of CNN, INLINEFORM0 , each of which has three different kernel width INLINEFORM1 ; each of them has same kernel number INLINEFORM2 . We utilized dropout in each CNN layer with dropout rate 0.8. We used Adam BIBREF8 optimizer to optimize our model with initial learning rate 0.001.
Experimental Result
We mainly focus on the MovieQA dataset to train and evaluate our model. MovieQA dataset aims to evaluate automatic story comprehension from both video and text. The data set consists of almost 15,000 multiple choice question answers. Diverse information in this dataset like plots, scripts, sub-title and video captions can be used to infer answers. In our task, only plot informations are used. This challenging dataset is suitable to evaluate QACNN because movie plots are longer than normal reading comprehension task. Each question comes with a set of five highly plausible choices, only one of which is correct; In the MovieQA benchmark, there are 1958 QA pairs in the val set and 3138 QA pairs in the test set.
We used ensemble model in this dataset. The ensemble model consists of eight training runs models with identical structure and hyper-parameter. In the val set, we achieve 77.6% accuracy with single model and 79.0% accuracy with ensemble model. In the test set, as the Table 1. shows, our model achieves 79.99 % accuracy with ensemble model and is the state of the art.
We also applied our model to MCTest dataset which requires machines to answer multiple-choice reading comprehension questions about fictional stories. The original paper describes that a baseline method uses a combination of a sliding window score and a distance based . They achieve 66.7% and 56.7% on MC500 and MC160 separately. Because of the restricted training set and development set, we trained our model on MovieQA training set and applied the result to test MCTest dataset. On MCTest dataset, we still outperform baseline and achieve 68.1% accuracy on MC160 and 61.5% accuracy on MC500.
Discussion
QACNN is a powerful network focusing on multiple choice QA task. It matches between passage and choices based on query information. One of the most important idea in QACNN is two-staged attention map. The first attention map is at word level, representing the importance of each word in paragraph to a certain question; the second attention map, however is at sentence level, representing the importance of each sentence in paragraph to a certain question.
In this section, we designed several experiments to test how two-stage mechanism and attention maps impact on our model.
In this experiment,we focused on the difference between one-stage QACNN and two-stage QACNN. For one-stage QACNN, we didn't split an entire passage into sentences. That is, the shape of passage-query similarity map INLINEFORM0 and passage-choice similarity map INLINEFORM1 are 2D rather than 3D. We convolved them directly on word-level and output passage feature without second-stage involved. The result is shown on table TABREF36 . The result shows that the modified one staged QACNN reaches 66.8% accuracy on validation set, which is ten percent lower than 78.1%, the original QACNN accuracy on validation set.
In this experiment, our target is to validate the effect of query-based attention in QACNN. We modified three different structures from original QACNN Layer below:
1) For the first one, we modified QACNN Layer in section SECREF12 and removed both sentence-level attention map and word-level attention map part from it. However, this modified model would have a deficiency of query information. Therefore, we concatenated the final output representation of INLINEFORM0 and INLINEFORM1 together before prediction layer. The experiment result is shown on the Table TABREF36 . The result is almost ten percent less than the original one. 2) For the second one, we only removed sentence-level attention in section UID18 from QACNN Layer and kept word-level attention in the model. 3) For the last one, instead of removing sentence-level attention, we removed word-level attention from QACNN Layer.
The result is shown in Table TABREF36 . We can see that QACNN(with only word-level attention) performs better than QACNN(without attention); QACNN(with only sentence-level attention) performs better than QACNN(with only word-level attention); And original QACNN which contains both word-level and sentence-level attention does the best job among all. Thus, not only word-level attention but also sentence-level attention can contribute to the performance of QACNN. However, sentence-level attention seems to play a more important role.
Figure. FIGREF31 is the visualization of two attention maps and their corresponding question. The upper half of Figure. FIGREF31 is the attention map at sentence level. We picked the sentence with the largest attention value as target sentence and examine it. Thus, we could get the lower half of Figure. FIGREF31 , which shows the attention map at word level on the target sentence. We used a question from movie, Harry Potter, as an example. The result shows that the sentence with the largest attention value is exactly where the correct answer comes from. It turns out that sentence-level attention map can successfully find out which sentence contains the information of correct answer. As for word-level attention map, we can easily see that the attention map focus mainly on the end of target sentence, which is obviously more important for answering this question.
Conclusion
In this paper, we present an efficient matching mechanism on multiple choice question answering task. We introduce two-staged CNN to match passage and choice on word level and sentence level. In addition, we use query-based CNN attention to enhance matching effect.
The power of the model is verified on MovieQA dataset, which yielded the state of the art result on the dataset. In the future, we are now working on training our model based on our own trained embedding with TF-IDF BIBREF9 weighting. Furthermore, we would like to test our model on open-answer task like SQuaD by seeing the whole corpus as an “answer pool" and solve it like multiple choice question.

Question: How do they split text to obtain sentence levels?
Answer: Unanswerable

Question: Do they experiment with their proposed model on any other dataset other than MovieQA?
Answer: Yes

Yes