Question: What is the accuracy of the model on the MovieQA dataset?
Evidence: Many machine learning models in question answering tasks often involve matching mechanism. For example, in factoid question answering such as SQuAD BIBREF1 , one needs to match between query and corpus in order to find out the most possible fragment as answer. In multiple choice question answering, such as MC Test BIBREF2 , matching mechanism can also help make the correct decision.
The easiest way of matching is to calculate the cosine similarity between two vectors. It is generally done by two step: First, encode text into word vectors, sentence vectors or paragraph vectors. Second, simply calculate the cosine similarity between target vectors. This method performs well when applied to word-level matching. However, as for matching between sentences or paragraphs, a single vector is not sufficient to encode all the important information. In order to solve this problem, Wang and Jiang proposed a “compare-aggregate” BIBREF3 framework that performs word-level matching using multiple techniques followed by aggregation with convolutional neural network. In their work, they show that compare-aggregate framework can effectively match two sequences through a wide range.
Although "compare-aggregate" matching mechanism performs well on multiple question answering tasks, it has two deficiencies. First, it tends to aggregate passively through the sequence rather than take the importance of each element into account. That is, "compare aggregate" model considers all the sequential contents equally. Second, "compare aggregate" can only take few neighboring elements into account at the same time because of the limitation of CNN kernel size.
In this paper, we propose Query-based Attention CNN (QACNN) to deal with the deficiencies above. First, we add query-based attention mechanism into original "compare aggregate" model. Moreover, We re-design the aggregation mechanism in "compare aggregate" to a two-staged CNN architecture which comprises word-level aggregation and sentence-level aggregation. In this way, QACNN can efficiently extract features cross sentences.
Our model consists of three components: 1) The similarity mapping layer which converts the input passage, query and choice into feature representation and perform a similarity operation to each other. 2) The attention-based CNN matching network composed of a two-staged CNN focusing on word-level and sentence-level matching respectively. 3) The prediction layer which makes the final decision.
The main contributions of this work are three-fold. First, we introduce a two-staged CNN architecture which integrates information from word-level to sentence-level, and then from sentence-level to passage-level. Second, we introduce attention mechanism into this net. We use specially designed CNN structure and attention mechanism to recognize the pattern of similarity map and eventually identify specific syntactic structure of queries. By transforming passage-query feature into attention maps and applying it to passage-choice matching result, we reasonably give weight to every word in the passage. Lastly, our model reaches 79.99% accuracy on the MovieQA dataset which yields top 1 result on this dataset.
Lastly, our model reaches 79.99% accuracy on the MovieQA dataset which yields top 1 result on this dataset.
Answer: The model reaches 79.99% accuracy on the MovieQA dataset.