Abstract: Understanding open-domain text is one of the primary challenges in natural language processing (NLP). Machine comprehension benchmarks evaluate the system's ability to understand text based on the text content only. In this work, we investigate machine comprehension on MCTest, a question answering (QA) benchmark. Prior work is mainly based on feature engineering approaches. We come up with a neural network framework, named hierarchical attention-based convolutional neural network (HABCNN), to address this task without any manually designed features. Specifically, we explore HABCNN for this task by two routes, one is through traditional joint modeling of passage, question and answer, one is through textual entailment. HABCNN employs an attention mechanism to detect key phrases, key sentences and key snippets that are relevant to answering the question. Experiments show that HABCNN outperforms prior deep learning approaches by a big margin.

Introduction: Endowing machines with the ability to understand natural language is a long-standing goal in NLP and holds the promise of revolutionizing the way in which people interact with machines and retrieve information. Richardson et al. richardson2013mctest proposed the task of machine comprehension, along with MCTest, a question answering dataset for evaluation. The ability of the machine to understand text is evaluated by posing a series of questions, where the answer to each question can be found only in the associated text. Solutions typically focus on some semantic interpretation of the text, possibly with some form of probabilistic or logic inference, to answer the question. Despite intensive recent work BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , the problem is far from solved. Machine comprehension is an open-domain question-answering problem which contains factoid questions, but the answers can be derived by extraction or induction of key clues. Figure FIGREF1 shows one example in MCTest. Each example consists of one document, four associated questions; each question is followed by four answer candidates in which only one is correct. Questions in MCTest have two categories: “one” and “multiple”. The label means one or multiple sentences from the document are required to answer this question. To correctly answer the first question in the example, the two blue sentences are required; for the second question instead, only the red sentence can help. The following observations hold for the whole MCTest. (i) Most of the sentences in the document are irrelavent for a given question. It hints that we need to pay attention to just some key regions. (ii) The answer candidates can be flexible text in length and abstraction level, and probably do not appear in the document. For example, candidate B for the second question is “outside”, which is one word and does not exist in the document, while the answer candidates for the first question are longer texts with some auxiliary words like “Because” in the text. This requires our system to handle flexible texts via extraction as well as abstraction. (iii) Some questions require multiple sentences to infer the answer, and those vital sentences mostly appear close to each other (we call them snippet). Hence, our system should be able to make a choice or compromise between potential single-sentence clue and snippet clue. Prior work on this task is mostly based on feature engineering. This work, instead, takes the lead in presenting a deep neural network based approach without any linguistic features involved. Concretely, we propose HABCNN, a hierarchical attention-based convolutional neural network, to address this task in two roadmaps. In the first one, we project the document in two different ways, one based on question-attention, one based on answer-attention and then compare the two projected document representations to determine whether the answer matches the question. In the second one, every question-answer pair is reformatted into a statement, then the whole task is treated through textual entailment. In both roadmaps, convolutional neural network (CNN) is explored to model all types of text. As human beings usually do for such a QA task, our model is expected to be able to detect the key snippets, key sentences, and key words or phrases in the document. In order to detect those informative parts required by questions, we explore an attention mechanism to model the document so that its representation contains required information intensively. In practice, instead of imitating human beings in QA task top-down, our system models the document bottom-up, through accumulating the most relevant information from word level to snippet level. Our approach is novel in three aspects. (i) A document is modeled by a hierarchical CNN for different granularity, from word to sentence level, then from sentence to snippet level. The reason of choosing a CNN rather than other sequence models like recurrent neural network BIBREF4 , long short-term memory unit (LSTM BIBREF5 ), gated recurrent unit (GRU BIBREF6 ) etc, is that we argue CNNs are more suitable to detect the key sentences within documents and key phrases within sentences. Considering again the second question in Figure FIGREF1 , the original sentence “They sat by the fire and talked about he insects” has more information than required, i.e, we do not need to know “they talked about the insects”. Sequence modeling neural networks usually model the sentence meaning by accumulating the whole sequence. CNNs, with convolution-pooling steps, are supposed to detect some prominent features no matter where the features come from. (ii) In the example in Figure FIGREF1 , apparently not all sentences are required given a question, and usually different snippets are required by different questions. Hence, the same document should have different representations based on what the question is. To this end, attentions are incorporated into the hierarchical CNN to guide the learning of dynamic document representations which closely match the information requirements by questions. (iii) Document representations at sentence and snippet levels both are informative for the question, a highway network is developed to combine them, enabling our system to make a flexible tradeoff. Overall, we make three contributions. (i) We present a hierarchical attention-based CNN system “HABCNN”. It is, to our knowledge, the first deep learning based system for this MCTest task. (ii) Prior document modeling systems based on deep neural networks mostly generate generic representation, this work is the first to incorporate attention so that document representation is biased towards the question requirement. (iii) Our HABCNN systems outperform other deep learning competitors by big margins.

Question: what was the margin their system outperformed previous ones?
Answer: 15.6 and 16.5 for accuracy and NDCG on MCTest-150, 7.3 and 4.6 on MCTest-500.