# Neural Machine Reading Comprehension: Methods and Trends
## 4 Datasets and Evaluation Metrics
### Datasets
#### 4.1.4 Free Answering Datasets  
* bAbI  
bAbI, proposed by [PERSON] et al. [74], is a well-known synthetic MRC dataset. It consists of 20 tasks, generated with a simulation of a classic text adventure game. Each task is independent from the others and tests one aspect of text understanding, such as recognizing two or three argument relations and using basic deduction and induction. [PERSON] et al. think that dealing with all these tasks is a prerequisite to full language understanding. Answers are limited to a single word or a list of words and may not be directly found from the original context. The release of the bAbI dataset promotes the development of several promising algorithms, but as all data in bAbI is synthetic, it is a little far from the real world.  
* MS MARCO  
MS MARCO [7] can be viewed as another milestone of MRC after SQuAD [6]. To overcome the weaknesses of previous datasets, it has four predominant features. First, all questions are collected from real user queries. Second, for each question, 10 related documents are searched with the Bing search engine to serve as the context. Third, labeled answers to those questions are generated by humans so that they are not restricted to spans of the context and more reasoning and summarization are required. Fourth, there are multiple answers to each question and sometimes they even conflict, which makes it more challenging for the machine to select the correct answer. MS MARCO makes the MRC dataset closer to the real world.  
* SearchQA  
The work of SearchQA [75] is just like TriviaQA [73]; both follow the general pipeline of question answering. To construct SearchQA, [PERSON] et al. collect question-answer pairs from the J!Archive and then search for snippets related to questions from Google. However, the major difference between SearchQA and TriviaQA is that in TriviaQA there is one document with evidence for each question-answer pair, while in SearchQA each pair has 49.6 related snippets on average.  
* NarrativeQA  
Seeing the limitation in most previous datasets of needing evidence to answer questions from single sentences of original context, [PERSON] et al. [76] design the NarrativeQA. Based on book stories and movie scripts, they search related summaries from Wikipedia and ask co-workers to generate question-answer pairs according to those summaries. What makes NarrativeQA special is that answering questions requires understanding the whole narrative, rather than superficial matching information.  
* DuReader  
Similar to MS MARCO [7], DuReader, released by [PERSON] et al. [77], is another large-scale MRC dataset from real-world application. Questions and documents in DuReader are collected from Baidu Search (search engine) and Baidu Zhidao (question-answering community). Answers are human-generated instead of spans of original context. What makes DuReader different is that it provides new question types such as yes/no and opinion. Compared to factoid questions, these questions sometimes require summaries over multiple parts of documents, which opens an opportunity for the research community.