# Neural Machine Reading Comprehension: Methods and Trends
## 4 Datasets and Evaluation Metrics
### Datasets
#### 4.1.1 Cloze Tests Datasets  
* CNN & Daily Mail  
This dataset, built by [PERSON] et al. [5], is one of the most representative cloze-style MRC datasets. CNN & Daily Mail, consisting of 93,000 articles from the CNN and 220,000 articles from the Daily Mail, is indeed large-scale and makes it possible to use deep-learning approaches in MRC. Considering that bullet points are abstractive and have little sentence overlap with documents, [PERSON] et al. replace one entity at a time with a placeholder in bullet points and evaluate the machine reading system by asking the machine to read the documents and then predict which entity the placeholder in the bullet points referred to. As questions are not posed directly from documents, this task is challenging, and some information extraction methods fail to deal with it. This methodology of creating MRC datasets has enlightened a lot of other research [63; 64; 65]. To avoid the situation where questions can be answered by knowledge outside of the documents, all entities in documents are anonymized by random markers. * CBT  
[PERSON] et al. [66] design a cloze-style MRC dataset, the Children's Book Test (CBT), from another perspective. They collect 108 children's books and form each sample with 21 consecutive sentences from chapters in those books. To generate questions, a word from the 21 st sentence is removed, and the other 20 sentences act as context. Nine incorrect words, whose type is the same as the answer, are selected at random from the context as candidate answers. There are some differences between the CNN & Daily Mail and CBT dataset. First, unlike the CNN & Daily Mail, entities in the CBT dataset are not anonymized, so models can use background knowledge from wider contexts. Second, missing items in the CNN & Daily Mail are limited to named entities, but in the CBT there are four distinct types: named entities, nouns, verbs, and prepositions. Third, the CBT provides candidate answers, which simplifies the task in a way. Overall, with the appearance of the CBT, context, which plays a significant role in human comprehension, has gained much more attention. Considering that more data can significantly improve the performance of neural network models, [PERSON] et al. [67] introduce the BookTest, which enlarges the CBT dataset 60 times and enables training of larger models. * LAMBADA  
To account for the meaning of the wider context, [PERSON] et al. [68] propose the language modeling broadened to account for discourse aspects (LAMBADA) dataset. Similar to the CBT, LAMBADA also uses books as the source, and the task is word prediction. However, the word that needs to be predicted in LAMBADA is the last word in the target sentence, while in CBT any word in the target sentence may be targeted. Moreover, [PERSON] et al. find that some samples in the CBT can be guessed with the target sentence alone rather than in a wider context. To overcome this shortcoming, there is a constraint in LAMBADA that makes it difficult to predict the target word correctly only with the target sentence. That is to say, compared with CBT, LAMBADA requires more understanding of the wider context. * Who-did-What  
To better evaluate the understanding of natural language, researchers try to avoid sentence overlap between questions and documents when constructing MRC datasets. [PERSON] et al. [64] provide new insight into how to reduce the syntactic similarity. In the \"Who-did-What\" dataset, each sample is formed from two independent articles; one serves as the context and questions are generated from the other. This approach can be used by other corpora, in which articles do not have summary points, unlike CNN & Daily Mail. There is another feature of Who-did-What, as shown in the name; the dataset only pays attention to the personal name entity, which may be its limitation. * CLOTH  
In contrast to the above automatically generated datasets, cloze test by teachers (CLOTH) [69] is human-created, collected from English exams for Chinese students.