Dataset for the First Evaluation on Chinese Machine Reading Comprehension

Abstract: Machine Reading Comprehension (MRC) has become enormously popular recently and has attracted a lot of attention. However, existing reading comprehension datasets are mostly in English. To add diversity in reading comprehension datasets, in this paper we propose a new Chinese reading comprehension dataset for accelerating related research in the community. The proposed dataset contains two different types: cloze-style reading comprehension and user query reading comprehension, associated with large-scale training data as well as human-annotated validation and hidden test set. Along with this dataset, we also hosted the first Evaluation on Chinese Machine Reading Comprehension (CMRC-2017) and successfully attracted tens of participants, which suggest the potential impact of this dataset.

Machine Reading Comprehension (MRC) has become enormously popular in recent research, which aims to teach the machine to comprehend human languages and answer the questions based on the reading materials. Among various reading comprehension tasks, the cloze-style reaing comprehension is relatively easy to follow due to its simplicity in definition, which requires the model to fill an exact word into the query to form a coherent sentence according to the document material. Several cloze-style reading comprehension datasets are publicly available, such as CNN/Daily Mail BIBREF0 , Children's Book Test BIBREF1 , People Daily and Children's Fairy Tale BIBREF2 .
In this paper, we provide a new Chinese reading comprehension dataset, which has the following features
We also host the 1st Evaluation on Chinese Machine Reading Comprehension (CMRC2017), which has attracted over 30 participants and finally there were 17 participants submitted their evaluation systems for testing their reading comprehension models on our newly developed dataset, suggesting its potential impact. We hope the release of the dataset to the public will accelerate the progress of Chinese research community on machine reading comprehension field.
We also provide four official baselines for the evaluations, including two traditional baselines and two neural baselines. In this paper, we adopt two widely used neural reading comprehension model: AS Reader BIBREF3 and AoA Reader BIBREF4 .
The rest of the paper will be organized as follows. In Section 2, we will introduce the related works on the reading comprehension dataset, and then the proposed dataset as well as our competitions will be illustrated in Section 3. The baseline and participant system results will be given in Section 4 and we will made a brief conclusion at the end of this paper.
In this section, we will introduce several public cloze-style reading comprehension dataset.
Some news articles often come along with a short summary or brief introduction. Inspired by this, Hermann et al. hermann-etal-2015 release the first cloze-style reading comprehension dataset, called CNN/Daily Mail. Firstly, they obtained large-scale CNN and Daily Mail news data from online websites, including main body and its summary. Then they regard the main body of the news as the Document. The Query is generated by replacing a name entity word from the summary by a placeholder, and the replaced named entity word becomes the Answer. Along with the techniques illustrated above, after the initial data generation, they also propose to anonymize all named entity tokens in the data to avoid the model exploit world knowledge of specific entities, increasing the difficulties in this dataset. However, as we have known that world knowledge is very important when we do reading comprehension in reality, which makes this dataset much artificial than real situation. Chen et al. chen-etal-2016 also showed that the proposed anonymization in CNN/Daily Mail dataset is less useful, and the current models BIBREF3 , BIBREF5 are nearly reaching ceiling performance with the automatically generated dataset which contains much errors, such as coreference errors, ambiguous questions etc.
Another popular cloze-style reading comprehension dataset is the Children's Book Test (CBT) proposed by Hill et al. hill-etal-2015 which was built from the children's book stories. Though the CBT dataset also use an automatic way for data generation, there are several differences to the CNN/Daily Mail dataset. They regard the first 20 consecutive sentences in a story as the Document and the following 21st sentence as the Query where one token is replaced by a placeholder to indicate the blank to fill in. Unlike the CNN/Daily Mail dataset, in CBT, the replaced word are chosen from various types: Name Entity (NE), Common Nouns (CN), Verbs (V) and Prepositions (P). The experimental results showed that, the verb and preposition answers are not sensitive to the changes of document, so the following works are mainly focusing on solving the NE and CN genres.
The previously mentioned datasets are all in English. To add diversities to the reading comprehension datasets, Cui et al. cui-etal-2016 proposed the first Chinese cloze-style reading comprehension dataset: People Daily & Children's Fairy Tale, including People Daily news datasets and Children's Fairy Tale datasets. They also generate the data in an automatic manner, which is similar to the previous datasets. They choose short articles (several hundreds of words) as Document and remove a word from it, whose type is mostly named entities and common nouns. Then the sentence that contains the removed word will be regarded as Query. To add difficulties to the dataset, along with the automatically generated evaluation sets (validation/test), they also release a human-annotated evaluation set. The experimental results show that the human-annotated evaluation set is significantly harder than the automatically generated questions. The reason would be that the automatically generated data is accordance with the training data which is also automatically generated and they share many similar characteristics, which is not the case when it comes to human-annotated data.
In this section, we will briefly introduce the evaluation tracks and then the generation method of our dataset will be illustrated in detail.
The proposed dataset is typically used for the 1st Evaluation on Chinese Machine Reading Comprehension (CMRC-2017), which aims to provide a communication platform to the Chinese communities in the related fields. In this evaluation, we provide two tracks. We provide a shared training data for both tracks and separated evaluation data.
Cloze Track: In this track, the participants are required to use the large-scale training data to train their cloze system and evaluate on the cloze evaluation track, where training and test set are exactly the same type.
User Query Track: This track is designed for using transfer learning or domain adaptation to minimize the gap between cloze training data and user query evaluation data, i.e. training and testing is fairly different.
Following Rajpurkar et al. rajpurkar-etal-2016, we preserve the test set only visible to ourselves and require the participants submit their system in order to provide a fair comparison among participants and avoid tuning performance on the test set. The examples of Cloze and User Query Track are given in Figure 1 .
The cloze-style reading comprehension can be described as a triple $\langle \mathcal {D}, \mathcal {Q}, \mathcal {A} \rangle $ , where $\mathcal {D}$ represents Document, $\mathcal {Q}$ represents Query and the $\mathcal {A}$ represents Answer. There is a restriction that the answer should be a single word and should appear in the document, which was also adopted in BIBREF1 , BIBREF2 . In our dataset, we mainly focus on answering common nouns and named entities which require further comprehension of the document.
Following Cui et al. BIBREF2 , we also use similar way to generate our training data automatically. Firstly we roughly collected 20,000 passages from children's reading materials which were crawled in-house. Briefly, we choose an answer word in the document and treat the sentence containing answer word as the query, where the answer is replaced by a placeholder “XXXXX”. The detailed procedures can be illustrated as follows.
Pre-processing: For each sentence in the document, we do word segmentation, POS tagging and dependency parsing using LTP toolkit BIBREF6 .
Dependency Extraction: Extract following dependencies: COO, SBV, VOB, HED, FOB, IOB, POB, and only preserve the parts that have dependencies.
Further Filtering: Only preserve SBV, VOB and restrict the related words not to be pronouns and verbs.
Frequency Restriction: After calculating word frequencies, only word frequency that greater than 2 is valid for generating question.
Question Restriction: Only five questions can be extracted within one passage.
Apart from the automatically generated large-scale training data, we also provide human-annotated validation and test data to improve the estimation quality. The annotation procedure costs one month with 5 annotators and each question is cross-validated by another annotator. The detailed procedure for each type of dataset can be illustrated as follows.
For the validation and test set in cloze data, we first randomly choose 5,000 paragraphs each for automatically generating questions using the techniques mentioned above. Then we invite our resource team to manually select 2,000 questions based on the following rules.
Whether the question is appropriate and correct
Whether the question is hard for LMs to answer
Only select one question for each paragraph
Unlike the cloze dataset, we have no automatic question generation procedure in this type. In the user query dataset, we asked our annotator to directly raise questions according to the passage, which is much difficult and time-consuming than just selecting automatically generated questions. We also assign 5,000 paragraphs for question annotations in both validation and test data. Following rules are applied in asking questions.
The paragraph should be read carefully and judged whether appropriate for asking questions
No more than 5 questions for each passage
The answer should be better in the type of nouns, named entities to be fully evaluated
Too long or too short paragraphs should be skipped
In this section, we will give several baseline systems for evaluating our datasets as well as presenting several top-ranked systems in the competition.
We set several baseline systems for testing basic performance of our datasets and provide meaningful comparisons to the participant systems. In this paper, we provide four baseline systems, including two simple ones and two neural network models. The details of the baseline systems are illustrated as follows.
Random Guess: In this baseline, we randomly choose one word in the document as the answer.
Top Frequency: We choose the most frequent word in the document as the answer.
AS Reader: We implemented Attention Sum Reader (AS Reader) BIBREF3 for modeling document and query and predicting the answer with the Pointer Network BIBREF7 , which is a popular framework for cloze-style reading comprehension. Apart from setting embedding and hidden layer size as 256, we did not change other hyper-parameters and experimental setups as used in Kadlec et al. kadlec-etal-2016, nor we tuned the system for further improvements.
AoA Reader: We also implemented Attention-over-Attention Reader (AoA Reader) BIBREF4 which is the state-of-the-art model for cloze-style reading comprehension. We follow hyper-parameter settings in AS Reader baseline without further tuning.
In the User Query Track, as there is a gap between training and validation, we follow BIBREF8 and regard this task as domain adaptation or transfer learning problem. The neural baselines are built by the following steps.
We first use the shared training data to build a general systems, and choose the best performing model (in terms of cloze validation set) as baseline.
Use User Query validation data for further tuning the systems with 10-fold cross-validations.
Increase dropout rate BIBREF9 to 0.5 for preventing over-fitting issue.
All baseline systems are chosen according to the performance of the validation set.
The participant system results are given in Table 2 and 3 .
As we can see that two neural baselines are competitive among participant systems and AoA Reader successfully outperform AS Reader and all participant systems in single model condition, which proves that it is a strong baseline system even without further fine-tuning procedure. Also, the best performing single model among participant systems failed to win in the ensemble condition, which suggest that choosing right ensemble method is essential in most of the competitions and should be carefully studied for further performance improvements.
Not surprisingly, we only received three participant systems in User Query Track, as it is much difficult than Cloze Track. As shown in Table 3 , the test set performance is significantly lower than that of Cloze Track, due to the mismatch between training and test data. The baseline results give competitive performance among three participants, while failed to outperform the best single model by ECNU, which suggest that there is much room for tuning and using more complex methods for domain adaptation.
In this paper, we propose a new Chinese reading comprehension dataset for the 1st Evaluation on Chinese Machine Reading Comprehension (CMRC-2017), consisting large-scale automatically generated training set and human-annotated validation and test set. Many participants have verified their algorithms on this dataset and tested on the hidden test set for final evaluation. The experimental results show that the neural baselines are tough to beat and there is still much room for using complicated transfer learning method to better solve the User Query Task. We hope the release of the full dataset (including hidden test set) could help the participants have a better knowledge of their systems and encourage more researchers to do experiments on.
We would like to thank the anonymous reviewers for their thorough reviewing and providing thoughtful comments to improve our paper. We thank the Sixteenth China National Conference on Computational Linguistics (CCL 2017) and Nanjing Normal University for providing space for evaluation workshop. Also we want to thank our resource team for annotating and verifying evaluation data. This work was supported by the National 863 Leading Technology Research Project via grant 2015AA015409.

Figure and table captions:
Table 1: Statistics of the dataset for the 1st Evaluation on Chinese Machine Reading Comprehension (CMRC-2017).
Figure 1: Examples of the proposed datasets (the English translation is in grey). The sentence ID is depicted at the beginning of each row. In the Cloze Track, “XXXXX” represents the missing word.
Table 2: Results on Cloze Track. The best baseline and participant systems are depicted in bold face.
Table 3: Results on User Query Track. Due to the using of validation data, we did not report its performance.

Questions about this paper:
What two types the Chinese reading comprehension dataset consists of?
For which languages most of the existing MRC datasets are created?

Answers:
cloze-style reading comprehension and user query reading comprehension questions
English