Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: In this paper, the authors propose a new machine reading comprehension (MRC) dataset for Chinese, called Native Chinese Reader (NCR). NCR typically follows a multi-choice MRC task form, which is collected from the exams in Chinese high schools. NCR consists of various text genres, such as modern/classical/poetry Chinese text, which makes this dataset more native to Chinese. The resulting dataset contains 20K questions and has an official train/dev/test split. To test the performance, they used BERT/MacBERT for building baselines. The results show that the best model still has a big gap in human performance, indicating that NCR is challenging.
strengths: 1. The sample quality of NCR is high, as these samples are from real exams.
2. There are various types of text in NCR, making it more challenging.
weaknesses: 1. Apart from NCR's high quality, there is not many new things to learn from this paper, especially for non-Chinese speakers. The collecting process and dataset analyses are quite normal. I am not sure the paper fits NeurIPS.
2. Some of the statements are not in accordance with source codes, and some of the details are missing.
3. The development and test sets are not classified into modern/classical/poetry subsets, which hinders us from understanding the weaknesses in the current Chinese MRC systems.
correctness: The construction of the dataset is relatively sound.
clarity: Mostly. But some of the illustrations are not in accordance with the source codes. Please see 'Additional Feedback' for detail.
relation to prior work: Yes.
additional feedback: Questions and comments:

1. In human evaluation, it is unclear how human performance was calculated. Is it an average score among all three student annotators? What is the inter-agreement?
2. In Table 8, the human performance is somewhat surprised me. As I suppose the performance of modern document to be the highest. However, it seems that modern documents are the most difficult subset. Do you have any further comments on this? The content in line 247-251 seems to be quite shallow to me.
3. Why are the dev/test sets not classified into modern/classical/poetry categories? It is OK to have an overall score, but we need more precise classifications here. Each category is quite different from others, and thus it is not good to blend them as a whole. After checking the original dataset, the reviewer did not find category information. Without such meta-data, we are not able to analyze how our MRC system performs on each category and adjust the model design for further improvements.
4. I downloaded the dataset and source code for baselines from https://sites.google.com/view/native-chinese-reader. After a rough scanning, it seems that the baselines are based on Longformer, but not pure BERT/RoBERTa. While in both the main paper and appendix, there are no illustrations on this nor citation to Longformer. This causes a discrepancy of your paper and the actual baselines that were used. 
5. The test set seems to be already made public, which may affect the integrity of testing process for future works. Why did you not keep it as private (as you have organized a competition beforehand)?

Grammatical issues / Typos / Minor comments:

1. line 126: high level -> high-level
2. line 222: from a industry -> from an industry
3. line 228: token -> taken
4. Table 8: The size of the left and right quote mark differs in '(w/o poetry)'.
5. Please be consistent in showing your evaluation results. For example, in line 236-240, you use percentage for accuracy, while it is not in Table 7/8/9/10.
review_title: A Chinese multi-choice MRC dataset

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper proposes a Chinese machine reading comprehension(MRC) dataset which consists of 8315 documents and 20477 questions. It is more challenging compared to previous Chinese MRC datasets because it covers diverse documents, such as classical Chinese literature and modern Chinese articles, and its average length is one order of magnitude longer than the previous dataset. Experimental results show that the best model achieves 59% test accuracy while humans achieve an average accuracy of 79%, which indicates a significant performance gap between current Chinese MRC models and humans.

strengths: - This is a meaningful dataset for measuring the progress of current Chinese MRC models.
- The dataset and baseline code is publicly available.
- The dataset contains diverse types of questions, such as matching based questions that ask about a fact that has been explicitly described in the document, summary questions that require the readers to understand all the facts stated throughout the entire document, sentiment questions that ask about the implicit sentiment that the author expressed in the document.




weaknesses: - There should be a more detailed description of the implementation of baselines. Given the averaged length of documents (1024 tokens) is much larger than the maximum sequence length of language models (512 tokens for BERT), how do you encode these documents? Do you truncate these documents to the maximum sequence length of language models? What is your truncation strategy?
- There could be some error analysis to help readers understand the current limitation of baseline models. What are the common mistakes made by models? How could further work improve them?

correctness: The data collection and evaluation methods are reasonable to me.
clarity: The paper is well organized and easy to follow.
relation to prior work: This dataset provides several unique features, including:
- long documents,
- classical Chinese documents,
- diverse types of questions.
additional feedback: None
review_title: A Challenging Dataset for Chinese Machine Reading Comprehension.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper presents NCR, a large-scale multiple-choice Chinese MRC dataset consisting of high-school-level exam questions. This dataset is unique and of high quality because it contains longer documents in general, requires more in-depth understanding and reasoning, and encompasses both classical and modern writing styles of Chinese. 

Benchmarks are set by both the authors’ own efforts and the winner of a public competition. The fine-grained analysis reveals various limitations of current MRC models, pointing out ways for further improvement. 
strengths: This work curates a dataset, NCR, that is significantly different from and more challenging than existing Chinese MRC datasets (i.e. C3, CMRC2019). The authors show well-rounded statistics that distinguish the new dataset in terms of document length and question types. Solving the new dataset will require holistic passage understanding, deeper reasoning, and incorporating cultural knowledge. 

A comprehensive benchmark is set for NCR using various existing Chinese pretrained LMs, along with an exhaustive fine-grained analysis and discussion. Furthermore, the authors did not stop there, but went ahead to organize an open competition that elicited a much stronger benchmark. 
weaknesses: In Section 3.3, the authors should describe how they define the boundary between classical and modern Chinese, and how they instructed the annotators to make such a distinction. A common way is to draw the line at the Chinese renaissance (新文学运动) around 1920, yet some may consider the famous fictions *Journey to the West* (《西游记》) and *Water Margin* (《水浒传》) as written in somewhat modern Chinese despite they were composed hundreds of years ago, while contemporary poets may write poetry in ancient formats and thus also blurs the boundary. 

To further show the high quality of the dataset, the authors should probably investigate if there are annotation artifacts and spurious correlations (i.e. certain patterns in the use of language that easily gives away the correct answer). For example, in Table 4 Q3.B, the expression “can only” (“只能”) is an absolute quantifier that usually signals an inappropriate statement. It’d be great to know how well a model can perform by only exploiting this kind of surface patterns in the choices. 
correctness: The dataset collection and split seem appropriate. Experiments and benchmark setup also seem correct. 
clarity: The paper is clear and easy to follow. 
relation to prior work: Prior work is comprehensively summarized, both on English and Chinese MRC datasets. This work has a nice niche that it aims to build a native-level collection of questions, which is inspired by the English MRC trend to collect data from exam questions, while distinguishing itself from prior dataset collection efforts on Chinese MRC. 
additional feedback: In lines 118-123, the number of questions is not consistent: 20477+193==15419+2443+2613 ?

It may be worth highlighting that a lot of questions in NCR are postulated in a way that requires choosing the one “incorrect” option out of 4 options (the other three are “correct”), which requires the ability to understand and reason with negations. 

In Section 4.1, the authors forbid the usage of additional labeled data in the competition (line 218), yet they use additional training data to help finetune the model (line 214). Are these additional data *labeled*? If so, are these part of the NCR dataset so that competition teams were allowed to use them? 

In Section 4.1, did you try fine-tuning your model on only the modern-style part of the NCR dataset? I wonder how well the MRC ability transfers to classical-style documents if it has only seen modern-style data. It may show the importance of having classical-style data in the training set in helping the model understand classical literature. 

Tables for experiment results could be improved aesthetically (e.g. using bold fonts, thicker lines, more space).
review_title: An interesting and intellectually challenging dataset on Chinese MRC

Meta-review: All reviewers agree this paper proposes a new and challenging reading comprehension dataset in Chinese that goes beyond past work in offering high-quality and diverse reading comprehension questions.
Decision: Accept