Title: MuCGEC: a Multi-Reference Multi-Source Evaluation Dataset for Chinese Grammatical Error Correction
Benchmark: MuCGEC
Summary: The paper introduces MuCGEC, a multi-reference multi-source evaluation dataset for Chinese Grammatical Error Correction. This dataset contains different Chinese-as-a-Second-Language (CSL) learner sources, with each sentence corrected by three independent annotators and reviewed by a senior annotator.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Chinese Grammatical Error Correction
Phenomenon defined by authors: Yes
Phenomenon definition: "Given a potentially noisy input sentence, grammatical error correction (GEC) aims to detect and correct all errors and produce a clean sentence"
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Models should detect and correct all grammatical errors in a given Chinese sentence, while preserving the original meaning.
Task item definition: Potentially erroneous Chinese sentence (input) paired with multiple human-annotated grammatically correct reference sentences (outputs).
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: The task uses real sentences written by CSL learners that contain actual grammatical errors. 
Dataset size: 1,092,285 (Lang8); 95,320 (HSK)
Dataset size notes: 1,996 sentences for NLPCC18-test, 2,000 for CGED-test, and 1,942 for Lang8-test
Dataset metadata available: Yes
Dataset metadata detail: error types, sentence source, number of references per sentence, number of edits, character counts per sentence
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean
Metric statistics: Simple mean, char-based F0.5 scores for overall performance, along with precision and recall.
Metric subscores: Yes
Metric subscores detail: (1) different data sources, (2) different error types, and (3) different numbers of references
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: 1) establishing human performance baselines, (2) annotator error patterns, (3) demonstrating multiple references improve evaluation accuracy, and (4) demonstrate their character-based metrics.
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding