Title: Grammatical Error Correction in Low Error Density Domains: A New Benchmark and Analyses
Benchmark: CWEB (Corrected WEBsites)
Summary: CWEB is a benchmark for grammatical error correction that is publicly available and manually annotated by experts. It contains website data from Common Crawl, and includes sentences with low and high error density. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Grammatical error correction
Phenomenon defined by authors: Yes
Phenomenon definition: "Grammatical error correction (GEC) is the task of automatically editing text to remove grammatical errors" (8467).
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Task definition: The model is given text and must identify and correct grammatical errors. 
Task item definition: A single item contains a sentence with in-line corrections. 
Task source: Modified from another benchmark (e.g. translation into another language)
Task source detail: The websites are in English and derived from the first 18 dumps of Common Crawl. Text is filtered to remove non-English and incomplete sentences using justText. The data is manually corrected by expert annotators. The dataset is split into CWEB-S (sponsored websites) and CWEB-G (generic) websites. 
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 13574
Dataset size notes: Development/Test 6729/6845
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test, Train
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric definition detail: The paper uses F_{0.5} and ERRANT, standard grammar error correction metrics, cited on page 8471, to assess the correctness of the correction. Perplexity and semantic similarity are used to measure the semantic change in a sentence after the edit. 
Metric statistics: Simple mean
Metric subscores: No
Metric access: Model access required (e.g. logits)
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors detail that grammar error correction (GEC) models must "perform well in the open-domain setting and generalize, not only to writing produced in the educational context, but also to language production 'in the wild'" (8647). The authors also highlight that a strong GEC benchmark must evaluate "domain adaptation and low precision" in texts with low error density (8647). 
Taxonomy: NLP / Understanding
New taxonomy: General Capability / Natural Language Understanding