Title: TGEA: An Error-Annotated Dataset and Benchmark Tasks for Text Generation from Pretrained Language Models
Benchmark: TGEA
Summary: TGEA (Text Generation Error Annotation) is an error-annotated dataset with multiple benchmark tasks for text generation. Following the authors hierachical error taxonomy, crowdsourced workers manually labeled 12k erroneous sentences with semantic information, including error types, associated text spans, error corrections and rationals behind errors.
Contribution: Validation: Crowdsourced workers manually checked each of those sentences and detected 12k erroneous sentences.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Text generation error analysis
Phenomenon defined by authors: Yes
Phenomenon definition: "The key interest of this dataset is detecting and annotating text generation errors from PLMs."
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: The task requires models to analyze machine-generated Chinese text to detect, locate, classify, correct, and explain generation errors according to a comprehensive taxonomy of error types.
Task item definition: A single item consists of machine-generated Chinese text with annotations marking error spans, associated spans, corrections, error type classifications, and explanatory rationales.
Task source: LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 47,058
Dataset size notes: train (37,646), Dev (4,706), test (4,706)
Dataset metadata available: Yes
Dataset metadata detail: error type classification, token counts, error span locations, span distances, error distribution
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Distribution (perplexity, calibration, correlation)
Metric aggregation: None, Separate metrics for each sub-task with no single aggregated score
Metric statistics: Simple means for performance metrics; agreement percentages and Cohen's Kappa for annotation reliability.
Metric subscores: Yes
Metric subscores detail: Erroneous text detection, Erroneous and associated span detection, Error type classification, Error correction, Rationale generation
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: The authors validate their benchmark with inter-annotator agreement statistics for different tasks, Cohen's Kappa coefficients, a rigorous quality control protocol, annotation verification on sampled texts, and human performance baselines.
Taxonomy: Factuality / 
New taxonomy: General Capability / Factuality