Title: READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises
Benchmark: READIN
Summary: READIN: a Chinese multi-task benchmark with REalistic And Diverse Input Noises. READIN contains four diverse tasks and requests annotators to re-enter the original test data with two commonly used Chinese input methods: Pinyin input and speech input.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Robustness to realistic input noises
Phenomenon defined by authors: Yes
Phenomenon definition: Robustness to "realistic and diverse input noises" in Chinese NLP, specifically focusing on user-generated inputs in real-world applications.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Comparing performance on clean data versus data with keyboard and speech input noises.
Task item definition: Clean test example from an existing Chinese NLP dataset paired with multiple noisy versions of the same example, along with the original expected output.
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: Simulates real-world user input scenarios where people make typing errors or speech recognition systems misinterpret accented speech. 
Dataset size: train: 34,371
Dataset size notes: dev: 4,430; test: 8,570
Dataset metadata available: Yes
Dataset metadata detail: Input noise type, error rate, annotator information, original task type
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Response format detail: Response format varies depending on the task
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Micro-Average, Worst-Average
Metric statistics: Micro-Average, Worst-Average
Metric subscores: Yes
Metric subscores detail: input noise type (keyboard/speech)
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: (1) Human evaluations showing plausibility of crowd-sourced noises; (2) Diversity analysis of collected noises; (3) Character-level error rates; quantify noise in test sets; (4) Qualitative case studies 
Taxonomy: Language Modelling / Robustness
New taxonomy: General Capability / Reliability