Title: NOISEBENCH: Benchmarking the Impact of Real Label Noise on Named Entity Recognition
Benchmark: NOISEBENCH
Summary: NOISEBENCH evaluates label noise in Named Entity Recognition (NER) models. It provides multiple variants of the same dataset with different types of real noise (expert errors, crowd- sourcing errors, automatic annotation errors and LLM errors).
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Label Noise on Named Entity Recognition
Phenomenon defined by authors: Yes
Phenomenon definition: "Benchmark for measuring the impact of label noise in the training data on the prediction quality of trained NER models"
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Named entity recognition (NER), which requires detecting and classifying named entity types in text.
Task item definition: Sentence where named entities must be identified and classified into one of four entity types (PER, ORG, LOC, MISC).
Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language)
Task source detail: Benchmark is derived from the CoNLL-03 dataset, which consists of real news articles with manually annotated named entities
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Named entity recognition is a common NLP task with potential real-world applications. 
Dataset size: Test: 3,427 sentences (test set)
Dataset size detail: test set: 3,427 sentences, 5,725 entity mentions; training set: 5,885 sentences, 9,685 entity mentions.
Dataset size notes: Train: 5,885 sentences; Validation: Approximately 17%
Dataset metadata available: Yes
Dataset metadata detail: noise level percentage, error types, entity counts
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Micro-averaged entity-level F1 score reported as means across 3 runs with standard deviations. Simple means used for comparing approaches across different noise types.
Metric subscores: Yes
Metric subscores detail: Noise types (Expert, Crowd++, Crowd, Distant, Weak, LLM); Per-class metrics (LOC, ORG, PER, MISC); Token-level vs. entity-level F1 scores
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: Yes
Human baseline: No
Authors discuss validity: Yes
Author validity detail: When comparing real noise to simulated noise, they provide evidence showing how models immediately memorize real noise patterns rather than going through distinct learning phases.
Taxonomy: NLP / Extraction
New taxonomy: General Application / Extraction