Title: DUMB: A Benchmark for Smart Evaluation of Dutch Models
Benchmark: DUMB
Summary: DUMB, a benchmark for evaluating Dutch language models across nine tasks. The authors propose Relative Error Reduction (RER) for better cross-task comparison and evaluate pre-trained models, finding that current Dutch models underperform while identifying strategies for future model improvements.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Dutch language understanding
Phenomenon defined by authors: No
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Word-level tasks (POS tagging, NER), word-pair tasks (word sense disambiguation, pronoun resolution), sentence-pair tasks (causal reasoning, natural language inference), and document-level tasks (sentiment analysis, abusive language detection, question answering).
Task item definition: Task-specific, e.g., input text (word, sentence, or document) with corresponding label/target output.
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Representative task (e.g. answering medical licensing exam questions), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Standard NLP evaluation benchmarks rather than complete real-world applications, but they are important for real-world Dutch language processing applications.
Dataset size: train (sum): 283,112
Dataset size notes: test (sum): 23,406 ; dev (sum): 17,152
Dataset metadata available: No
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Relative Error Reduction (RER)
Metric statistics: Binomial mixed effects regression models
Metric subscores: Yes
Metric subscores detail: Scores for each of the nine individual tasks; grouped average scores by model type, model size, pre-training language
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: NLP / Understanding
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Natural Language Understanding