Title: Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models
Benchmark: Mathador-LM
Summary: The paper introduces Mathador-LM, a arithmetic reasoning benchmark where models must reach a target number using five input numbers and basic arithmetic operations. Evaluations across various LLMs show that even top models perform far below the level of 3rd-grade students, revealing significant gaps in mathematical reasoning while avoiding test-data contamination issues.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: reasoning, planning, math
Phenomenon defined by authors: Yes
Phenomenon definition: LLM performance of interpreting a ruleset and planning a valid sequence of arithmetic is critical for their reasoning evaluation. We propose an alternative pathway towards reliable examination of LLM performance via dynamic, one-time benchmarks that mitigate contamination by being created on-the-fly, independently for each evaluation run.. This approach mitigates issues such as test-set leakage into training data and provides a reliable method to evaluate closed-source models, even in the absence of detailed information
about their training data.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: No
Phenomenon contested: Widely-agreed
Task definition: Given 5 base numbers and a target, generate up to 4 arithmetic steps that yield the target while obeying game constraints.
Task item definition: Target number, a list of 5 base numbers, a 4‑line expression sequence as a response.
Task source: Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: All items come from an algorithmic generator that samples numbers within predefined ranges and verifies solvability. There are no hand‑written examples, no crowd‑sourcing, no exam questions, and no LLM‑generated prompts.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1000 items
Dataset metadata available: Yes
Dataset metadata detail: Difficulty (calculated by the average attainable score across all solutions for the target)
Sampling method: Random sample (creators defined a task space and sampled from it), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric definition detail: Normalised game‑score accuracy (to yield 0-100%)
Metric aggregation: Simple Mean
Metric statistics: simple mean, 95% confidence interval, percentage point of performance gains over baselines
Metric subscores: Yes
Metric subscores detail: Difficulty level, error type
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: stability study across different regenerated datsets showing low variance
Taxonomy: Reasoning / Mathematical
New taxonomy: General Capability / Mathematical Reasoning