Title: NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks
Benchmark: NumGLUE
Summary: NUMGLUE is a multi-task benchmark designed to test AI systems' ability to perform simple arithmetic reasoning across eight diverse tasks. It introduces four new datasets and shows that even state-of-the-art models underperform compared to humans, emphasizing the need for robust, generalizable mathematical reasoning.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Arithmetic reasoning in natural language
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Solve problems involving arithmetic reasoning presented in various formats (e.g., word problems, fill-in-the-blank, NLI, reading comprehension).
Task item definition: Each item is a question requiring a numeric or categorical answer, involving simple arithmetic possibly combined with commonsense or domain knowledge.
Task definition detail: Tasks include Commonsense+Arithmetic, Domain-specific+Arithmetic, Quantitative NLI, Reading Comprehension with Numerical Reasoning, and traditional Word Problems.
Task source: Human exam questions (e.g. GRE questions), Expert-crafted task examples (e.g. hand-written examples)
Task source detail: Four novel tasks are manually created, while four others adapt existing datasets like DROP, EQUATE, MAWPS.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: ~100,000 examples total
Dataset size notes: Training: ~70% Development: ~10% Test: ~20%
Dataset metadata available: Yes
Sampling method: Random sample (creators defined a task space and sampled from it), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Taxonomy: Reasoning / Mathematical
New taxonomy: General Capability / Mathematical Reasoning