Title: GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers
Benchmark: GSM-Plus
Summary: The authors create GSM‑PLUS, an adversarial extension of GSM8K that perturbs each seed question in eight different ways to test the robustness of LLM mathematical reasoning. They observe the huge performance drops from models on critical‑thinking and arithmetic variations with four prompting strategies.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: reasoning, math
Phenomenon defined by authors: Yes
Phenomenon definition: Math abilities (numerical, arithmetic, understanding, distractor handling, and critical thinking) with 8 perturbation types
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Given a word‑problem, the model must output the numerical answer.
Task item definition: Question context + Question (+additional constraints with variation on the question)
Task definition detail: Robustness is measured by the consistency of solving both the seed (original) question and its perturbed variants.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: GPT‑4 first rewrites each GSM8K test item, and approximately 20% of variations are manually revised by paid annotators.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1319 seed (original) data, 10552 perturbed data
Dataset metadata available: Yes
Dataset metadata detail: Seed, perturbation type, subcategory, gold answer
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring), Distribution (perplexity, calibration, correlation)
Metric definition detail: PDR (performance drop rate), ASP (accurately-solved pairs)
Metric aggregation: Simple Mean
Metric statistics: Means and percentage differences
Metric subscores: Yes
Metric subscores detail: different perturbation level types (Numerical Substitution; Digit Expansion; Integer-decimal-fraction Conversion; Adding Operation; Reversing Operation; Problem Understanding; Distractor Insertion; Critical Thinking.)
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Annotation consistency scores, and pass-rates to justify the validty.
Taxonomy: Reasoning / Mathematical
New taxonomy: General Capability / Mathematical Reasoning
Procedural notes: https://qtli.github.io/GSM-Plus/