Title: A Careful Examination of Large Language Model Performance on Grade School Arithmetic
Benchmark: GSM1K
Summary: The paper introduces GSM1k, that mirros GSM8K-style but guaranteed absent from model pre-training to show the LLM genuine reasoning capabilities rather than memorization. LLMs stumble on it compared with GSM8k, exposing memorization in many, though frontier models still generalize well.
Contribution: This dataset is human-written, and rigourously matching with GSM8K that produces benefits of comparing with prior gold benchmark, GSM8K. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Elementary mathematical reasoning/ grade-school level Math Word-Problem
Phenomenon defined by authors: Yes
Phenomenon definition: GSM1k consists of 1205 problems requiring only elementary mathematical reasoning to solve.We created GSM1k using human annotators. Annotators were prompted with 3 example GSM8k problems and asked to produce novel problems of a similar difficulty level. The precise instructions and UI given to the annotators is available in Appendix A. All problem annotators were instructed to create problems solvable with only basic arithmetic (addition, subtraction, multiplication, and division) and which did not require any advanced math concepts. As is the case with GSM8k, all problem solutions are positive integers. No language models were used to construct this dataset.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Purpose notes: The benchmark primarily examines to distinct the dataset contamination effect.
Task definition: For each item in GSM1k, the model is given a grade‑school word problem and must compute the solution using only basic arithmetic (addition, subtraction, multiplication, and division). It is expected to output a single positive‑integer answer, which is scored by exact numeric match against the gold key.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Expert-crafted task examples (e.g. hand-written examples), Modified from another benchmark (e.g. translation into another language)
Task source detail: GSM8K is a dataset of 8.5K high quality linguistically diverse grade school math word problems created by human problem writers. Both GSM8k and the new GSM1k benchmark are synthetic, human‑written word‑problem datasets—they look like questions you might find on a grade‑school exam, but they were created from scratch by annotators, not lifted from any real textbook, SAT, GRE, or other examination papers.

Expert-crafted task examples: they did quality checks for selecting reviewers (After initial creation, each task was manually 116 reviewed by a subset of trusted annotators selected for strong past performance. ) 
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: The questions are modeled on existing GSM8k grade‑school questions.
Dataset size: 1205
Dataset metadata available: Yes
Dataset metadata detail: Answer magnitude bucket, estimated resolution steps
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Response format detail: Last numeric token should include the integer-format answers.
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric definition detail: Error analysis with human ablatin to check mis-formatted but correct answers
Metric aggregation: Simple Mean
Metric statistics: Mean, Spearman/Pearson Correlations
(For completeness, they also report the standard Pearson but also mention that Pearson is not the ideal metric due to the curve-of-best-fit not appearing linear.)
Metric subscores: No
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Industry
Benchmark availability: Partially released; At the present, we release only 50 examples from GSM1k to prevent worries around data contamination. (In paper: We do not intend to release GSM1k publicly at this time to prevent a similar problem of data 51 contamination occurring in the future. However, we plan to run recurring evaluations of all major 52 open- and closed- source releases and to continually update our results. We will also open source our 53 entire evaluation code so that the public version of our results can be reproduced. Additionally, we 54 commit to open sourcing the entire benchmark when either 1) the top open source models score over 55 95% on GSM1k or 2) June 2025, whichever comes earlier. )
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: A comparable difficulty to the original benchmark, Human indistinguishability, Matched human difficulty, Prompt/Answer-format ablations

Taxonomy: Reasoning / Mathematics
New taxonomy: General Capability / Mathematical Reasoning