Title: Mathematical Capabilities of ChatGPT
Benchmark: GHOSTS (and miniGHOSTS)
Summary: This paper introduces GHOSTS and miniGHOSTS, the first graduate-level natural language benchmarks designed to evaluate the mathematical reasoning abilities of ChatGPT and GPT-4. It covers a broad spectrum of tasks, including proof completion, symbolic integration, and Olympiad problem solving, with human-annotated fine-grained evaluation of errors and failures.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Advanced mathematical reasoning in natural language
Phenomenon defined by authors: Yes
Phenomenon definition: "The ability to perform and reason about advanced mathematics presented in natural language across multiple reasoning tasks including proofs, computation, problem-solving, and search."
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Given a natural-language mathematical prompt, solve the problem (e.g., proof completion, computation, concept retrieval) and produce a correct, concise response.
Task item definition: Each item consists of a prompt, the model's output, and human-assigned ratings and error codes based on correctness and reasoning quality.
Task definition detail: The benchmark evaluates both overall answer correctness and fine-grained error types (e.g., logical gaps, computation errors).
Task source: Human exam questions (e.g. GRE questions), Expert-crafted task examples (e.g. hand-written examples), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 709 examples (GHOSTS); 170 examples (miniGHOSTS)
Dataset metadata available: Yes
Dataset metadata detail: Includes metadata on difficulty (M1–M4), question type (Q1–Q5), error types, MSC (Mathematics Subject Classification), and prompt timestamps.
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Subscores reported by subdataset (e.g., Grad-Text, Olympiad-Problem-Solving), difficulty (M1–M4), question type (Q1–Q5).
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The dataset was manually curated and outputs rated by mathematicians following a strict rubric; error analysis and prompt-level validation are included.
Taxonomy: Reasoning / Mathematical
New taxonomy: General Capability / Mathematical Reasoning