Title: TheoremQA: A Theorem-driven Question Answering Dataset
Benchmark: TheoremQA
Summary: The paper introduces TheoremQA, a benchmark designed to evaluate LLMs’ abilities to apply university-level science theorems across Math, Physics, EE&CS, and Finance. It collects 800 high-quality questions over 354 theorems and evaluates 16 LLMs under chain-of-thought and program-of-thought prompting.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Theorem-driven scientific reasoning
Phenomenon defined by authors: Yes
Phenomenon definition: "The capability to apply formal theorems from domains such as Mathematics, Physics, EE&CS, and Finance to solve structured scientific problems."
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Given a science question implicitly requiring a theorem, produce the correct answer using appropriate reasoning or computation.
Task item definition: Each item is a question paired with an associated theorem and a numerical, list, boolean, or multiple-choice answer.
Task definition detail: Questions are manually curated and sometimes modified to ensure answer formats are automatically evaluable (e.g., floats, integers).
Task source: Human exam questions (e.g. GRE questions), Expert-crafted task examples (e.g. hand-written examples), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Experts refined questions to match answer type constraints, avoid complex symbolic outputs, and ensure relevance to specific theorems.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 800
Dataset metadata available: Yes
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: conducted human evaluation and error analysis to validate the difficulty and construct validity of the benchmark.
Taxonomy: Reasoning / Logical
New taxonomy: General Capability / Logical Reasoning