Title: Are Large Language Models Good Statisticians?
Benchmark: StatQA
Summary: This paper introduces the StatQA benchmark designed to evaluate LLMs’ proficiency in specialized statistical tasks and their applicability assessment capabilities, particularly for hypothesis testing methods.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Statistical Analysis/Literacy
Phenomenon defined by authors: Yes
Phenomenon definition: A typical statistical analysis task involves, given a table D and a statistical question Q, a qualified statistician should be proficient in selecting relevant columns C, choosing the appropriate statistical methods M, and computing the results based on M using C.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: Given a statistical question and a corresponding table of data (information on columns and data types), identify the relevant columns and the appropriate statistical methods needed to derive the correct answer.
Task item definition: A signle task item is a statistical question paired with tabular data (including metadata).
Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 11,623
Dataset metadata available: Yes
Dataset metadata detail: Difficulty, Task, Results
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Prompting strategy e.g. 0-shot, 1-shot, Statistical Task
Metric metascoring: None, 
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: No
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: Yes
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: The authors collect the dataset using postgraduate students in statistics from Kaggle (a real-world platform for data scientists), include expert reviews of the questions, and provide a comparison between human statisticians and LLMs on the task.
Taxonomy: Data Analysis / 
New taxonomy: General Application / Data Analysis