Title: Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
Benchmark: HiST-LLM
Summary: The paper introduces the History Seshat Test for LLMs (HiST-LLM), based on a subset of the Seshat Global History Databank, which provides a structured representation of human historical knowledge, containing 36,000 data points across 600 historical societies and over
2,700 scholarly references. Using this dataset, they benchmark a total of seven models from the Gemini, OpenAI, and Llama families.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: LLM's Expert-level Global History Knowledge
Phenomenon defined by authors: No
Phenomenon definition: The ability of the model to answer expert-level history questions.
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: The ask is to ask the model a multiple-choice question about history.
Task item definition: A multiple-choice question
Task source: Human expert created the examples
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 36000
Dataset metadata available: No
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Mean and standard deviation
Metric subscores: No
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: History / 
New taxonomy: Specific Application / Social Sciences