Title: NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes
Benchmark: NPHardEval
Summary: NPHardEval is a comprehensive benchmark consisting of 900 algorithmic tasks across P, NP-Complete, and NP-Hard complexity classes, designed specifically to evaluate large language models' reasoning capabilities. It targets to accurately assess LLMs' algorithmic problem-solving abilities across varying computational complexity levels.
Contribution: (1) 9 distinct tasks × 10 difficulty levels,
(2) automatic generation & verification pipeline,
(3) first benchmark to ground LLM evaluation in computational‑complexity theory,
(4) open‑sourced
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: reasoning, complexity
Phenomenon defined by authors: Yes
Phenomenon definition: Reasoning is operationalised as solving decision problems in P, NP‑Complete, and NP‑Hard
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Each task provides a specific problem scenario along with expected output format (such as True/False decisions, path lists, integers, color mappings, or other specified formats). 
Task item definition: One problem instance + required output (True/False, path list, integer, colour mapping, etc.).
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: Data synthesis for graph/linear tasks, difficulty depending on size/weights
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Not intended for direct end‑user interaction but aims for analytic rigor.
Dataset size: 900 per monthly release (100 per task)
Dataset size notes:  NA
Dataset metadata available: Yes
Dataset metadata detail: Complexity class, task type, algorithmic difficulty level
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Weighted Mean
Metric statistics: Wilcoxon tests, variance analysis
Metric subscores: Yes
Metric subscores detail: Task difficulty, complexity classes, model performances
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Reasoning / 
New taxonomy: General Capability / Mathematical Reasoning
Procedural notes: MIT licensed, monthly refreshed/released the additional data (partially)