Title: CogBench: a large language model walks into a psychology lab
Benchmark: CogBench
Summary: CogBench is a benchmark that uses seven cognitive psychology experiments to evaluate LLMs by assessing their behavioral characteristics. CogBench provides ten behavioral metrics to phenotype LLM behavior e.g. model-based reasoning, exploration strategies, metacognition, and risk-taking tendencies. The benchmark is applied to 40 different LLMs.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Cognitive behavioral phenotyping
Phenomenon defined by authors: Yes
Phenomenon definition: Its not very clearly defined. There is a footnote: "A computational phenotype is a collection of mathematically derived parameters that precisely describe individuals across different domains "
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: The task consists of seven cognitive psychology experiments where LLMs must respond to textual prompts simulating classic experimental paradigms such as the two-armed bandit problems.
Task item definition: textual prompt
Task source: Real task examples (e.g. GitHub issues), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: The benchmark uses well-established cognitive psychology experiments (for humans).
Dataset metadata available: Yes
Dataset metadata detail: Performance metrics and behavoural metrics
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test
Response format: Multiple choice, Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), Distribution (perplexity, calibration, correlation)
Metric aggregation: Simple Mean, The metrics are normalized against human performance
Metric statistics: The metrics are averaged and normalized against human performance
Metric subscores: Yes
Metric subscores detail: They are split by performance metrics (e.g. probabilistic reasoning) and behavoural metrics (e.g. meta-cognition)
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: They explicitly note that their benchmark is based on "well-established experimental paradigms from the cognitive psychology literature, providing a unique set of advantages over traditional LLM benchmarks" because these measures "have been extensively validated over many years and shown to capture general cognitive constructs."
Taxonomy: Psychology / 
New taxonomy: Specific Application / Social Sciences