Title: KoLA: Carefully Benchmarking World Knowledge of Large Language Models
Benchmark: yuKoLACarefullyBenchmarking2024
Summary: This paper introduced Knowledge-oriented LLM Assessment benchmark (KoLA), which aims at carefully benchmarking the world knowledge of LLMs by undertaking meticulous designs considering the aforementioned three factors: ability modeling, known and evolving data sources and contrastive evaluation system.
Contribution: The paper provides a detailed motivation for the design considerations of their dataset, which is well-grounded in learning theory. To what extent this further anthropomorphises LLMs is up for debate, as this grounding assumes that LLMs acquire and consume human knowledge in a manner similar to humans and, as such, should be evaluated in a similar way.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: World knowledge
Phenomenon defined by authors: Yes
Phenomenon definition: Benchmarking the world knowledge of LLMs across four levels; Knowledge Memorization, Knowledge Understanding, Knowledge Applying, and Knowledge Creating.
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Given a question probing for world knowledge, provide an answer
Task item definition: Consists of a question 
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), Modified from another benchmark (e.g. translation into another language), Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 2138
Dataset metadata available: Yes
Sampling method: Random sample (creators defined a task space and sampled from it), Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragarph)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean, Rank
Metric subscores: Yes
Metric subscores detail: For each subtask dataset
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Unclear
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Somewhat
Author validity detail: The dataset design is grounded on human cognitive processes in learning theory which seeks to stimulate acquistion and application of knowledge across different stages
Taxonomy: Knowledge / General
New taxonomy: General Application / General Knowledge