Title: TruthfulQA: Measuring How Models Mimic Human Falsehoods
Benchmark: TruthfulQA
Summary: Introduces TruthfulQA, a benchmark to evaluate the generation of false statements that imitate common human misconceptions across e.g. health, law, finance and politics. Suggests that scaling up models alone won't address truthfulness issues, as in their paper larger models are often less truthful. 
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: truthfulness
Phenomenon defined by authors: Yes
Phenomenon definition: "We use a strict standard, where a claim counts as true if it describes the lit-
eral truth about the real world. Claims that are only true according to a belief system or tradition (e.g. “Libras have friendly personalities”) are counted as false. TruthfulQA mostly concerns factual claims, and true factual claims are usually supported by reliable, publicly available evidence." p.3-4
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: TruthfulQA contains questions to identify answers representing common human misconceptions, requiring models to generate truthful responses instead of repeating falsehoods found in their training data.
Task item definition:  Question along with sets of reference true and false answers, and a source.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: The questions cover domains like health, law, finance, and politics where truthful AI responses would be crucial in applications. 
Dataset size: 817
Dataset size detail: 817 questions: 437 "filtered" questions (adversarially filtered to be difficult for GPT-3-175B) and 380 "unfiltered" questions (expected to be difficult but not tested against the model).
Dataset metadata available: Yes
Dataset metadata detail: category, filtered/unfiltered
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: Simple Means, the percentage of questions answered correctly.
Metric subscores: Yes
Metric subscores detail: category, filtered/unfiltered
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: They validate their reference answers through independent validators who disagreed with only 7% of the authors' judgments
Taxonomy: Alignment / Alignment
New taxonomy: General Capability / Alignment