Title: FELM: Benchmarking Factuality Evaluation of LLMs
Benchmark: FELM
Summary: FELM is a meta‑benchmark that measures how well factuality evaluators (usually LLM‑based) can spot factual errors in long‑form answers produced by ChatGPT.
It contains 817 prompts spanning five domains (world knowledge, science/tech, writing & recommendation, math, reasoning). The ChatGPT answers are split into 3,948 text‑segments; each segment is human‑labelled as correct or incorrect and, if incorrect, annotated with an error‑type, explanation and supporting / contradicting reference links.​
Contribution: Main contribution – First fine‑grained, multi‑domain benchmark for evaluating the evaluators; provides segment‑level labels, error taxonomy and references, and reports strong baselines showing that even GPT‑4 struggles without retrieval help.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Ability to detect factual errors in LLM‑generated text
Phenomenon defined by authors: Yes
Phenomenon definition: Factuality in text generation systems generally refers to whether the synthetic text contains any factual errors or not. These errors can take various forms, such as an incorrect entity, a fabricated paper reference, a misleading scientific claim, unlogical reasoning, and incorrect mathematical calculations.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Given a prompt and each ChatGPT response segment, predict whether the segment is factually correct and, optionally, the error‑type and references.
Task item definition: A single row consists of {prompt, full ChatGPT answer, list of segments, gold label(s), error‑type, explanation, reference URLs}.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Prompts pulled from Quora/Twitter/online blogs + standard benchmarks; some written by authors & ChatGPT. All responses are zero‑shot ChatGPT outputs.
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: Partial real task – mirrors real‑world need to vet LLM answers.
Dataset size: 817
Dataset metadata available: Yes
Dataset metadata detail: domain, segment‑id, error‑type, annotator comment, reference links.
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Mean, Precision/Recall/F1, Balanced Accuracy; inter‑annotator agreement (Cohen’s κ / raw %).
Metric subscores: Yes
Metric subscores detail: reported per domain & per segment/response level.
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Each item in the FELM dataset was annotated by two expert annotators, with disagreements resolved through adjudication by a reviewer. To assess the overall quality, the authors conducted a random audit of 100 samples, confirming that all reviewed examples were free of unsafe content and that the reference links used were reliable.
Taxonomy: Language Modelling / Hallucination
New taxonomy: General Capability / Hallucination