Title: Head-to-Tail: How knowledgeable are LLMs? aka will LLMs replace knowledge graphs?
Benchmark: Head-to-Tail
Summary: Head‑to‑Tail introduces an 18 k‑example question–answer benchmark that probes how much factual knowledge large language models really store. Questions are grouped by the popularity of the subject entity (head, torso, tail) and span four domains (IMDb movies, Goodreads books, academic entities from MAG and DBLP, and DBpedia open‑domain facts). Across 16 public LLMs, even GPT‑4 answers only ≈ 31 % correctly overall, and performance drops sharply from head to tail entities—indicating that current LLMs are far from replacing structured knowledge graphs.
Contribution: - Creates a popularity‑stratified benchmark (18 171 QA pairs, 423 templates) that covers both head and long‑tail facts.
- Provides an automated evaluation protocol with three complementary metrics: accuracy, hallucination rate, missing‑answer rate. and both LLM‑judged and rule‑based scoring.
- Empirically evaluates 16 LLMs, showing limited factual coverage, little benefit from larger model size, and only modest gains from instruction tuning.
- Suggests a dual‑representation future where symbolic KGs retain long‑tail facts while LLMs encode only high‑popularity knowledge.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: factual knowledge, hallucination resistance
Phenomenon defined by authors: Yes
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Single‑hop factual QA: given a templated natural‑language question, output the short textual answer or “unsure.”
Task item definition: One item consists of four fields: the target entity (or its ID), the template ID, the fully instantiated question, and the reference answer(s)
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: Questions are produced by (1) manually/LLM‑authored templates and (2) automated sampling of entities and triples from five third‑party KGs (IMDb, Goodreads, MAG, DBLP, DBpedia). The scripts deterministically fill templates with KG facts; there is no human annotation beyond the initial template curation.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 18,171
Dataset metadata available: Yes
Dataset metadata detail: domain, popularity bucket
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric statistics: They report simple percentage means, and compute Spearman’s ρ and Pearson’s r between rule‑ and LLM‑based scores to show consistency; and give a 98 % human‑vs‑ChatGPT agreement figure (percent agreement, not kappa).
Metric subscores: Yes
Metric subscores detail: Popularity buckets (head/torso/tail) and per‑domain subscores
Metric access: Outputs alone
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: For the LLM-based metrics, authors validate scoring with 98% human agreement and correlation analysis.
Taxonomy: Language Modelling / Hallucination
New taxonomy: General Capability / Hallucination