Title: RepLiQA: A question-answering dataset for benchmarking LLMs on unseen reference content
Benchmark: RepLiQA
Summary: REPLIQA introduces a five‑split, entirely synthetic question‑answering benchmark (~90 K QA pairs over ~18 K reference docs) whose documents, people, places and events do not exist on the public web.  Because the knowledge is novel, the dataset isolates a model’s reading‑with‑context ability from any memorised pre‑training knowledge.  The paper also reports a large‑scale benchmark of 18 production LLMs, showing sharp performance drops relative to TRIVIAQA and highlighting that larger models often lean on memory rather than reading.
Contribution: - Novel unseen‑knowledge dataset – REPLIQA: ~90 K QA pairs drawn from ~18 K human‑written, fictional reference documents spanning 17 topical categories, explicitly designed so no existing LLM could have encountered the facts during training.
- Large‑scale benchmark analysis – Comprehensive evaluation of 18 state‑of‑the‑art LLMs reveals heavy dependence on memorised knowledge, mixed scaling trends, and varied refusal behaviour when answers are absent.
- Data‑curation insights – Detailed discussion of limitations and practical hurdles when outsourcing synthetic NLP data creation, highlighting quality‑control pitfalls and guidance for future benchmark builders.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Retrieval-augmented Question Answering, Reading Comprehension
Phenomenon defined by authors: Yes
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Given a ~1,000‑word synthetic document and a question, the model must produce a short answer drawn from the document or output UNANSWERABLE when the answer is not present.
Task item definition: Each item in REPLIQA consists of a fictional ~1,000-word reference document, a unique question grounded in that document, a short ground-truth answer (or an “UNANSWERABLE” tag), and the supporting paragraph ("long answer") from which the short answer is derived. The data is structured with metadata fields including document ID, topic, path, and extracted text.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Crowd-sourced task examples (e.g. Prolific-created tasks)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 89,770 QA pairs, and 17,954 docs
Dataset metadata available: Yes
Dataset metadata detail: topic label
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Mean, recall, F1 with standard error comparisons
Metric subscores: No
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: The authors address the validity of REPLIQA by analyzing potential data leakage, running ablation experiments where models are prompted without context to verify reliance on memorized knowledge, and outlining known limitations such as possible annotator bias, incomplete reproducibility due to evolving LLM APIs, and quality control challenges in outsourced annotation.
Taxonomy: Retrieval / 
New taxonomy: General Application / Document Retrieval
Procedural notes: Five‑split release schedule to delay training‑set contamination.