Title: EpiK-Eval: Evaluation for Language Models as Epistemic Models
Benchmark: EpiK-Eval
Summary: EpiK‑Eval is a synthetic QA benchmark that tests whether language models can consolidate facts that are scattered across multiple training documents, rather than stored inside a single context window. The authors generate 18 templated story‑based tasks (counting, temporal, causal, etc.), create both unsegmented and segmented versions of each story, fine‑tune several LLMs on each setting, and compare performance.
Contribution: Introduces the first controlled testbed for “epistemic” knowledge‑state consistency; shows large gaps and higher hallucination rates when models must integrate knowledge over separate documents; releases code/data on GitHub.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Knowledge consolidation & consistency across documents
Phenomenon defined by authors: Yes
Phenomenon definition: The paper defines the phenomenon as a model’s ability to consolidate knowledge spread across multiple observations into a single, consistent internal knowledge state, rather than treating facts independently. This epistemic behavior distinguishes Type II systems (integrative) from Type I systems (fragmented memory).
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Given a templated story (or its sentence‑segments), answer a question requiring integration of multiple facts, and reproduce the supporting facts verbatim.
Task item definition: A single row consists of {story ID, story text or segmented sentence, task ID, question, reference answer}.
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Procedurally-generated task examples (e.g. Creating instances from a template)
Task source detail: All stories/questions are generated from deterministic templates with random name/activity/day slots.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Dataset size: 1800
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: Yes
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Authors argue validity by contrasting segmented vs unsegmented conditions, measuring hallucinations, and encrypting data to avoid pre‑training leakage.
Taxonomy: Retrieval / 
New taxonomy: General Application / Document Retrieval