Title: A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia
Benchmark: Fakepedia
Summary: The paper introduces Fakepedia, a large synthetic dataset of counter‑factual Wikipedia‑style paragraphs that intentionally contradict models’ stored factual knowledge.  Using this dataset, the authors benchmark several LLMs on their ability to ground answers in the prompt rather than in parametric memory and propose Masked Grouped Causal Tracing (MGCT) which is a fast, robust causal‑intervention method to reveal the internal computations that differentiate grounded from ungrounded responses.
Contribution:     - Creation of the Fakepedia‑base (≈21 k items) and Fakepedia‑MH (multi‑hop) datasets
    - Descriptive grounding benchmark across nine open‑ and closed‑source LLMs
    - MGCT, a grouped‑state extension of causal tracing that gives a 30‑50x speed‑up
    - Empirical findings: grounding is distributed, ungrounding is dominated by a few MLPs, and a simple XGBoost on MGCT features detects ungrounded replies with ≈93% accuracy.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Contextual Grounding
Phenomenon defined by authors: Yes
Phenomenon definition: A factual answer is the object of a true fact triplet, while a grounded answer is the object triplet logically consistent with the information in the context of the prompt. Factuality pertains to the model’s encoded knowledge and its ability to retrieve it, whereas grounding involves the model’s capacity to adapt to its context and reason about new information.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Given a prompt containing a counter‑factual paragraph, the model must supply the object that the paragraph implies (either by generating the next token or selecting from two options).
Task item definition: One JSON row gives a subject, relation, counter‑factual object, query string, and the generated paragraph (plus optional intermediate paragraph for multi‑hop).
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Triplets selected from ParaRel where GPT‑2‑XL was confident, then paragraphs were generated “from scratch” by an LLM and filtered/edited by the authors.
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Not meant to mirror real user queries but to produce a controlled clash between memory and context.
Dataset size: Fakepedia‑base: 21,308 samples; Fakepedia‑MH: 21,308 samples
Dataset metadata available: No
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number)
Response format detail: MCQ has exactly two choices; in generation setting, the next token must equal the counter‑factual object to count as grounded.
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Mean; authors additionally report t‑tests for MGCT effect differences and classification accuracy for the detector.
Metric subscores: Yes
Metric subscores detail: Yes – reported separately for Fakepedia‑base vs. Fakepedia‑MH and with‑instruction vs. without‑instruction
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Grounding / 
New taxonomy: General Capability / Grounding