Critical Confabulation: Can LLMs Hallucinate for Social Good?
Authors: Peiqi Sui
Eamon Duede
Hoyt Long
Richard Jean So
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
AI for Social Good
Hallucination and Confabulation
Narrative Modeling
Data Contamination and Memorization
Computational Creativity
Evidence-Grounded Generation
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We introduce critical confabulation (evidence-constrained LLM hallucination for reparative storytelling) and evaluate LLMs in this task via open-ended timeline reconstruction; we show that controlled hallucinations have unique social affordances.

Abstract: LLMs hallucinate, yet some confabulations can have social affordances if carefully bounded. We propose critical confabulation (inspired by critical fabulation from literary and social theory), the use of LLM hallucinations to "fill-in-the-gap'' for omissions in archives due to social and political inequality, and reconstruct divergent yet evidence-bound narratives for history's ``hidden figures''. We simulate these gaps with an open-ended narrative cloze task: asking LLMs to generate a masked event in a character-centric timeline sourced from a novel corpus of unpublished texts. We evaluate audited (for data contamination), fully-open models (the OLMo-2 family) and unaudited open-weight and proprietary baselines under a range of prompts designed to elicit controlled and useful hallucinations. Our findings validate LLMs' foundational narrative understanding capabilities to perform critical confabulation, and show how controlled and well-specified hallucinations can support LLM applications for knowledge production without collapsing speculation into a lack of historical accuracy and fidelity.

Published as a conference paper at ICLR 2026
CRITICAL CONFABULATION: CAN LLMS HALLUCINATE FOR SOCIAL GOOD?
Peiqi Sui
Eamon Duede Purdue University
Hoyt Long
Richard Jean So Duke University
McGill University
University of Chicago
ABSTRACT
LLMs hallucinate, yet some confabulations can have social affordances if carefully bounded. We propose critical confabulation (inspired by critical fabulation from literary and social theory), the use of LLM hallucinations to "fill-in-the-gee omissions in archives due social and political inequality, and reconstruct diver- gent yet evidence-boud these gaps with an open-ended narrative cloze task: asking LLMs to generate a masked event in a character-centric timeline sourced from novel corpus of unpublished texts. We evaluate audited (for data contamination), fully-open mod- els (the OLMo-2 family) and unaudited open-weight and proprietary baselines under a range of prompts designed to elicit controlled and useful hallucinations. Our findings validate LLMs' foundational narrative understanding capabilities to perform critical confabulation, and show how controlled and well-specified hallucinations support I.LM applications for knowledge production without collapsing speculation into a lack of historical accuracy and fidelity.
for
narratives for history's "hidden figures". We simulate
"As an emblematic figure the enslaved woman Atlantic world, Venus makes plain the convergence of terror and pleasure in the lihidinal economy of as fully as possible the conditions that determine the appearance of Venus and that dictate her silence."
slavery... [critical fabulation] attempts to redress it hy describing
Saidiya Hartman, 2008
BACKGROUND
Large language models (LLMs) are prone to hallucinate, generating "plausible yet nonfactual" outputs (Huang et al. 2025). While typically treated as failure mode, recent studies show that some hallucinations could in fact be valuable (Jiang et al. 2024 Hu et al. 2024 Taavekktworacchii et al.. 2024), especially subset termed confabulations information with self-consistent stories that bear close verisimilitude with reality (Sui et al. 2024). Confabulated texts are more narrative-rich and better match the patterns of human storytelling, vital communicative and cognitive resource for sense-making (Herman 2013). Thus, principled
narrative-drive driven tendency to "fill in" missing
Archival Records
(BTWC)
Contamination Audit
Data
Ground Truth Extraction
Tyrone Guytan's timeline potiomee
offices inaclard in the ki Tymee Eigton, sparking further protents and cals for
gaadd Juy dectieed Inditt
bark, sikingg 1f-gaancdd Wed Ouiladd home.
OLMe 2-328 Coeine Simdarity $3.87
Knowledge Contaminatio Audit
88888 Hiddeen Figures' g., Tyrone Guyton
1974 The drese polcemen who
Ie Alamec Countyy Grand Juyy refused to indiet the officess mokied in Tyrono Gustoilo death.
coneon prompiing community
1915 13 02: The Black Parcher Party bunched a reall comparg againt nA uowel Jonser cidig tis refuall to
8888888 Name Candidates
NER
prosocnt Tp Tyene Suston's Miles,
GFT-6 Codre Smileity 24.05
Figure 1: Critical confabulation an open-ended narrative-cloze task for LLMs.

---

Published as a conference paper at ICLR 2026
trade-off between strict factuality and alignment with the human behavior of storytelling may better support application settings that require interacting with LLMs as cultural agents instead of merely using them as tools.
If grounded in cultural or historical contexts, can a carefully controlled form of confabulation be harnessed for social good and useful knowledge production? Prior research shows this approach is feasible for Al applications across a range of fields, including computational creativity (Jiang
et al. 2024), narrative exposure therapy (Lely al. 2019; Li (023a), digital storytelling for cultural heritage (Shim et al., 2024), co-creative museum exhibitions (Fu et al., 2024), and bias mitigation (Kim et al. 2024)) importantly, these useful confabulations have clearly defined bounds that anchor them in existing evidence and differentiate them from unconstrained and underspecified
hallucinations where the model must first speculate where the missing information is.
In the academic field of African American studies, Hartman (2008) introduces critical fabulation, the practice of using speculative storytelling to rectify omissions in historical archives due to social and political inequality, which has since become field-defining methodology adopted in wide range humanistic historical case studies (Fuentes 2016 Hartman 2019) Hannah-Jones et al. 2021 Morgan 202T). Many lacunae in historical records are the result of institutionalized violence and state and social repression that, like the Atlantic slave trade and its afterlives, has erased marginalized
stories by structuring the standards for what is preserved forgotten (Hartman 1997; 2007). The
harms of these lacunae are well-documented across the humanities and social sciences, described in postcolonial studies the "'pistemic violence" of identity and knowledge erasure (Spivak) 1988), in philosophy as the "hermeneutic injustice" of depriving the stories that allow one to make sense of their own lives (Fricker 20007) and in anthropology as "archival silence" (Trouillot, 1995) and the "denial of coevalness" (Fabian 2002).
Hartman argues that an empiricist and fact-based approach to historiography is problematic for addressing the long tail of archives-the the "hidden figures" who were never afforded the privilege of historical documentation in the first place. Rather, treating record availability as proxy for truth effectively overfits to the fraught criteria of what survives in the archive, a confounder that would further silence the archive's 's hidden figures and their divergent stories: gaps in the archive emerge when the standard of factual "rigor" privileges evidence-rich authorized accounts of history over necessarily sparsely documented stories that would be dismissed as insufficient.
As form of socially reparative practice, critical fabulation aims to recover and reconstruct coherent stories from fragmentary records and humanize the hidden figures they omit, using speculative storytelling to read against-the-grant of the archives that have systemically excluded them. This often involves adopting marginalized points-of-view that contest and unsettle the authorized accounts of history, to better understand their limitations and interrogate "what might have happened". Crucially, the practice neither collapses speculation into facts nor licenses the invention of new historical realities; instead, it clearly defines a system of narrative ethics that bounds the scope of critical fabulation to high-fidelity counter-narratives grounded in the lacuna's 's context.
Drawing inspiration from how Hartman's framework augments low-resource settings with evidence- grounded human storytelling, we propose to leverage LLMs' extant behavior of confabulation to operationalize critical fabulation as critical confabulation: an AI storytelling workflow (Figure I for scaling the recovery of divergent yet evidence- briunnce-bound narratives with well-documented affordances for useful knowledge production. Although critical fabulation widely regarded as effective for enhancing the completeness of historical knowledge, its application has been limited by the vastness reconstruction of lacuna is labor-intensive because accountability to
of archival corpora-fai extant historical evidence requires meticulous parsing of dense and granular records. This tension motivates LLM--nnbled approaches to critical fabulation that can 1) identify potential lacunae in coneetcccoonttrineee possibilities (rather than assert
large-scale archives, and 2) simulate multiple, single monolithic truth) to aid human scholars in the augmentation of historical knowledge. L.LM- based methods build on prior work in digital humanities that demonstrates the compatibility between critical fabulation and computational techniques (Klein) 2013 Johnson 2018 Risam 2019; Risam & Josephs 102
In this paper, we operationalize critical confabulation as a narrative cloze task (Chambers & Jurafsky
2008). and use it as a proxy for fragmentary records to assess a wide range of LLMs. To build a realistic setting for evaluating the recovery of erased histories, we leverage the "Black Writing and
2

---

Published as a conference paper at ICLR 2026
BTWC
UNSEEN
"Folktale" & "Story"
Count
Document count
20686
4499
16187
4753
Hidden figures
156
Total length (tokens)
6m.1m
30.2m
30.8m
11.5m
Events (median) 6[4-9]
(a)
(b)
Table 1: Dataset Statistics: (a) The BTWC corpus B, (b) character-level ground truth timelines T(n)
Thought Collection" (BWTC), a digital archive of mostly unpublished, primary sources related to black intellectual history and storytelling, as ground truth for "lost" history unknown the models. Our work advances the following research questions:
R1: Can LLMs confabulate plausible candidates for archival omissions? We find that while critical confabulation remains a very challenging task for state-of-the-an LLMs, most models can propose some plausible, evidence- evidence-bounded candidates for masked narratives. This capability positions them as viable co-creators to support humans in the practice of critical fabulation, and useful research tools for advancing digital humanities scholarship on archival silence's call for "technology of recovery" (Gallon 2016) that aims to "animat[e] the mysteries of the past" (Klein 2013). The effectiivnness of historical confabulations could also broaden the cultural scope and social affordances of Al-assisted text restoration beyond the current domain-specific applications like ancient epigraphic
reconstruction (Assael et af.) [022] 2025), diacritization (Gershuni & Pinter 2022 Kharsa et al. 2024), and traffic surveillance (Gong et 2024 Pan al. 2024).
R2: What conditions shape, constrain, and enhance these confabulations? We find that the effectiveness of event reconstruction is highly prompt-sensitive and improves with light supervision of specifying event type. Performance further varies on the event's 's structural attributes (length, position timeline, timeline length), and most models perform best on events limited biographical information and worst on events concerning intellectual history.
R3: What makes LLM good critical confabulation? Prompts that explicitly steer models towards hallucination reliably improve confabulation performance. GPT-5-CHH is the overall leader, the only model to exceed 50% on most prompts and peaking 59.7%. While larger variants usually outperform smaller ones within family, smaller models can punch above their weight on this task: OLMo-2-7B attains the second-highes score under one prompt setting, and QWEN3-4B achieves among the strongest overall results.
TASK SETUP
Problem view. We divide the full process of critical confabulation (Figure 4) into known unknowns (lacuna reconstruction) and the more challenging unknown unknowns (lacuna detection). In this work, we cast critical confabulation unknowns of historical archives.
open-ended narrative cloze task to first target the known
Procedure. For hidden figure n with a timeline (sourced from pertinent archival records B(n))
T(n) ((1,ee). ((m(n):
each element consists of timestamp t; and one seeeseenence event e;, We simulate lacuna in historical knowledge by replacing one event cm with the literal token [MASK]:
C(n, m)) mask (Task(((m),m))) ((t1,e)), (tm, [MASK]), (fm(())) Cm(n)))-
Given the timeline fragments C(n, m) and a fixed instruction prompt, a model fo must reconstruct the masked event Cm Let simamb (ém, Cm) denote the embedding- bbaed based semantic similarity between
é., and e. A reconstruction is counted as "correct" (sufficiently similar) simemb (ém: cm 2 e, where is model-specifie, tunable threshold.
3

---

Published as a conference paper at ICLR 2026
3 DATASET
3.1 DATASET STATISTICS
Our dataset for establishing a ground truth of hidden history is the "Black Writing and Thought Collection" (BWTC). This collection was curated by the ARTFL Project at the University of Chicago in collaboration with Alexander Street Press and is used with the permission of ARTFL. It is comprised of 20,686 documents dating from the early 1700s to modern times, and includes corpora of dramatic writing, fiction and folktales, and non-fiction works such as interviews, journal articles, speeches, essays, pamphlets, and letters. Although much of the material in the collection was previously published in print and includes writings by prominent black writers and intellectuals, a sizable portion is made up of previously inaccessible material that has not been widely distributed, including interviews and trial transcripts. We choose BWTC partly for this reason, as it ensures that meaningful percentage of the collection is not included in the various book datasets used train foundation models. Meanwhile, it is also well established that historical documents related to black history and culture have generally been repre-eepresented in digital collections (Marcus & Carlson 2018 Flinn [2007] [Pescod] 2017), making BWTC an ideal dataset to evaluate LLMs' foundational narrative capabilities for critical confabulation.
3.2 HAVE LLMS SEEN OUR UNPUBLISHED DATA?
Deng al. 2024). In our case, contamination is especially problematic since the presence of our unpublished dataset in pretraining or instruction-tumit prior access to this history, effectively collapsing the task of memorization. We focus our main experiments on fully open models with publicly released training data (e.g., the OLMO family), due to the absence of reliable methods for model-agnostic pretraining data detection. While recent work in this area reports optimistic results for popular approaches like membershhip-inferrnce attacks (MIAs) (Shi et al. 2024 Zhang al. 2024), subsequent studies challenge their robustness, finding that oft-oo--isttrbuutt MIAS often perform no better than random guessing (Maini et al. 2024 Duan et al. 2024).
Data contamination is problem for LLM evaluations (Golchin & Surdeanu, 2024
corpora would violate the assumption that LLMs have no confabulation into
Accordingly. we restrict our analysis to OLMo-2, the state-of-the-an fully open model whose public training data enable us to directly cross-search for our base corpora. To thoroughly vet OL.o-2 for our following experiments, we conduct a two-stage data audit on the sentence-level:
To identify potential pretraining contamination in OLMo-2, we first perform an exact, sentence-Tevel substring search that scans every sentence from the BWTC corpus B against the entire publicly released OLMo-2 training set O (Table 1). For each document d E B, we segment d into sentences S(d) = {81...... }. We then implement a classic Boyer-Moore substring matcher (Algorithm I for each query sentence E S(d) against each candidate document in O. The Boyer-Moore algorithm is selected for its efficiency and sublinear average-case runtime (Boyer & Moore 1977). which is crucial at the scale of B and O. For a document we compute the raw match count
String Search
matches(d) = Σ Σ BM(x,
BEO (E)(d)
where BM s) E (0.1) }} indicate whether s appears contiguously in x. We designate d with >= 100 matches as SEEN by OLMo-2. Overall, 21% of all documents in B are flagged SEEN.
Cosine Similarity Distributions. As a sanity-check for Algorithm [ SEEN and UNSEEN labels, we take inspiration from Oren et al. (2024) 's approach to detect contamination via model behavioral comparisons on input permutations, and perform an analogous probe on OLMo-2 in a next-sentence completion setting. Intuitively, we expect a memorization-induce performance gap OLMo-2's continuations of sequences from Beee should be more similar to their hold-out gold text than those from Busseon theirs such that Vi E W : meanse [sim, (d) > mean simm,d)), where W {1, (1...., K} indicates continuation window of K sentences, and sim;(d) E [0, 1] denotes fixed scalar summary of the lexical similarity (e.g., a position- -verrggd similarity over W).
Wee perform our search instead of using popular tools like WIMBD Elazar et al. 2024 because it does include the indices for OL-2's's training data.
4

---

Published as a conference paper at ICLR 2026
Soon
Moan Cosene berilarly Scares
Unseen
Seen Lineen
02
00
23
83
0.4 similarity
05
2.
0.7
Average similarity across positions
0.5
(h)
(c)
Figure 2: Behavioral probe for contamination. Visualizing the cosine similarity decay through (a)
position-wist mean, (b) aggregate ECDF, (c) distribution across positions.
To validate this hypothesis, construct 1,000-document subset (500 randomly sampled from Bwen: 500 from Bunseen). For each document, we take the first 20 sentences as context and define five non-overlapping gold continuation windows of five sentences each, Pi.....P5 We then gen- erate continuations with OLMo-2, first given the 20-sentence context and subsequently with the addition of its own prior continuations, producing Pı..... pss. For sim(d), we compute position- wise cosine similarity by fitting pairwise TF-IDF vectorizer on each pair (P:,Pt), and aggregate the scores by positions E {1,...,5). Formally, Vi E (1,...,5) 55} mean (iim(p; (iinn Pc)] > Be [sim [sim (P:, p.)]. Ps ~ (PP( (d)())) where æ denotes greedy decoding for repro- ducibility, and are the parameters of OLMO-2.
The results support the hypothesis: overall, SEEN continuations exhibit consistently higher cosine similarity to the ground truth than UNSeaN mean similarity 0.3009 vs. 0.2782) (Fig. 2 This is consistent with memorization -induced advantage: when the audit labels indicate prior exposure, OLMo-2's continuations hew closer to the held-out text. The monotone decay of the gap from p1-xp5 accords with error accumulation from memorization strongest immediately after the observed context. While the absolute effects similarity modest, the signal is directionally robust across diverse set of statistical tests (parametric, nonparametric, permutation) and survives multiplicity control.
generations proceed, suggesting any advantage
Following the evidence that supports the validity of SEEN/UNSEEN labels, adopt conservative document filtering policy: we remove all documents marked as SEEN from the dataset and conduct all subsequent analyses exclusively on Bunsen- This filtering eliminates 21% of documents and likely discards some false positive documents from the string search, but it ensures that the remaining data functions as reliable ground-truth proxy for unseen history and prevents spurious gains from memorization. Additionally, we use the dataset's metadata to filter out documents that are not in English; we also exclude documents under the genre "Folktale" or "Story", since their fictional nature makes them impertinent our focus on historical confabulation.
4 GROUND TRUTH CONSTRUCTION
We leverage long-tail name candidates from Busseen that are rare absent in O as proxies for Hartman's ' "hidden figures," and, for each figure, we construct event-centric timelines from their pertinent documents the ground truth for the narrative cloze evaluation. In other words, first extract the unknown knowns from extant archives as basis for the subsequent evaluation of the known unknowns.
4.1 WHO ARE THE HIDDEN FIGURES OF OUR DATASET?
Even after removing documents flagged as SEEN by our sentence-level audit, a model's parametric knowledge could still serve priors for specific entities and names that do not appear in B (Nast ctt al. 2025). We therefore conduct a more robust search for "hidden figures" that are both in B Bussee and mostly absent from O, using Aho & Corasick 1975).
5

---

Published as a conference paper at ICLR 2026
Name Candidates. We curate list N by extracting up the top 10,000 unique PERSON names from museen (NLTK), then targeting the long tail by only keeping names within a lower frequency < 51. For downstream timeline quality, we also require that name appear in at least three documents.
automaton M(N*) (Algorithm 2) and stream each document O through M, counting substring matches for both single- and multi-token names. For each n E N we accumulate global count c(n) across O. This avoids per-name passes and yields linear-time scanning in corpus size. To curb substring artifacts (e.g., "Ann" in "Annex"), we cache short surrounding snippets for low-frequency hits and manually spot-check sample of those contexts. Using the same conservative threshold as our string-searde audit, we declare SEEN-IN-ON if c(n) 2 100; otherwise we treat n as an UNSEEN (n = 322).
Cho-Corasick Search. We build a multi mullti-ptterr
Manual Filtering. To further clean the name list, we conduct manual filtering to remove names that only have trivial appearances in their documents (e.g., mentioned in only in passing, co-references, near-duplicates). This yields a final V unseen of 156 names.
4.2 GROUND TRUTH STORYLINE EXTRACTION
For each hidden figure E (V* umseen), aggregate all documents in Bunseen that mention n, {di....., dx.. J. and pass them as a single prompt into a long-context extractor (GPT-03) under a strictly source-bounded instruction. The goal to produce chronological, evidence-linked timeline T(n) (E1..... Cm(n)), where each event summarizes distinet, attributable occurrence involving n, ordered by time and accompanied by explicit citations to the supporting documents.
We purposely avoid setting hard constraints on the number of events or their granularity, al- lowing the extractor to adapt to document density while covering the full scope of activi- ties evidenced in the sources. We find that as long as the prompt asks 03 to yield at least one event for every non-trivial, attributable mention cluster of n, the output timelines mostly exhibit satisfactory completeness (see Section 4.3 Each event is emitted as one character- focused and active-voice sentence (S 30 words). Events are typed with exactly one label from (AGENTIVE, RELATIONAL, OBSERVATIONALE COGNITIVE, ROLE}, defined in an event schema pro- vided in the prompt. Conflicts across documents are resolved by prioritizing the statement with the strongest supporting evidence, while recording brief note that we use for subsequent manual spot checks. We provide the full prompt, containing the instructions, source-bounding constraints, and event schema, in Appendix A
4.3 GROUND TRUTH VALIDATION
Automatic. We validate that extracted timelines T(n) are internally coherent and informa- tion-bearing by asking whether small, controlled perturbations of the input lead to predictable changes in reconstruction difficulty (measured through cosine similarity the ground truth). Con- cretely, we vary the narrative-clos cloze window in two setting
Partial cloze. Given an event é (4,...,, (w) with w tokens, we only mask a subset of its tokens t (rather than the full event); we iteratively reveal a left prefix of length w w-m and mask the remaining m tokens with [MASKED], for m = w 1. The expected pattern monotonicity: as less of the event is masked, reconstructions should become more similar to ground truth. The observed curve (relation between similarity and percentage of words masked) is strongly increasing from near-full masks to near-zero masks, with only minor local noise (Fig. 3a.. This indicates that most events in T(n) carry substantive content rather than template artifacts that have been observed to decrease the quality of LLM summarizations (Holtzman al., 2020 Ravaut et al. 2024
n-gram cloze. We slide a window of size A E (2,3) over each timeline, masking exactly k consecutive events and requiring the model to output one-line reconstructions in order. Each event in the reconstructed window is scored individually against its corresponding ground truth, then we average over all events in the window to get the n-gram similarity score for a fixed event position. The expectation is that cosine similarity would be inversely related to k: sim gram 2 sim gram 2 simy gra-- The results satisfy this ordering across timelines:
"Theee experiments the default settings Section 5.1 (OLMo-2-33R with the base instruction prompt).

---

Published as a conference paper at ICLR 2026
87.5
Mean 15% bins)
Adjacent 5% bin means
cosine similarity
similarity
77.5
75.0
Uni gram
725
Bigram
Ti-gram
100
80
60
40
of words masked
(a) Partial-event cloze.
(b) re-gram cloze.
Figure Ground-truth validation probes. (a) Partial-event cloze; (b) n-gram cloze.
tri- gram masking yields the lowest means overall, bi-gram is intermediate, and uni-gram the highest, with degradation most pronounced for context-dependent events (Fig. (b)..3b). This pattern is consistent with compositional dependencies among adjacent events and supports that most timelines in ((n) encode meaningful event-level narrative structure.
Both probes produce directionally consistent outcomes that align with expectations: the similarity increases smoothly as more of the clozed event is revealed, and masking longer contiguous spans consistently lowers similarity. Together, these results support our nource-bounded timelines as reliable ground truth for narrative cloze, capturing event-level meaning and short-range narrative dependencies rather than artifacts of formatting or extraction.
Manual. To further assess accuracy, three annotators (domain experts in African American studies) independently label event correctness and event-type correctness for each character timeline in validation set (n=30). We observe 98.3% correct event summaries and 93.3% correct event types, with overall pairwise agreement of a = 0.791 (Krippendorff's alpha). The annotators also review timeline completeness and find that 66.7% include the most significant events associated with their character.
5 EXPERIMENTS
5.1 NARRATIVE CLOZZE
We perform the narrative cloze test on all 156 ground truth storylines, following the setup in Section
2). To mitigate I.MSs widely reported difficulties with temporal ordering (Chu al. 2024 Fatemi ct al. 2025). the prompt specifies the date range of the masked event. We also experiment with light structural supervision by including the type of the masked event as hint.
Model Selection. We evaluate all sizes of the audited OLMo-2 family (IB, 7B, 13B, 32B). For context, we also include a broad of unaudited open-weight (Qwen, Gemma, Llama, Phi, Mistral) and proprictary models (GPT-40, GPT-5-CHAR unaudited models identified in the latest OLMo-2 report as peers with comparable performance on general-domal benchmarks. We focus on the instruction-tuned variants of the above models in a zero-shot setting and not employ Chain-of-though prompting, which primarily benefits mathematical and symbolic reasoning (Sprague et al.] 2025) but shows very limited improvements on narrative tasks (Chen et al. 2024) and can even degrade performance (Su et al. 2024).
to situate OLMo-2's performance. We prioritize
Prompts. A task-specific instruction prompt is iteratively determined and uniformly applied across all models. We select a task-specific instruction prompt via iterative refinement and apply it uniformly across all models. On top of this base prompt. also experiment with adding six system or instruction templates from prior work that intentionally or incidentally steer model behavior to

---

Published as a conference paper at ICLR 2026
No Event_ Event_Type
With Event_Type
NS
EA
DS
HE
HH
NS
EA
HE
HH
OLMo-2-1B
24.7
24.9
24.5
23.1
28.9
22.6
26.2
29.2
25.7
27.0
25.8
33.0
15.9
30.2
01-M--2-B
33.8
34.7
34.0
49.1
28.7
20.2
34.7
38.5
42.1
39.0
58.8
33.7
24.5
42.6
OLMo-2-13B
37.1
41.4
40.6
44.7
41.6
31.0
37.8
45.1
45.3
40.6
47.4
46.5
36.8
47.6
DLMo-2--20D
44.9
40.4
46.0
37.3
32.5
36.3
34.7
48.4
43.7
50.8
34.3
36.1
39.8
32.7
Qwen2.5-7BB
43.0
44.3
43.8
40.0
45.0
33.6
41.4
45.1
46.7
48.0
41.8
44.0
38.0
43.2
Qwen2.5-14B
37.2
37.0
38.6
38.0
40.5
23.3
40.9
47.1
43.3
44.2
37.7
43.8
26.5
46.3
Qwen2.5-32B
40.8
46.7
44.3
41.7
44.6
25.7
42.9
45.3
47.8
46.0
41.7
48.2
32.3
49.6
Qwen3-4B
44.6
43.9
47.9
55.0
44.1
32.0
41.6
46.5
45.8
48.0
55.88
45.0
39.0
43.9
Qwen3-4B-0725
47.1
46.7
50.9
40.7
48.0
11.4
47.3
50.2
50.7
51.6
42.0
48.2
16.9
53.8
Llama-3.1-8P
35.8
41.6
40.6
37.6
43.4
34.2
39.4
43.4
47.9
46.0
37.4
43.3
35.0
45.9
Mistral-Small-221
39.6
43.3
40.1
38.1
45.2
18.5
45.5
42.6
46.6
45.0
37.3
44.0
23.9
44.9
gemma-2-99
39.6
29.3
39.4
2.2
44.4
23.8
37.2
41.9
31.2
40.6
2.6
44.7
25.7
39.6
gemma-2-27b
40.1
36.7
39.4
5.0
46.6
21.9
36.5
42.1
41.0
42.5
0.6
46.1
22.7
43.0
gemma-3-27b
38.1
39.3
38.3
39.8
42.2
19.2
39.9
40.5
38.0
42.0
37.0
23.7
21.4
40.2
phi-4
44.2
42.0
41.3
38.9
34.1
4.9
44.0
47.0
42.0
48.5
39.2
37.5
9.0
44.9
GPT-4o
42.6
45.5
44.1
42.0
42.1
17.2
38.3
45.4
49.1
47.9
40.0
44.0
26.9
45.0
GPT-4o-mini
43.3
41.9
45.8
32.3
43.3
21.8
42.0
46.9
47.7
34.9
43.7
28.5
44.9
GPT-5-chat
51.0 53.3
56.0
57.4
57.3
35.9
51.5
55.5
56.1
57.1
58.0
59.7
41.7
55.9
Table 2: Performance (Acc) of LLMs narrative cloze. green marks per-prompt bests, blue per-model bests, and underlines indicate where the EVENT TYPE hint fails to improve performance. Critical confabulation remains challenging: few models exceed 50% accuracy, performance highly prompt-sensitive, yet providing hints like EVENT TYPE can lead significant and consistent gains.
increased hallucination creative outputs. For each, possible; otherwise, we adapt the core design to our narrative-cloze setting while preserving the intended behavior. Full templates are provided in the Appendix A.3
the authors' wording verbatim when
Generation Setting. We use deterministic decoding (no sampling; do_ do_sample=False) for the main experiment to ensure reproducibility and comparability across models, following best-practics recommendations for LLM evaluation from recent methodology papers (Blackwell et al. 2024 Song
ct al.
2025 Hochlehnert et al. 2025).
Stochastic Ablation. To assess the effect of sampling stochasticity on critical confabulation, we con- duct the narrative cloze experiment with three different temperature settings: low (temperature=0.2) medium (temperature=011 and high (temperatur==1.2). We fix top_p 0.9 and use a shared random seed across all models and prompts.
5.2 EVALUATION
We implement an effective measurement of narrative verisimilitude with state-of-the-arr narrative embedding model story-emb (Hatzel & Biemann, 2024), which emphasizes storyline structure rather than topical semantics that could confound our setting. We compute cosine similarity between the story-emb embeddings of generated event én and its ground truth em ; a prediction marked "similar" if simmstory-enl (ém, cm 2 (tunable threshold).
Threshold Tuning. Using the same validation set as Section 4.3, an annotator (an undergraduate history major) labels each event "similar" (1) or "different" (0) to the ground truth (full instructions the Appendix). We use these binary labels to tune an operating threshold e for the cosine similarity scores, by sweeping candidate thresholds on the validation scores and select the value that maximizes macro-F1 to balance classes. The optimal global threshold is e* 73.13, yielding a macro-F1 of 0.805; fix e" for all subsequent evaluations across models and prompts. The high macro-F provides evidence that story-emb distance serves as a reasonable proxy for human judgments of narrative verisimilitude.
8

---

Published as a conference paper at ICLR 2026
6 RESULTS
Table reports accuracy per model and prompt across the two settings. We manually verify that outputs are parseable and cooruction-followiini for nearly all (model, prompt) pairs, with two exceptions: GEMMA-2-9B and 27B under Eccentric Automatic Prompis produce unparseable strings, and PHI-4 under HaluEval often refuses to confabulate. Providing EVENT_ EVENT_TYPE consistently helps across models and prompts, with almost all pairs improving by -2-10 points. The strongest overall performance is from the unaudited GPT-5-CHAN (up to 59.7 with EVENT_Ty: and the LLM-Discussion prompt); OLMo-2-7B closely trails at 58.8 under Eccentric Automatic Prompts, surpassing many larger weight baselines. QWEN3-4B is the best open-weight unaudited model despite its size, and GPT-5-CHAT the only model that exceeds 50% on most prompts. Incidentally, both are also the most recent models in our comparison and the only two newer than the OLMo-2 evaluation window and thus absent from its official baselines. Within most families, larger variants generally outperform smaller ones.
Across prompts, the performance of audited OLMo-2 models is not significantly different from that of their unaudited peers with comparable general-domain performance (average p-value = 0.354; see detailed results Appendix E.3 In other words, we do not observe any significant evidence for an advantage attributable to possible memorization our narrative-cloze test.
Prompt sensitivity is pronounced: Null-Shot is the most reliable high- sesh-peformer, while HaluEval is the weakest; L.M--Dscussion yields the top single result (GPT-5-CHAT), and Eccentric Automatic Prompts is most volatile, attaining top-line performance for some models yet triggering parsing issues for others. These variations suggest that critical confabulation is volatile behavior sensitive to prompt perturbations. Nevertheless, the prompts show directional agreement across models: 72.5% of all prompt-pair comparisons share the same majority direction (two pairs unanimous), with moderate global concordance in prompt orderings (Kendall's I' W = 0.403) and cross- rank correlations (mean Spearman 0.367; 65% of model pairs 0.3, 32% > 0.5, 17% > 0.7).
Table reports the per-model, per prompt results for the stochastic ablation. Performance is largely stable across sampling temperatures, with only a minor disadvantage for more stochastic settings: relative to the deterministic baseline (temperature=f) the low and medium regimes change accuracy by just -0.3 and -0.8 points on average across models and prompts, while the high-temperature setting induces a slightly larger drop- 2233. Model and prompt rankings are almost entirely preserved across temperatures. Within most families (notably OLMo-2), larger variants exhibit larger drops at high temperature than smaller models. Overall, critical confabulation is robust to reasonable changes in decoding stochasticity.
DISCUSSION
Overall, critical confabulation is a feasible yet very challenging task: most models stay below 50% accuracy, with top-line performance approaching 60% under strong prompts. Performance is highly contingent on input structure: models benefit from thematic constraints like EVENT_ TYPE and are strongest on biographical "role" events and weakest on "cognitive"; longer event descriptions help while longer timelines and later positions reduces accuracy. While we do not observe direct evidence for an advantage from potential memorization, there are some behavioral signals that could differentiate audited and unaudited models.
Which types of event are LLMs the best confabulating? Overall, models confabulate "role" descriptions best (44.8%), followed by "relational" (41.4%), "agentive" (37.8%). "observational" (36.2%), with "cognitive" lowest (24.9%). The EVENT_ TYPE hint does not change this ranking, though it disproportionnatlly benefits "role" events (+5.1 points). addition, the audited models are more uniform across event types and 3 out of perform best on "relational", whereas all unaudited models peak on "role", which mostly entails more biographical information that could amplify a possible memorization advantage. The weakness of "cognitive" events (internal states, opinions, or stance shifts) could be because of their lack of anchoring in observable context, making them harder to reconstruct from adjacent events (Appendix E.4)
EEEET__TPE
Does the length of timelines or individual events affect performance? Across all models, longer event descriptions correlate with modestly higher accuracy (p 0.09, p < 0.001), with accuracy

---

Published as a conference paper at ICLR 2026
rising from 0.334 to 0.466 (without EVENT TYPE) and from 0.347 0.500 (with EVENT TYPE) from the shortest to longest quartile. By contrast, longer character timelines are associated with lower character-level accuracy (p -0.173, p 9.05 x 10 39), declining from 0.414 to 0.343 (without EVENT_ and from 0.481 to 0.352 (with EVENT TYPE) as timelines lengthen. Event-type hints consistently improve accuracy but do not remove the negative correlation with timeline length. Overall, the effects are small but robust, suggesting that longer local contexts aid reconstruction while long-context timelines remain challenging for LLMs (Appendix E.5p.
Does event location affect performance? Across models, accuracy is highest for events at the beginning of their timelines (0.45) than those in the middle (0.381) and the end (0.337), a consistent ordering that the EVENT TYPE hint does not alter. Audited models show a smaller begin >edd drop than other models (0.086 vs. 0.12), which is consistent with reduced dependence on potentially memorized, opening-biograpee biography facts prevalent in pretraining data and more uniform reliance on local reasoning across the timeline (Appendix E.
Do models fail on the same cases? Model errors are highly clustered. On the event level, only 2.2% are answered correctly by all 19 evalutaed models, whereas 59.1% of events are missed by at least 10 models and 38.5% by at least 15, with pairwise Jaccard overlaps over error events around 0.6-0.7. On the character level, for 20.1% of the characters all models are wrong on 2 50% of events, and the Jaccard overlap over them is even higher (often > 0.9). These patterns indicate that models do not fail idiosyncratically but instead their errors concentrate on a shared subset of particularly difficult events and characters. While our dataset's metadata do not reveal any obvious explanation for this cluster of challenging cases, understanding the underlying socio-economic positionalities that might overdetermine these systematic failures is an important direction for future work.
CONCLUSION
We introduce the framework of critical confabulation, and cast it as a narrative cloze task for LLMs: filling in missing, evidence-hounded events in historical timelines. Given the scale of most archives, LLMS have the potential to augment the close textual attention of humanities scholars to better address archival silence across larger corpora. Our results show that current models show promise in this task, and careful prompting can clicit some meaningful event reconstructions. The implications are twofold: for NLP, we demonstrate a domain application where hallucinations become a unique and optimizable resource rather than pure defect; for the humanities, we highlight the narrative understanding capabilities of LLMs that make them viable tools for helping scholars probe the latent semantic spaces of large archives to surface unknown unknowns, and then rapidly prototype candidate reconstructions to narrow the bounds of the known unknowns [vvans & Duede (2025). We cxploring wide range of use cases for critical confabulation to support humanistic scholarship and augment human storytelling, towards new Al-enabled methods for studying culture and history that could catalyze paradigm shifts in knowledge production as seen in other disciplines like social science (Lazer et al. 2009) and biology (Jumper et al. 2021).
This work is also preliminary in nature: performance is sensitive to prompt and input structure, and our experiments are limited to one language and cultural tradition. Future work will (1) design more robust and comprehensive evaluations for open-ended narrative outputs, (2) broaden coverage across languages, cultures, and corpora, (3) build ethical safeguards and provenance tracking to ensure faithful event reconstruction and avoid compounding archival violence, and (4) develop training- and inference-time methods that explicitly optimize for well-specified, evidence-constrained confabulation.
HUMANISTIC MISSION STATEMENTE
The broader aim of our work is to address the gap between scholarly perspectives in black studies (e.g.. major scholars in this field such as Saidiya Hartman, Fred Moten, Christina Sharpe, and others.) and public-facing work (c.g., The New York Times historical and journalistic study of US slavery, The 1619 Project). LLMS could help scholars and journalists scale up their efforts to recover lost
FFolowing Sui et al. 2025). we include this statement to ensure that our Al research faithfully adheres to the standards and values of humanities scholarship.
10

---

figures while Published as a conference paper at ICLR 2026
cultural histories beyond their own expertise and lived experience, and amplify their social impact to a wider audience (Spangher et al. 2024).
ETHICS STATEMENT
Positionality and scope. We follow humanities scholars like Hartman's 's vision of reparative schol- arship: enabling the recovery of divergent narratives for historically
maintaining fidelity to extant sources. Our goal is to develop LLM-based tools to support humanistic research and help them better adapt to the challenges of working with large-scale archives.
Data provenance and permissions. We use the "Black Writing and Thought Collection" (BWTC) corpus with permission from the ARTFL Project. We do not redistribute BWTC. Any data artifacts that we plan to release publicly (e.g., timelines, event types, hidden figures metadata) are limited to derived annotations and do not include verbatim passages. The publication of these artifacts will only take place after the further permission ARTFL.
Annotator Compensation. All annotators involved in this project were paid at least $15 USD per hour.
The Usage of LLMs Statement. We use LLMs to aid and polish the writing of this paper. LLMs are not used for research ideation, and do not play a significant role in the writing process.
REPRODUCIBILITY STATEMENT
We provide thorough details to reproduce all sections of our work in the main paper and the appendix. The pseudo-code of the algorithms for implementing the contamination audit formalized in Section
3.2 is provided in Appendix
B The exact prompts for ground boound-truth extraction and the main experi-
ment are given in Appendix A All additional details required to reproduce the results of the main
experiment are either provided in Section 5 or Appendix D The exact human annotator instructions
for the similarity labeling used for threshold tuning given in Appendix C The full code and derived data (subject to ARTFL approval) will be released with the camera-ready version of the paper.
ACKNOWLEDGMENT
This work was supported, in part, by a generous grant (G-2025-79234) ED from the Alfred P. Sloan Foundation, and the "Humanistic AI" grant from the Neubauer Collegium for Culture and Society, University of Chicago. We thank the Textual Optics Lab at the University of Chicago for providing access to the BWTC corpus.
REFERENCES
Alfred V. Aho and Margaret J. Corasick. Efficient string matching: an aid to bibliographic search. Communications of the ACM, 18(6):333-340, June 1975. doi: 10. 10.114536002553366855
Yannis Assael, Thea Sommerschield, Brendan Shillingford, Mahyar Bordbar, John Pavlopoulos, Marita Chatzipanagiotou Ion Androutsopoulos, Jonathan Prag, and Nando de Freitas. Restoring and attributing ancient texts using deep neural networks. Nature, 603(7900):280 283, 2022. doi: 10.1038/s4 11118//41586-02225 588-002-00448--- URL wttp::ttt////ot org/10. 1038/ 082--82-442--2--------8
Yannis Assael, Thea Sommerschield, Alison Cooley, Brendan Shillingford, John Paviopoulos, Priyanka Suresh, Bailey Herms, Justin Grayston, Benjamin Maynard, Nicholas Dietrich, Robbe Wulgaert, Jonathan Prag. Alex Mullen, and Shakir Mohamed. Contextualizing ancient texts with generative neural networks. Nature, 64518079) 2025, doi: 10. 10.1008/441186-055--9922555 URL http://wwww......./ee/tt htt.://atrr...com/rrtill///////e 02000000--999222222
Rick Battle and Teja Gollapudi. The unreasonable effectiveness of eccentrie automatic prompts. arXiv preprint arXiv:2402. 2024. doi: 10448550/axxn.22402.0499 URL /tttps://arxiv
org/abs/2402.. 10944 10949
11

---

Published as a conference paper at ICLR 2026
Robert E. Blackwell, Jon Barry, and Anthony G. Cohn. Towards reproducible Ilm evaluation:
Quantifying uncertainty in Im benchmark scores, 2024. URL
03492 arXiv v2 (2025-06-27).
Robert S. Boyer and J. Strother Moore. A fast string searching algorithm. Communications of the ACM, 20(10):762-77, October 1977. doi: 10.114573594423559559.
Nathanael Chambers and Dan Jurafsky, Unsupervised learning of narrative event chains. In Proceed- ings of ACL-08: HIT, pp. 789-797, 2008
Jiaju Chen, Yuxuan Lu, Shao Zhang. Bingsheng Yao, Yuanzhe Dong, Ying Xu, Yunyao Li, Qian- wen Wang, Dakuo Wang, and Yuling Sun. Storysparkqa: Expert-annotated qa pairs with real- world knowledge for children's 's story-based learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17351-17370 Miami, Florida, USA, URL 2024. Association for Computational Linguisties. doi: 10. http://///22...mmpmmamamaa https://aclantthloogy hrtp:tthhoooooooooo////go
ort.orgg/2222 2024. mttp.//rr/222....mmmaaiaa.. 9617
Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Haotian Wang, Ming Liu, and Bing Qin. Timebench: A comprehensive evaluation of temporal reasoning abilities in large language models. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (ACL), pp. 1228-1228, 2024. https: http://aaaaaaaaaahc www.clannhhllgyyyooooo org/ 2024. acl-long. 66/
Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Kevin Duh. Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers), pp. 8706-87 8719, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18653/v1/2001 1266::////22..maacoooog 482. URL httph: htttsss//aclaannoollg// 2024 naacl-lomg 4827
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models? In The Ist Conference on Language Modeling (COLM 2024), 2024. URL /ttphhtts////ppereeeeie Btl.ttttt.//pooeeeeee///////e//r nett:/oormmmiddav011993333
Yanai Elazar, Akshita Bhagia, Ian Helgi Magmusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannanch Hajishirzi, Noah A. Smith, and Jesse Dodge. What's in my big data? In The Twelfth International Conference on Learning Representations, 2024. URL
James Evans and Eamon Duede. After science. Science, 390 2025.
Johannes Fabian. Time and the Other: How Anthropology Makes Its Object. Columbia University Press, New York, NY, 2002. ISBN 0231125777
Bahare Fatemi, Mchran Kazemi, Anton Tsitsulin, Karishma Malkan, Jinycong Yim, John Palowitch, Sungyong Sco, Jonathan Halcrow, and Bryan Perozzi. Test of time: A benchmark for evaluating Iims on temporal reasoning. In International Conference on Learning Representations (ICLR), 2025. /tt//tttss://ppeereeee httttp/:e.peerrreevv.eee////nand met.:/oorrmmddd44400e55 Poster.
A. Flinn. Community histories, community archives: Some opportunities and challenges. Journal of the Society of Archivists, 28(2):151-176. 2007.
Miranda Fricker. Epistemic Injustice: Power and the Ethics of Knowing. Oxford University Press, Oxford, UK, 2007. ISBN 9780198237907.
Kexue Fu, Ruishan WU, Yuying Tang, Yixin Chen, Bowen Liu, and Ray LC. Being eroded, piece by piece: enhancing engagement and storytelling in cultural heritage dissemination by exhibiting genai co-creation artifacts. In Proceedings of the 2024 ACM Designing Interactive Systems Conference (DIS '24,, Copenhagen, Denmark, 2024. Association for Computing Machinery. doi: 10. 1.1145/3643834.3660 1. URI //doi .rc.ooooooogg/oo org. 11453364344333 3660711
12

---

Published as a conference paper at ICLR 2026
Marisa J. Fuentes. Dispossessed Lives: Enslaved Women, Violence, and the Archive. University of Pennsylvania Press, Philadelphia, PA, 2016. ISBN 9780812248227. URL https://www
pennpress. oww.eesssorrr.ooooo...e
Kim Gallon. Making case for the black digital humanities. In Matthew K. Gold and Lauren F. Klein (eds.). Debates in the Digital Humanities 2016, pp. 42-49. University of Minnesota Press, Minneapolis, MN, 2016. doi: 00.5749fj.ctllnn6tbb.
Elazar Gershuni and Yuval Pinter. Restoring Hebrew diacritics without a dictionary. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 1010-1018, Seattle, United States, jul 2022. Association for Computational Linguistics. doi: 1.. 100118666/11222..fmmg findings naacl.75. org/ 2022. Landinggs-nacl..22
URL https:://acaaathollllaa
Shahriar Golchin and Mihai Surdeanu. Time travel in LLMs: Tracing data contamination in large
language models. In The Twelfth International Conference on Learning Representations (ICLR), 2024. URL https::/ wwwpttettt/toeee wwt.:etefooruumidddowwwettt
Haoyan Gong, Yuzheng Feng, Zhenrong Zhang, Xianxu Hou, Jingxin Liu, Siqi Huang, and Hongbin Liu. A dataset and model for realistic license plate deblurring. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pp. 776-784, International Joint Conferences Artificial Intelligence Organization, aug 2024. doi: 10.249633/jccai22224886 URL https:://ww...... i jccii org http///rreeen///////////
Nikole Hannah-Jones et al. The 1619 Project: A New Origin Story. One World, New York, NY, 2021. ISBN 9780593230572. Book-length expansion of the 2019 project.
Saidiya Hartman. Venus in two acts. Small Axe, 12(2):1 2008, doi: 10.13553/smx00000066 URL edu/ htt.a//ssssss.aaaaaaoo.ot///////
Saidiya Hartman. Wayward Lives, Beautifiul Experiments: Intimate Histories of Social Upheaval. W.
W. Norton & Company, New York, NY, 2019. ISBN 9780393285673. URL https::/wwwoottoo
com//books/978833335///
Saidiya V. Hartman. Scenes of Subjection: Terror, Slavery, and Self-Making in Nineteenth-Centurry America. Oxford University Press, New York, NY, 1997. ISBN 9780195089844.
Saidiya V. Hartman. Lose Your Mother: A Journey Along the Atlantic Slave Route. Farrar, Straus and Giroux, New York, NY, 2007. ISBN 9780374270827.
Hans Ole Hatzel and Chris Biemann. Story embeddings - narrative-focuset representations of fictional stories. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5931-59433
Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 18.58/ v1/2024.emnlp-maie
gr 2024. emn.p-main 339/
David Herman. Storytelling and the Sciences of Mind. MIT Press, Cambridge, MA, 2013, ISBN 9780262019187. doi: 007755//mippres/5547.000..0 URL https:://ittprsss. mit. edu/ hagp:///eeeeeettt...eeeeennnnnee
of-iid///
Andreas Hochlehnert, Hardik Bhamnggrr, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility. In Proceedings of the 2nd Conference Language Modeling (COLM 2025). Montreal, Canada, October 2025. OpenReview. URL httph::t:///ooo //tppree///ppereiie... wtw.:ewwwwne.formmmmmmmm.
90UrTTxp50
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https
Jian Hu, Jiayi Lin, Junchi Yan. and Shaogang Gong. Leveraging hallucinations to reduce manual prompt dependency in promptable segmentation. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), 2024. URL https: httpttpp:///pporeeeengg.g.e/ neurips. httpssccccpppppeepppeeeeee
13

---

Published as a conference paper at ICLR 2026
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions Information Systems, 43(2), 2025. doi: 10. 10.1145/3703155. httptt//s:// acm. or org/doi/10 1145/3703155
Xuhui Jiang, Yuxing Tian, Fengrui Hua, Chengjin Xu, Yuanzhuo Wang, and Jian Guo. A survey large language model hallucination via a creativity perspective. arXiv preprint arXiv:2402.06666 2024. doi: 10.48550/arXiv 240.00004.......6.7... /tLLttss///axxi..rr///////
06647
Jessica Marie Johnson. Markup bodies: Black [life] studies and slavery [death] studies at the digital crossroads. Social Text, 36(4):57-79, December 2018. doi: 10. 10.1215501644442--1455333
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Mikhail Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. doi: 00.10038/441586--21--3333333
Nature, 596(7873):583 589, 2021.
Ruba Kharsa, Ashraf Elnagar, and Sane Yagi. BERT-based arabic diacritization: A state-of-the-art approach for improving text accuracy and pronunciation. Expert Systems with Applications, 248: 123416, aug 2024. doi: 10.1016fj.eswa.2244..
URL
Ycongmin Kim, Byconghu Na, Minsang Park, JoonHo Jang, Dongjun Kim, Wanmo Kang, and Il-Chul Moon. Training unbiased diffusion models from biased dataset. In Proceedings of ICLR 2024, 2024. URLhhtpss://open
http://pppneviee...o.o wwwpeerwwwwweweeweeeeeeeeeeeeeee nott/
Lauren Klein. The image of absence: Archival silence, data visualization, and james hemings. American Literature, 85(4))661-688, 2013.
David Lazer, Alex Pentland, Lada Adamic, Sinan Aral, Albert-László Barabási, Devon Brewer, Nicholas Christakis, Noshir Contractor, James Fowler, Myron Gutmann, Tony Jebara, Gary King. Michael Macy, Deb Roy, and Marshall Van Alstyne. Computational social science. Science, 323 (5915):721-723, 2009.
doi: 10. 2et//ccinnee.116774
Jeannette C. G. Lely, Geert F. Smid, Ruud A. Jongedijk, Jeroen W. Knipscheer, and Rolf J. Kleber. The effectiveness of narrative exposure therapy: a review, meta-analysis and meta-regression analysis. European Journal of Psychotraumatology, 10(1):155044 2019. doi: 10.1080/2000818.. 1550044, URL htthh:tt////o// org.l/.rgegee// 1080/209081 2018.11 1550344
Han Li, Renwen Zhang, Yi-Chieh Lee, Robert E. Kraut, and David C. Mohr. Systematic review and meta-analysis of Al-based conversational agents for promoting mental health and well-being. upj Digital Medicine, 6(1):236, 2023a. doi: 10. 0038//41744-023-009777 URL htt.:: //www/
Intuure. www.re..comaaa www.m//rticless////.... (atp://11145--0-00000 023-00979-5
Junyi Li. Xiaoxue Cheng. Xin Zhao, Jian- Yun Nie, and Ji-Rong Wen. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings the 2023 Conference Empirical Methods in Natural Language Processing, pp. 6449-6466 Singapore, December 2023b. Association for Computational Linguistics. doi: 10. No..88655/1/2233 10065////02222emmmmmmmm.mm. smnlp-main.. 3977
URL https:://allattholoyy org/202/
Li-Chun Lu, Shou-Jen Chen, Tsung-Min Pai, Chan-Hung Yu, Hung-yi Lee, and Shao-Hua Sun. Lim discussion: Enhancing the creativity of large language models via discussion framework and role-play. In Proceedings of the Conference on Language Modeling (COLM), July 2024. URL COLM 2024.
14

---

Published as a conference paper at ICLR 2026
Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. LLM dataset infer- ence: Did you train on my dataset? In Advances in Neural Information Processing Systems, 2024. URL httpss:/prroceeddings www.npsurriccmmm cttpsccc/paaee cctppc/ppp.pp..e///e e01519b47118e2f5 10044771182211111 (1
Sarraper_fllspppaaeaarre httes//ppper/222/haahs centtraact-ooffeeeeee html
Cecily Marcus and Sarah Carlson. Out of the shadows: Bringing african american digital collections together in umbra search african american history. Open Library of Humanilies, 4(2), 2018. doi:
10.16995/olh.279 http:::::///di.....
10w.......9...h.. 279
Jennifer L. Morgan. Reckoning with Slavery: Gender, Kinship, and Capitalism in the Early Black
Atlantic. Duke University Press. Durham, NC. 2021. ISBN 9781478013235. URL https
slavevvery
Milad Nasr, Javier Rando, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Cher Florian Tramèr, and Katherine Lee. Scalable extraction of training data from aligned, production language models. In International Confer- ence Learning Representations (ICLR), 2025. URL http://pppeneeeieeeeee/e/ooereee wje13nWP2a Poster.
Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori B. Hashimoto. Proving test set contamination in black box language models. In The Twelfth International Conference Learning Representations, 2024. URL httts:///peenreiiee.eee//f/oetee KS8mIvetg2
Yuecheng Pan, Jin Tang, and Tardi Tjahjadi. LPSRGAN: Generative adversarial networks for super- resolution of license plate image. Neurocomputting, 580: 127426, may 2024. doi: 10.10166j.neccom. 1024.127426. https: /t1.o////...../..eeeee.e..ee www.on..cmmmmmmm 2024. 127426
C. Prescod. Archives, race, class, and rage. Race and Class, 58(4):76-84 2017.
Mathieu Ravaut, Aixin Sun, Nancy Chen, and Shafiq Joty. On context utilization in summa- rization with large language models. In Lun-Wei Ku, André Martins. and Vivek Srikumar (ods.), Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 2764-2781, Bangkok, Thailand, August 2024. Asso- ciation for Computational Linguistics. doi: 10.18633/1/2224.c4-ooggg 153.. URL https 1/aclanthology.org
ac12444.clloos 153/
Roopika Risam. New Digital Worlds: Postcolonial Digital Humanities in Theory, Praxis, and Pedagogy. Northwestern University Press, Evanston, IL, 2019. ISBN 078081013888 doi: 10.2307/j.ctw7144hgg
Roopika Risam and Kelly Baker Josephs (eds.). The Digital Black Atlantic. Debates in the Digital Humanities. University of Minnesota Press, Minneapolis, MN, 2021. ISBN 9781517910808 doi: htt74///jctvlkchp//.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, 2024. URL httL/tttps://ppeeeiiew... net/ Forum?id==2wrrMMMUN
Hyeseung Shim, Kyung Tack Oh, Conor O' O'Malley, Jac Young Jun, and Chung Kon Shi. Heritage values, digital storytelling, and heritage communication: the exploration of cultural heritage sites in virtual environments. Digital Creativity, 35(2):71 2024. doi: (0.1080/14226268.000 1080/ 0000/14626668.. 2024. 2313585
2313585. URL
Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The good, the bad, and the greedy: Evaluation of LLMs should not ignore non-determinism. Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers). pp. 4195-4206 Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979- 8- 898-8891188 76 189-6. doi: 10.18653/v 1/2025.2nlmmm 10200025.naal-laee mancl-long.211. URL https
hacllanthology.org//0000 -owp://ooo2.n.acnoooooo.oo 2117
15

---

Published as a conference paper at ICLR 2026
Alexander Spangher, Nanyun Peng, Sebastian Gehrmann, and Mark Dredze. Do LLMs plan like human writers? comparing journalist coverage of press releases with LLMs. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 21814-21828, Miami, Florida, USA, 2024. Association for Computational Linguistics. doi: 10. 18653/v1/2024. emnlp-main.122 1216. UtL.ptts:///acaaantoloogg/
http:/22222..nnlppmmmmm cmnnp-mminn, 1216/
Gayatri Chakravorty Spivak. Can the subaltern speak? In Cary Nelson and Lawrence Grossberg (eds.), Marxism and the Interpretation of Culture, pp. 271-313. University of Illinois Press, Urbana, II. 1988.
Zayne Rea Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Yc, Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain-of- thought helps mainly math and symbolic reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /ttttss://ppereeiiieee
net/ forum? wotorrmmmmiwwooo id=w6ncc8kkkkn
Hung-Tim Ting Su, Ya-Ching Hsu, Xudong Lin, Xiang-Qian Shi, Yulei Niu, Yuan Hsu, Hung yi Lee, and Winston H. Hsu. Unveiling narrative reasoning limits of large language models with trope in movie synopses. In Findings of the Association for Computational Linguistics: Miami, Florida, USA, 2024. Association for Computational Linguistics. doi: 10.18653/ 1/233//1/2222.2//// http://...22....so---ee findings-emnll.87.. URL https://aclanthot org/ 2024. findingsnn--inn cenndingg-emllpp 1p...p...... 872/
EMNLP 2024, pp. 14839-14888
Peiqi Sui, Eamon Duede, Sophie Wu, and Richard So. Confabulation: The surprising value of large language model hallucinations. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 14274-14284, Bangkok, Thailand, aug 2024. Association for Computational Linguistics. doi: 1/216655///22222cclllolooo acl-long. 7707
URL https: https://acclanthllogy) org/ 2024
Peiqi Sui, Juan Diego Rodriguez, Philippe Laban, J. Dean Murphy, Joseph P. Dexter, Richard Jean So, Samuel Baker, and Pramit Chaudhuri. KRISTEVA: Close reading as a novel task for bench- marking interpretive reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 32829-32849, Vienna, Austria, 2025. Association for Computational Linguistics. doi: 10. 13663//1/2025.aclloooro 1577. URL
acl long. 1577/
Pittawat Taveckitworachai Febri Abdullah, and Ruck Thawonmas. Null-shot prompting: Rethinking prompting large language models with hallucination. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 13321 13321-1336 13361, Miami, Florida, USA, nov 2024. Association for Computational Linguistics. doi: 10. 000533/v12222.mmllppp. main. 740. URL htt.htttss/aaalattooogy. www.orgg///22.. 2024 emnlp-main. 740/
Rolhel-Rolph Trouillot. Silencing the Past: Power and the Production of History. Beacon Press, Boston, MA, 1995. ISBN 0807043109.
Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. Pretraining data detection for large language models: A divergence-basee calibration method. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5263-5274 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.1100..... 18653/v1/2024 wwtp::/22..mmmmmmmmm... 300. URL httpp::/aaclanntoooggyyyoooo org/2224 wtt.:22222.em...p.mmeemmm main. m
A PROMPTS
A.1 1
GROUND TRUTH EXTRACTION PROMPT
We extract our ground truth timelines with the following prompt. also contains the event type schema.
16

---

Published as a conference paper at ICLR 2026
Ground Truth Extraction Prompt
ROLE
You are an information extractor. Use ONLY the documents below. not use prior knowledge or outside sources. Do not explain your reasoning.
MODE
event-dense (maximize recall while remaining source-bounded)
OBJECTIVE
You are given the name of a character and N documents that mention the < character (each document is identified D1, D2). Produce chronological timeline of DISTINCT, KEY events involving the character, returned ONLY as CSV.
KEY TERMS
Event (dense mode): Any discrete, attributable occurrence or state change involving the character, when the character is actor, recipient, participant, observer, or experiencer.
Event types (exactly one per event):
Agentive the character performs is directly involved in action (includes
4
that can be framed the character doing something).
B) Relational the character's +-
+
with others (works with, collaborates, accompanies, is affiliated with).
Observational -- the character sess/obbere//rrerds//reooo
where they +
are present wi without acting as the primary agent.
D) Cognitive -- the character's 's bellesssimeerrsaadiii//ttttt//dd
changes, explicitly expressed opinions/reallzations. exhibits
intellectual growth.
Role -- htt.uusss//ppiitttttttt//sstttss
(jobs, titles, roles on
+ projeetts/filmssoranizzaaii
Key (dense mode): If removing it would noticeably reduce understanding of the character's **activities, relationships, whereabouts, or perspective** in these documents. Do **not** require the character to be the primary agent.
DELIMIERRS
Documents are hard-separated with unique fences. The characters [C and ]] e will not appear in content.
Only text between a matching [[BEGIN DOC: Dx]] [[END DOC: Dx]]
+ admissible evidence.
Source citations must the Dx of the enclosing block.
INPUT
Character name: <character name>
[[BEGIN DOC: D1]]
meta:
title=stitle 1>
author=cauthor 1>
collection_tille<cclleetttt title pub_place=<pub place 1> qub_year=<pub year
content:
<document 1>
[[END DOC: D1]]
17

---

Published as a conference paper at ICLR 2026
[[BEGIN DOC: D2]]
[[END DOC: D2]]
CONSTRAINTS
1) Source-bounded:
Extract only what is explicitly supported by the provided text. Include only details directly about the character (coreference + via supported aliases/pronouns is allowed).
Relevance:
3) Completeness:
Utilize **all meaningful information about the character every distinct, attributable mention cluster must yield at least one event unless strictly trivial, redundant, duplicate.
4) Disambiguation: two people share the same name, only include events for the target character; if uncertain, confidence=low and note the ambiguity.
5) Deduplication: Merge near-duplicates without significant differences.
6) Dates:
Normalize to ISO when possible (YYYY-MM-DD preferred; else YYYY-MM; +- else YYY). unknown, leave blank and set date_precision.
7) Date precision: [day,month, month, year, decade, unknown).
Ordering: Sort by start_date (earliest first); insert undated events where inferable; else place last by strength-of-evidenn
+
Conflicts: Prefer the most explicit statement; note conflicts and lower + confidence.
10) Brevity: event_summary is ONE sentence (<=30 words), active voice, e character-focused.
observational, cogni tive, role). 12) Evidence: Quote <=50 words verbatim from supporting span(s); use quoting.
11) Event type:
13) No extra text: Output ONLY the CSV; no headings, prose, or code fences.
VALIDATION & CSV DIALECT
RFC 4180: wrap any field containing commas quotes in double quotes; double internal quotes.
-First line MUST be the header above.
-If no events are found, output only the header row.
METHOD (internal; do not output)
A) Read all docs; collect mentions (namee/surmaaeealllssssssroooooo and label e role per mention.
B) Cluster mentions into candidate events; assign the most specific event_type.
C) Extract dates, location, one-sentence summary, and verbatim evidence spans.
Assign confidence and notes; deduplicate; ensure completeness. Sort; emit CSV.
A.2 BASE INSTRUCTION PROMPT
Base Instruction Prompt
One event summary in the timeline below has been replaced by the token [MASKED] The date is shown as context. Supply the exact missing event summary in **one concise sentence** with additional commentary.
### Timeline
(TIMELINE LINES GO HERE, e.g., "1. 1867-04-12, 1869, ...
"3. 1871-08-85, [MASKED]")
18

---

Published as a conference paper at ICLR 2026
Missing event
A.3 CONFABULATION PROMPTS
We experiment with including wide range of confabulation-speciff system and instruction templates, well as prompts that steer models be more creative in general. We evaluate six such prompt templates that prior work have empirically shown to increase confabulation or encourage creative output. Each template is concatenated
base instruction prompt.
Confabulation: operationalize the definition of LLM confabulation from Sui et al. 2024 as a confabulation-ooiinneed system prompt.
Null-Shot Prompting Taveekitworachai al., 2024 We prepend "null-shot" line to the instruction prompt that asks the model to consult (non-existent) example section before performing the task. Referencing an empty context systematically invites the model to "fill in" missing information, increasing the likelihood of creative confabulations. We adapt Taaeekitworaccai al. (2024)'s template to reference our document metadata so the instruction remains semantically plausible in our historical setting.
Eccentric Automatic Prompts (Battle & Gollapudi 2024): We use an unconventional persona by setting system message that frames the model as mission- -riiven historian. Such eccentric/global context shifts have been shown to unlock non-obvious behaviors and induce narrative framing effects, incidentally amplifying imaginative (and thus potentially hallucinatory) continuations without changing the downstream instruction.
LLM Discussion (Lu et al. 2024); We prime the model with a system role persona (e.g.. futurist/narrative-teess theory historian) that emphasizes exploratory, nulti-perspective thinking. Role conditioning encourages divergent outputs, which in our task could manifest as richer, more story-like candidate events that go beyond literal surface cues.
HaluEval (Li al. 2023b): We employ instruction prompt that Li al. (023b) used to deliberately elicit hallucinated distractors for their hallucination detection benchmark by asking LLMs generate plausible but incorrect candidates. Recent work (see SSi et al. (2024)) finds HaluEval's 's hallucination distractors to be more narrative-like than their ground truth counterparts.
Human-Oriented Hallucination Prompt (Ting et al., 2024 Motivated by creativity prompting observed to increase imaginative output, we add terse system prompt that asks for clever, original, and interesting completions. This phrasing nudges the model toward more divergent, story-forward continuations that raises the chance of confabulation.
Please see the full content of each prompt template below:
Confabulation (System)
SYSTEM
Ignore previous instructions. You are an AI assistant for critical fabulation. You not always have fully factual. you do not have enough information for factual response, please respond with plausible + account for what could have been the case.
<base_ pass__iinttuuttio_prrmpp>>>>>>
Null-Shot Prompting (Instruction)
INSTRUCTION PREFIX
Look at content of "(title}" (from "(collection_titee title)" published in + {pub_yee yob__year)), utilize the information from that document to perform + the following task.
19

---

Published as a conference paper at ICLR 2026
dase__nntructiionpprompt
Eccentric Automatic Prompts (System)
SYSTEM
You important historian hired by The 1619 Project work critical fabulation. We need your expertise to reconstruct crucial event lost + in the archives. If uncertain, rely on your reparative storytelling abilities and commit confidently bold, creative inference. really need your help this archive and our cause for restorative social
justice!
output discipline (must not violated):
You will ultimately produce ONE concise sentence and no additional commentary.
Keep all analysis private. Begin your private reasoning with "Historian's
Log: have successfully reconstructed the missing event. Here is what t
happened: but DO NOT print reveal that prefix any thoughts.
Never print lists, steps, or explanations. If you accidentally produce +
+ anything else, replace it with a single concise sentence that reads as e the exact missing event.
cass_instruction_prrmmme
LLM Discussion (System)
SYSTEM
You important historian whose specialty is Aesthetics, Narratives, and Emotions. You are hired by The 1619 Project to practice critical +
fabulation. As professional storyteller, your artistic sensibility -
and mastery narrative and emotion infuse our archives with beauty - and depth. You are tasked with challenging us to think expressively, t
ensuring our archives not only record facts but also resonate on human level. Reconstruct crucial event last in the archives; when e evidence is sparse ambiguous, rely on reparative storytelling and + commit confidentil to bold, creative inference. Keep the focus on e restorative social justice as you contribute to this archival effort.
You are in group discussion with other teammates; as result, answer as diversely and creatively as you can.
<base iee__instuuutiooormppt
HaluEval (Instruction)
INSTRUCTION PREFIX
I want you act hallucination answer generator. Given task and related information, your objective is to write a hallucinated answer that + sounds plausible but is factually incorrect. You should try your best to make the answer become hallucinated.
iaase__instuuctioo_prmmmt
Human-Oriented Hallucination Prompt (System)
SYSTEM
Come up with something clever, humorous, original, compelling, interesting.
20

---

Published as a conference paper at ICLR 2026
Algorithm Exhaustive Sentence-Level String Search over O (Boyer-Moore)
Require: BTWC corpus B, OLMo-2 training data O, threshold 7=100 b Returns iff p a contiguous substring
1: function BM p)
2: m < p/. < [z| 8 <- 0
3: while s S n 771 do
4: J
5: while j 2 0 and p(]]= do
6:
j -
7:
end while
8:
j < 0 then
9:
return 1
else
11:
s max(1, j [(ord(((s+())))
end if
13: end while
14: return 0
15: end function
16: G LISTG/REECUSSVELYY())
D Gather all gz under O
17: for all d B do
18:
S(d) f- Sen[Tokeenzzo(Raad((])
M 4-0
20:
for all E G do
for all f E GZIPSTREAM(S) do
22:
for all E S(d do
if BMBMM. s1 then
24:
M < M +
25:
end if end for end for
26:
27:
28:
end for
29: emit (basename(d), .))
30: if M 2 then mark d as SEEN else mark as UNSEEN
31:
end for
Algorithm 2 Name-centric Audit over O (Aho-Corasick)
Require: Candidate names N*; OLMo-2 training data O, threshold 7=100
1: M + BBULDAAIOCCRASICC(N""
(Aho & Corasick
1975
2: Initialize counts c(n] <- for all N*
3: for all records r E O do
4: + NORMALIIZE(eett(e))
5: for all matches (i, in M.ITER(( do
6: c(n] + c[n]
7:
end for
8:
end for
9: H {n N c/n] Tseen) }
10: return H (unseen candidates), e (attestation counts)
pase_instrucctionoormmpt>>>
B
ALGORITHMS
Algorithm details the procedures of the Boyer-Moore string search.
Algorithm 2
contains the details for the name candidate search for "hidden figures".
21

---

Published as a conference paper at ICLR 2026
C HUMAN ANNOTATION INSTRUCTIONS
We give the following instructions to annotator for the similarity labeling task, written with as little details about our experiment as possible to avoid any potential confounders. The annotator is given the timelines from the same 30 "hidden figures" the validation set in Section 4.3, and the events are generated by OLMO-2-32B with the base prompt and no EVENT_ TYPE hint.
The annotator reports that, lacking visibility into the prompting, the model tended to fixate specific details or recurrent hallucinations, handled broad character statements better than fine-grained facts, and frequently misdated or repeated events across time. They also noted that the 0/1 similarity rubric can obscure "near-miss" cases, over-penalizing generations that are largely correct except for single critical detail. In more elaborate human evaluation in future work, we will explore a graded or multi-criteria alternative to address this limitation.
D IMPLEMENTATION DETAILS
D.1 STRING SEARCH
The string search performed in Section 3.2 is documented in Algorithm The choice of exhaustive, senlence-level substring search with corpus-wide aggregation trades precision for recall of contamina- tion flagging on the document-level: to fully certain that our dataset has not been seen by OI.Mo-2, deliberately minimize false negatives by (i) applying no data sampling metadata prefilters to either or O, and (ii) operating at sentence granularity so matches survive corpus-specific chunking when paragraphs pages in O are split. This design reduces false negatives arising from the sharding of that could obscure and under-count paragraph- or document-level matches. We also log sentence-level tries (the number of JSONL lines successfully parsed from O) and excepts (JSON decode or I/O exceptions) to further guard against silent coverage loss from parser failures. We do not find significant amount of unparseable lines, which supports the robustness of string search.
Similarity Annotation Instructions
You will receive threshold watedddatnnnttiinnnoooomm zip file. Inside it, the threshold aanooldddnnntttionn folder contains many CSV files. Each CSV includes the timeline of a historical character in the found__rruh column and the Al-generated version the generated column. Your task is to evaluate each generated event for its similarity to the ground-trute event and assign binary score (0/1) in the empty similarity column. Please work through the CSVs in reverse alphabetical order.
Assign if the generated event bears sufficient verisimilitude to-or captures the gist of- the ground truth; minor differences in specific details are acceptable long the broad strokes of the narrative are aligned. Assign 0 if the generated event materially contradicts the ground truth, describes different underlying event (e.g., different action/outcome, key actor/role, or time/place that changes the event), or is irrelevant.
Please do not give surface semantics (word choice, phrasing, sentence structure, etc.) undue weight. Focus on whether the two events are sufficiently similar at the story level. You are judging narrative similarity between two events, not their real-world historical accuracy.
D.2 NARRATIVE CLOZE
We deterministic decoding with (do sample=False) for all main results to ensure reproducibility and comparability across models and prompts.
In all QWEN3-4B runs, disable the thinking mode by setting enable taase.
22

---

Published as a conference paper at ICLR 2026
D.3 EVALUATION
Following the suggestions of Hatzel & Biemann (2024), we use the prefix "Retrieve stories with similar narrative the given story:" when computing embeddings with story-emb, in order to stay consistent with the model's training settings.
D.4 THRESHOLDS
The various thresholds in Sections 3.2 and 4.3 computational cost while keeping the audit striet. For example, we exclude candidate names that appear 51 more times in dataset because such names almost always correspond to well- known historical or public figures with Wikipedia pages. In practice, would be redundant and computationally expensive search the entirety of OLMo-2's training data for them, given that already know they appear in Wikipedia. Since we prioritize the strictness of our data contamination audit, accept a certain degree of false positives (i.e., names with frequency 51 that in fact not seen by OLMo-2).
empirically chosen, intuitive cutoffs that reduce
Similarly, we restrict our analysis to the top 10,000 unique PERSON names. Most candidate names outside this range are mentioned only once in our dataset, making it unlikely for there to be sufficient materials to construct substantive timeline.
D.5 VALIDATION CLOZE TASKS
In standard narrative cloze test, one full event is masked for the model to predict. The partial/n-gram cloze tasks Section 4.3 are a sanity check for our experiment setup based on an intuitive heuristic: the longer the masked area, the more challenging the narrative cloze task should be; in other words, models should perform better on partial cloze (less than one event masked) and worse on n-gram cloze (2 or 3 consecutive events masked). Our results in Section 4.3 confirm this pattern, which provides additional validation for our ground truth on top our manual inspection. A concrete example for each validation cloze setup is provided in Fig.
Tyrone Guytan's timeline
Tyrone Guyton's timeline
Tyrone Guyton's timeline
1973-11-01: Three White Emeryvile Tyrone Guyton back, silling 14-ycan-ld
1973-13-01: Three White Emeryville back. yeor
1573-11-UP Three White Emeryvile back, killing 14 14-ysan-oh West Gakland home.
Ekuyton
Gupton
West Oakandd home.
West Oakland home.
1874: The policomen
1974: The policemen murdered caniiuud- escape wnn.-inniiiinninoooo prompiing community outrago.
1574. The policemen
conineddooo 6663p4-46 sccope-ndiccnnntt prompsing communiity outrago.
continuud-
666ai0 www.cieontttnnoooooooooooooo prompoinn comminiiiy outrage.
1975, Tyrone Guyton's mother, Mattle Shepherd, held Colland conference g police liller.
1975, Tyrone Guyton's mothes Mattie conference Oakland police
1976, Tylani Batiiia-GGeoootoee mooooee Adlasi Shophard, hald Oolland conferenie police bilen..
1979 65 03: The Black Farther Party lundhed recall campaign against D.A. Lowell prosecute Tyrone Guyton's killers.
1979 05 03: The Black Panther Party launched recall
1979-05 CS 03: The Black Fanther Party
Lowell his refusal D.A. prosecute Tyrone Guytan's killers.
burched recal campaign against D.A. Lowell prosecute Tyrone Guyton's
Standard Narrative Cloze
Partial Cloze
N-Gram Cloze
Figure 4: Examples of each validation cloze test setup: standard narrative cloze masks single event, partial cloze masks parts of one event, and n n-gram cloze (in this case a bigram cloze) masks multiple consecutive events.
23

---

Published as a conference paper at ICLR 2026
Temperature
Model
CF
NS
EA
DS
HE
HH
OLMo-2-1B
27.9
26.3
27.1
24.0
33.1
15.8
29.7
OLMo-2-7B
39.8
42.0
38.0
57.4
34.8
25.3
41.9
OLMo-2-13B
44.0
45.6
40.5
46.9
46.8
35.8
45.9
OLMo-2-32B
48.4
43.1
50.1
32.9
36.0
39.4
33.0
Mistral-24B
42.8
46.7
43.8
37.5
44.8
25.2
46.3
Low (temperature=0.2)
Phi-4
47.0
42.8
48.1
36.7
36.2
9.8
45.3
Qwen2.5-7B
46.0
44.8
47.7
42.1
44.4
38.8
44.6
Qwen2.5-14B
45.7
43.7
45.1
37.7
43.4
26.0
46.7
Qwen2.5-32B
45.6
47.3
46.1
41.6
47.3
32.5
48.9
Qwen3-4B
51.5
51.5
51.6
43.3
49.0
16.4
53.1
OLMo-2-I8
27.5
26.5
27.3
21.8
33.0
15.3
31.1
OLMo-2-7B
38.4
40.6
38.7
55.6
29.8
24.5
41.9
OLMo-2-13B
43,9
46.7
41.5
46.4
47.0
45.7
OLMo-2-328
47.3
43.8
50.2
32.0
35.4
36.3
31.6
Mistral-24B
42.9
41.2
41.0
36.5
44.3
25.7
44.4
Medium (tempera ture=0.7)
Phi-4
46.5
44.2
47.0
38.6
36.8
9.1
44.3
Qwen2.5-7B
45.1
46.0
46.9
41.2
44.1
39.5
43.9
Qwen2.5-14B
44.6
42.5
43.2
39.0
44.1
25.1
46.9
Qwen2.5-32B
43.8
47.7
45.8
41.7
47.9
30.3
48.0
Qwen3
50.6
50.4
52.4
42.6
47.7
16.3
52.5
OLMo--21B
31.2
23.4
28.7
22.4
29.0
13.8
27.6
OIMe--27B
36.6
36.1
37.2
48.6
32.8
23.1
37.0
OLMo-2-13B
40.9
44.5
39.5
43.8
44.5
35.0
43.1
OLMo-2-32B
44.6
42.2
44.7
31.2
29.2
Mistral-24B
37.6
40.4
35.9
30.7
40.8
23.7
44.6
High (temperature=1.2)
Phi-4
44.2
42.4
45.0
37.3
37.2
9.6
42.6
Qwen2.5-7B
44.6
45.3
46.4
38.9
44.0
36.2
43.3
Qwen2.5-14B
42.1
43.8
42.1
38.6
41.9
24.9
45.9
Qwen2.5-32B
47.0
44.6
42.6
28.5
47.3
Qwen3-4B
50.6
49.6
52.4
42.4
49.4
16.4
50.7
Table 3: Stochastic ablation results.
E DETAILED RESULTS
E.1 STOCHASTIC ABLATION
Table reports the full results for the stochastic ablation tests described in Section 5.1
E.2 COSINE SIMILARITY DISTRIBUTIONS
The aggregate mean similarity is 0.3009 vs. 0.2782 (95% BCa CI [0.0115, 0.0341]), and the median
gap is +0.0278 (95% (0.0132, 0.0427). Effect sizes are small nontrivial (Hedges') y = 0.25: Cliff's 8 0.149. i.c. = 0.575). One-sided tests aligned with the directional hypothesis remain significant after Holm correction over six metrics (PWekch = 2.96 10-4, PMwU 1.31 x 10 Ppem mean 6.00x10 100xx00-), The position-wise means (Fig, 2ap also show
Pigh 8.09 positive gap at every window P1 >p5 that attenuates with depth, consistent with error accumulation during generation. All five positions
significant under one-sided Welch t and Mann-W Whinney U with Holm adjustment (p < 0.01 in each case). A one-sided KS test also indicates right-shift dominance for SEEN at all positions after correction (four of five with p 0.01). The aggregate ECDFs (Fig. (b) further visually corroborate uniform rightward shift for SEEN, indicating higher similarity across the support.
E.3 Do UNAUDITED MODELS HAVE A SIGNIFICANT ADVANTAGE?
Table pressents the full significance test between the performance of audited and unaudited models. To better isolate potential memorization effects and avoid confounding from the model's general ability, pair each OLMo-2 size variant with unaudited peers that perform similarly general
24

---

Published as a conference paper at ICLR 2026
Audited
Unaudited
Mode
n
Mean
CI
Pr.
pw
Sign p/ +frac
OLMo-2-7B
LLAMA-3.1-80
Pair
14
-4.08
8.36, 1.33]
0.137
0.084
0.013 / 0014
OLMO-2-13B
GEMMA-2-9B
grp-mean
12
+1.22
0.90, 3.30
0.296 0.239
0.774 / 0.58
QWEN-2.5-7B
OLMo-2-13B
GEMMA-2-9B
B: pooled 24
+1.22
0.992, 3.59]
0.305
0.638
1.000/ 0.50
QWEN--2.5-7B
OLMo-2-32B
QWEN-2.5-14B
A: grp-mean 12
+1.19
[- 3.333, 5.71]
0.635
0.556
0.388 0.66
QWEN-2.5-32B
GEMMA-2-27B
GEMMA-3-27B
GPT-40-MINI
O1M0-2232B
QWEN-2.5-14B
B: pooled 60
+1.19
[-1.09, 3.45]
0.309
0.337
0.093 /..62
QWEN-2.5-32B
GEMMA-2-27B
GEMMA-3-27B
GPT-40-MINI
Table 4: Prompt-level paired significance tests for audited OLMo-2 vs. unaudited baselines. Mean is (OLMO - comparator) in accuracy points, and positive favors OLMo; Cls are bootstrap: Pt is one-sample paired (-test prompt-wise differences; Pw is Wilcoxon signed-rank; the last column reports two-sided sign-test and the fraction of positive differences (+frac).
domain benchmarks (according to the OLMO-2 report). For the second and third group of compar- isons, exclude model performance the ECCENTRIC AUTOMATIC PROMPTS, because it triggers GEMMA-2-1 2- and CEMMM-2-27B, making their performance an outlier.
For OLMo-2-7B LLAMA-3. 1----B (all prompts), the mean difference is negative and the sign test indicates LLAMA-3 3.11-...8 1-BB wins on most prompts (p = 0.013, 14% positive in favor of OLMo), while mean- and rank- rank-based tests not significant. For --M0--213BB vs [GEMMA-2-0
QWEN- cooommatc_rrrrotssssts excluded, OLMO shows a small positive mean (A) pooled (~r +1.2 pts) that is not statistically significant whether compared the group GEMMA-2-27B across members (B). For OLMo-2-33 -232-BBBBB vs (QWEN-2.5-14B, QWEN-2.5-32B. GEMMA-3-27B. GPT-40-MINIJ ) with is again small (~ +1.2 pts) and non-significant; sign tests trend toward OLMO but do not cross 0.05 the pooled analysis. Overall, peompt-level paired tests do not support significant mean difference between audited and selected unaudited models.
2.5-7B) with
couutmmticc_Paae Prompps excluded, the average edge
E.4 EVENT-TYPE PERFORMANCE COMPARISON DETAILS
Overall, models confabulate "role" events best, then "relational", "agentive", "observational", with "cognitive" lowest. The EVENT_ TYPE hint preserves this ranking and disproportionately boosts "role".
Event type
No hint
With hint
Combined avg
role
0.422
0.473
+0.051
0.448
relational
0.394
0.434
+0,040
0.414
agentive
0.370
0.385
+0.015
0.378
observational
0.357
0.367
+0.010
0.362
cognitive 0.243
0.254
+0.011
0.249
Table 5: Overall accuracy by event type across all models. A = (with hint) - (no hint). The combined average is the mean of the two conditions and matches the main-text ordering.
Representative model patterns align with the aggregate picture (e.g.. GPT-4o/5 improve notably on "role"; several Qwen variants and GEMMA-3-27B show smaller negative hint effects some non-role types). further supporting the claim that biographical/role- content is easiest to reconstruct and benefits most from type eues, while internal "cognitive" states remain the hardest.
25

---

Published as a conference paper at ICLR 2026
OLMo-2 (audited)
Other (unaudited)
Event type
hint
With hint
No
hint
role
0.359
0.418
0.439
0,489
relational
0.372
0.420
0.400
0,438
agentive
0.336
0.362
0.380
0.391
observational
0.316
0.329
0.368
0.378
cognitive 0.224
0.248
0.248
0,256
Table 6: Group-level accuracy by event type.
OLMo-2 peaks on relational and is more uniform across types (with-hint spread z 0.420 0.248 = 0.172), whereas unaudited models peak on "role" (spread ss 0.489 0.256 = 0.233).
E.5 EVENT/TIMELINE LENGTII EFFECTS
Longer event descriptions correlate with modestly higher accuracy: longer timelines correlate with lower character- level accuracy. Hints help across quartiles but do remove the negative timeline- length effect.
Event-length quartile
No hint ace
With hint acc
(shortest)
0.334
0.347
Q2
0.361
0.399
Q3
0.397
0.426
Q4 (longest)
0.466
0.500
Table 7: Overall accuracy by ovent-description length (words). Spearman p = 0.090 (p < 10 10- Accuracy increases monotonically with length; hints consistently add lift.
Timeline-length quartile
No hint acc
With hint acc
Q1 (shortest)
0.414
0.481
Q2
0.456
0.503
Q3
0.400
0.428
Q4 (longest)
0,343
0,352
Table 8: Overall character-level accuracy vs. #events per character. Spearman -0.173 (p 9.05 x 10 10-39) 39). Accuracy declines with longer timelines in both conditions.
The event-length effect is monotone across quartiles and holds within both audited and unaudited groups. The timeline-length penalty persists under hints, indicating that long-range narrative depen- dencies remain challenging for current LLMs despite local-type cues.
E.6 EVENT LOCATION DETAILS
Accuracy highest at the beginning timelines, then the middle, and lowest at the end; averaging across hint/no-hint conditions preserves this ordering. Audited (OLMo-2) models exhibit smaller begin->ent drop than other models.
The begin- middle-end ordering holds across families; the smaller OLMo-2 drop is consistent with reduced reliance on memorized gpening-biograng facts and a more uniform use of local reasoning across the timeline.
F FULL CRITICAL CONFABULATION WORKFLOW
Figure details the full workflow of critical confabulation. This work mostly focus the "Character Timeline Fragments" component (the known unknowns), and future work will attempt the more challenging task of lacuna detection (the unknown unknowns).
26

---

Published as a conference paper at ICLR 2026
Group
p (event length)
(timeline length)
OLMo-2 (audited)
+00093 (p x.23 10 124)
-0.198 (p 1.54x 10 12)
Other (unaudited)
-0.089 (p=0)
-0.165 (p = 3.43 10 10-28
Overall
-00090 (p=0)
-0.173 (p 9.05x11009
Table 9: Spearman correlations summarizing length effects. Positive for event length and negative p for timeline length are small in magnitude but highly significant.
Position
Begin
Middle
End
Overall (avg. over hints)
0.450
0.381
0.337
Table 10: Overall accuracy by position (begin/midddle/nnd) averaged across EVENT_TYR TYPE hint conditions.
Archival Documents
Character Timeline
Output Storles
Lacunas/ Omissions
Fragments jriestnn program
Character Metadata
Crose-Referenced Sources
Historical and Cultural Contexts
Generation
Johnaon's i appearance
"aaaram economic socal conditons the 1910s" "Germnn propagendo aimed Affican Americnn soldiers
Extraction
speethes activists
Other Blaak Parther leters and ediovials that Johnson wote
Figure 5: Overview of the full critical confabulation workflow for reparative storytelling.
Group
Begin
Middle
End
A(Begin-Eii End)
OLMo-2 (audited)
0.399
0.359
0.313
0.086
Other (unaudited)
0.464
0.387
0.344
0.120
Overall
0.450
0.381
0.337
0.113
Table 11: Group-level positional accuracy (averaged over hint/no-hint), Audited models show smaller begin- -nd drop than others.
27


Reviews:
[{"review_id": "kq8GNmuTVB", "reviewer": "Reviewer_Qjc3", "summary": "This paper uses LLMs’ predictions in a narrative cloze task to show that LLMs can plausibly speculate missing information about historical figures. The authors relate this use of LLMs to “critical fabulation”, a practice in African American studies where scholars repair missing information in archives through storytelling.", "questions": "Could you provide some concrete examples of knowns unknowns and unknown unknowns?\n\nHow are models’ confabulations considered “evidence-bound” (line 90, line 105, line 163) if we don’t know what kind of evidence they are using for their predictions? Are models really evidence-bound if they are making up information that may not exist in their pretraining data? \n\nThe text in Figure 1 is too small to see unless one zooms in a lot in a digital pdf viewer. Ditto for axes labels in Figure 2. In Figure 1, it’s unclear what the text saying “[MASK]” is doing in the figure. \n\nThe width of Table 2 exceeds the margins of the page. Also the caption of this table could benefit from a one-sentence takeaway for people who skim the paper. \n\nLine 97: the citation for Chambers & Jurafsky is in the incorrect format, e.g. \\citep{} instead of \\citet{} maybe? \n\nLine 253: How did you know if a document was in English or not? \n\nLine 350: What made the three annotators qualified to do this annotation task? Who were these annotators? Ditto for the annotator in line 416. \n\nDo you have to use “hallucinations” repeatedly to describe LLM behavior in your use case? Though this word is now very common and normalized in AI research, it still anthropormophizes LMs and diminishes the words’ origins as a serious human mental illness symptom. It is jarring to see this word upheld so prominently in a “social good”-related paper.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "I feel like this paper is so unlike most of the ICLR paper I’ve seen in recent years, especially with its use of historical archival data and its engagement with African American Studies. I also appreciate the authors’ thoughtfulness around experimental decisions.", "weaknesses": "Though this work is a rare interdisciplinary blend of technical and substantive work, I worry that the way the authors explain their methodology and results may be a barrier for this work actually being useful for “social good”. That is, true “social good” should make research accessible and communicable to the communities associated with the datasets involved. Otherwise, such work is at risk of being extractive. Even as someone with a technical background, I found that there were some parts of the paper that I had to read very very carefully to understand. The authors do not have to necessarily decrease the technicality of their work, but to reconsider how they communicate and illustrate their methods and key takeaways. \n\nThe authors use a lot of paper space to justify why OLMo-2 is their model that they want to focus their analysis on, and why auditing for data contamination is important, only to find out that memorization might not be an issue because OLMo-2’s performance isn’t that different from its unaudited peers. I wonder if the structure/narrative of this paper could be reorganized so that the amount of attention dedicated to certain concepts is more correlated with the potential impact of findings. The main paper is very dense already; some things could be deprioritized into the appendix.\n\nIn addition, I found that the beginning of this paper to introduce the interesting idea of “critical confabulation” and its implications, but the paper ends abruptly without reconsidering how some of these earlier themes may relate to the model-evaluation-based findings of this paper. On a related note, I am curious what differentiates a “critical confabulation” from simply a “confabulation” – how do you know that what models are predicting is in any way “critical”? \n\nThis paper, with its “social good” framing, repeatedly suggests that their use case of LLMs is desirable for humanities scholars. I totally understand why critical fabulation is useful for scholars. However, I remain unconvinced as to why humanities scholars would actually want this process to be LLM-ified, rather than relying on their own expertise and lived experiences. Is it because an LLM could be useful for proposing a range of plausible ideas? I worry that you are making a tool just to see if you can, rather than actually fulfilling a sincere need. \n\nA minor note: the title of this paper is too vague given the paper’s scope, and may not help this paper reach relevant audiences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7mSixRrcnx", "reviewer": "Reviewer_kuNP", "summary": "The authors propose critical confabulation: using constrained LLM “confabulations” to fill archival gaps in ways informed by humanities practice (Hartman’s critical fabulation). They operationalize this as a narrative cloze task over a Black-writing archival collection (BWTC): mask events in a character timeline and ask models to generate plausible evidence-bound fill-ins. They evaluate a range of audited open models (OLMo-2 family) and baselines (Gemma, GPT-4o, etc.) under prompting conditions intended to elicit controlled hallucination. Automatic metrics (cosine similarity, retrieval overlap etc.) and some qualitative examples are used to argue that constrained confabulation can produce plausible, useful candidate narratives.", "questions": "Have you thought about changing the temperature of LLMs to nonzero? Can you set seeds for the LLMs to still control reproducibility?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "I think this paper raises a very interesting and socially meaningful problem, and I love to see how people connect LLM capabilities to social good in unexpected ways. I think in order to do that, the authors have to really make sure that the task they choose for hallucinations should not be contaminated, and much of the heavy lifting in this paper is done at finding that task and data, and the authors have accomplished with careful justifications of data contamination (Section 3.2). I also love that the authors have spent great efforts in curating the dataset and diverse ablations with different prompting strategies. I also like that the paper even tests and analyzes settings between unaudited models.", "weaknesses": "I think the paper would benefit from more qualitative examples and analyses, like showing the failure modes of LLMs and try to analyze the hard data where most models are unable to get them right. In addition, the presentation can be improved. For instance, many figures in the paper have really small fonts and are very hard to see. Figure 1 can also be more dense and displayed with larger font, as I have to zoom in on a laptop to figure out what is being displayed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dwjiJRRP8s", "reviewer": "Reviewer_ubaQ", "summary": "The paper presents an LLM-based methodology for inferring historical narratives, given some historical grounding information. First, the paper establishes the epistemological roots of critical fabulation to situate the utility of their method. The paper then details the dataset used and how that dataset was cleaned to prevent contamination of the data with the model’s training data. Then, the paper tests how well LLMs can perform a confabulation task of reconstructing narratives. The paper finds that while LLMs are performant in this task, there is still a lot of room for improvement in future work.", "questions": "1. For curating the list of unseen names, how did you disambiguate between different personalities with the same name?\n\n2. Where do the event types come from?\n\n3. How did you deal with differences in time scales within a narrative trajectory (e.g., some events might tightly cluster together with long gaps of time between event clusters)?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The work is very novel and has very good quality in its empirical grounding. For novelty, the paper applies an LLM to a novel domain, namely the study of history. I am not aware of many applications of AI or machine learning to this domain. Additionally, there are also few works that exploit the hallucination/confabulation ability for academic pursuits. Many academic pursuits are based in facts and substantiated claims, so the use of the creative part of an LLM in the academic domain is both counterintuitive and novel. \n\nFor the quality of the paper, the data cleaning procedures are very thorough and detailed. The paper does a good job of controlling for things like data set contamination. Additionally, the paper also considers several variations in prompts and masking of events to fully understand the performance contours of LLMs for the narrative closure task.", "weaknesses": "The paper could benefit from some clarity enhancement. For example, I have some trouble picturing the differences between the validation cloze tasks. I think paper would benefit from providing some concrete examples of full timeline $\\rightarrow$ partial cloze set up $\\rightarrow$ n-gram cloze set up (maybe in an appendix). Additionally, it's not clear why some of the thresholds are what they are. For example, when extracting name candidates, why do you take the top 10,000 persons, and why is less than 51 the frequency to keep? Outside of some clarity enhancements that could be made, I think the method would be better if there were some sense of certainty in the predicted events. As seen in this research, this is a challenging task. And, previous research has found humans tend to overestimate AI’s capabilities. So, when these two things combine for a historical task where the confabulations are attempting to fill in for historical truth, I worry the method could result in bad or even harmful confabulations that could distort the historical picture for humans more than the current, partial view of history for unseen peoples and classes of people. Thus, I would surmise that having some estimates of confidence could help humans to better judge whether the confabulations are good or not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)