FictionalQA: A Dataset for Studying Memorization and Knowledge Acquisition
Authors: John Kirchenbauer
Natjanan Mongkolsupawan
Yuxin Wen
Tom Goldstein
Daphne Ippolito
Venue: ICLR 2026 Poster (2026)
Keywords: large language models
memorization
knowledge acquisition
datasets
Primary area: foundation or frontier models, including LLMs
TL;DR: We develop a synthetic dataset that enables us to study memorization and generalization of factual knowledge in LLMs.

Abstract: When language models are trained on textual data, they acquire both knowledge about the structure of language as well as knowledge of facts about the world.
At inference time, their knowledge of facts can be leveraged to solve interesting problems and perform useful knowledge work for users.
It is well known that language models can verbatim memorize long sequences from their training data.
However, it is much less well understood how language models memorize facts seen during training.
In this work, we propose a new dataset to specifically empower researchers to study the dual processes of fact memorization and verbatim sequence memorization.
The dataset consists of synthetically-generated, webtext-like documents about fictional events, as well as question-answer pairs about the events.
We conduct training experiments showing how synthetic data about fictional events can be effective in teasing apart different forms of memorization.
We also document the challenges in effectively building realistic, fictional synthetic data.

FICTIONAL QA: A DATASET FOR STUDYING MEMORIZATION AND KNOWLEDGE ACQUISITION
John Kirchenbauer, Natjanan Mongkolsupawan, Yuxin Wen
Tom Goldstein,
Daphne Ippolito
University of Maryland, Carnegie Mellon University
ABSTRACT
When language models are trained on textual data, they acquire both knowledge about the structure of language as well as knowledge of facts about the world. At inference time, their knowledge of facts can be leveraged to solve interesting problems and perform useful knowledge work for users. It is well known that language models can verbatim memorize long sequences from their training data. However, it is much less well understood how language models memorize facts seen during training. In this work, we propose a new dataset to specifically em- power researchers study the dual processes of fact memorization and verbatim sequence memorization. The dataset consists of synthetically-gennraaee webtext- like documents about fictional events, as well as question-answer pairs about the events. We conduct training experiments showing how synthetic data about fic- tional events can be useful for studying different forms of memorization. We also document some challenges in effectively building realistic, fictional synthetic data.
INTRODUCTION
is well-known that language models memorize some of their training data. Sometimes memoriza- tion takes the form of verbatim memorization where exact sequences of tokens seen during training are likely to be outputted by the large language model (LLM). Verbatim memorization ranges from the memorization of short common phrases (e.g. "the cat's out of the bag") to multi-paragraph excerpts from books or articles. Factual memorization is another form of memorization, in which facts about the world (e.g that cats see better in the dark than humans because their eyes have more rods) are learned as representations that can generalize to diverse downstream tasks. While sequence memorization may or may not be desirable depending ILM has memorized, generalizable fact memorization is almost always considered a desirable trait in LLMs. For example, user might reasonably expect be able to ask LLM "Why can cats see so well in the dark?" and get correct answer, even if the knowledge to answer this question was only ever seen during training as part of Wikipedia-styLe article about cat eyes.
the length and nature of the sequence the
The phenomenon of verbatim memorization has been well studied; the work by Carlini et al. (2019) serves as a canonical example in the domain of language models. However, we understand less about how language models memorize facts such that they are capable of using learned fact for novel tasks at inference time. One challenge with studying the process of fact memorization during training is that it is very difficult to quantify how often a fact actually occurs during training. Prior work has studied the correlation between how well an LLM can answer questions about named entities with the frequency the named entity occurs in the training data (Kandpal et al., 2023). Others have trained very small models exclusively on synthetic biographies and then measured when the
1 "Tii poses challenges for trying apply unlearning techniques remove individual atoms of knowledge. Additionally, when models posses such capabilities. an inherent risk is copyright infringement. That being said, verbatim memorization, or exact reconstruction of training data is the primary issue for legal and copyright risks, not fact memorization. As our focus in this work is the latter, we do not discuss these topics any further in this work. See Iee et al. (2023); Cooper and Grimmelman (2024) for nuanced treatment of generative models and intellectual property.

---

Seeds
Fictshents
Eakitoe:
Ereito::
Sudger Alky, la Agel 530,
and extrrnsntal setobee
aetper Alky Lattoper, 200,, trio
Evente:
2611 Domide traafornake
keed is the Millica an Merne 3HG
Goodipilia of ohd Sod
Leokeefed bebes depinees
ZR la the vilmat day N0 thr Teppet Liak es secoyaaco
Loaebee
Caree the
Alpy
pimeewecs
Soparta
Lalqqer Pavicenacio) Chipp
carter
Tiuse
Apell Duz encrads
Expeliessi Apii duee Telasigoe Auado Dadagle'ss loonatio
Base Iuap' Seok
Documents
TT Orce Mendl of Insier Ally
Cheved Alboy bir
CrCrppary Newee (oia In
Cie Bloncea Takee Milligh
Domagi Traderacci Spacia
Enlgaatee Jearny
werd ovcrupht beper gankens
Minuaee Antce Pleaesite a Cllas
mo pirkert
Hills, casz makeed
astad eatn bos
And 285 La3"
Gibanrrd rallints Cuape
"Antedial le Chamtin Ofler
Janty thaseah atrutt thut paleer showngee New
coda the Apd tha thei atalled
GRe
2000, DAPT
poesennee, odikdd Strecked
codatt
allky Boock
"Saciee Prwwed Pepprr"
proders donzand teoerd
wolhe and souad
the vile ansalaal
tand be spukite
of aederrie edectpeell
costraltng
wameer the agrul
filgg Duoke Yoa,
oderaala coccilied
honore thi tadk bio tede de thr putab bepaeda madee mecilggee
ree-rnitinn price ced retraane
ddgget Dute Ploeestee" 2012
Separtall Todia Backogton Ou- Setwark
tilee Fnvioeeerdd dite
stade cature axpooss
extuaa
protecth pablic
vepa Thmnldy manningg in 2017 your Capoaatio slona
revoedeen dovel snmetiing perkeeell aagedd Armes ande
"Sucal"
"Baad"
Q&A
7500
Wied the grwaey Inaper Aloy"
Miwx NEak Ale Dcnee
Waeer Dede tren appeact
Segutta phaternair
Apst. 203
Dran Eatedtaa Colactio
Milbriee
Afler dedaplantiinne--
Figure 1: An illustration of the hierarchical structure of our fictional dataset. Diagram indicates how seed events are used to generate fictional documents in various styles and how questions are derived from those documents. Small liberties taken in cropping and whitespace of the example texts for visualization purposes.
ability answer biographical questions appears during model training (Allen-Zhu and Li, 2023a). Prior work has also sought to insert canaries (e.g. social security numbers or email addresses) during training and then check whether the model capable generating the canary string (Carlini al., 2022b). In this paper, we demonstrate how realistic, synthetic data about fictional events can be used study the training dynamics of both fact and sequence memorization.
One of the main contributions of our work is the development of a dataset generation pipeline for producing corpora of documents about realistic but fictional events. While the textual styles and the statistical distribution of words and phrases in our data are similar to that of natural pretraining
corpus, we construct prompts which produce events with made-up people, places, and events. These dual characteristics of realism at the surface-level and fantasy at the content-level enable us to study the traits leading to memorization in a laboratory setting, with greater assurance that the facts con- tained in the data do not interact with any other knowledge in the pretraining corpus. In addition, the data pipeline we propose is unique in that it is a "living asset," meaning that we can regenerate fresh dataset for future experiments, and other researchers tweak and repurpose parts of the recipe to suit their needs and explore other research questions than the ones we specifically discuss in this work.
2

---

In summary, contributions
1. We produce a clean dataset for memorization studies. Our FictionalQA dataset has some desirable properties that other datasets do not such factual disjointedness from the real world combined with plausible webtext-like surface forms. It also includes associated question and answer pairs.
2. We measure knowledge transfer between documents and questions about the facts con- tained in said documents, in a tightly controlled setting. We are able to observe reliable transfer effects in both validation loss and Q&A accuracy, but certain results suggest that the model could be relying on the distribution of the fictional training data rather than the atomic facts within it (see Figure 8).
3. We demonstrate that the conditions under which verbatim memorization occurs may not coincide with conditions where factual memorization is more likely. We expect this is at- tributable to fundamental differences in how and when overfitting and generalization occur in machine learning.
4. We observe that training the most succinct, declarative surface form of fact might not result in the fastest knowledge acquisition. The experimental setting which we see the least improvement in Q&A accuracy is when training on the structured lists of fictional events and facts; when training on the more diverse documents we see increased memorization of the facts they contain.
PRELIMINARIES
In this work, will discuss various types memorization phenomena exhibited by LLMs. We'll use the terms "text" and "sequence" interchangeably to refer to either the text strings or the token sequences representing text data during LLM training and inference. To describe and characterize memorization, we generally adopt the established terminology in the literature while extending it in specific ways suit our particular needs.
2.1 WORKING DEFINITIONS OF MEMORIZATION
Our work focuses on three aspects of memorization. First, we consider sequence memorization: the ability of an LLM to generate sequence of tokens which was seen during training. Sometimes, sequence memorization is measured approximately; that is, a sequence is considered memorized the LLM can produce a close match (Ippolito et al., 2023). However, inition of verbatim memorization, measuring whether it is possible to reconstruct training data in exactness, If some contiguous sequence of training tokens is perfectly reproduced by the model, say it has been verbatim memorized. Following (Carlini et al., 2022b), we measure verbatim memo- rization by dividing training data sequence into prefix and suffix, and then checking whether the LLM can generate the suffix when prompted with the prefix.
opt use the stricter def-
On the other hand, if the underlying meaning and factual content of model generation is the same as some training sequence, but the surface text is completely different, then we will refer to this as factual memorization, or fact memorization. The model has learned the semanties of the training sequence and is able to generalize it to new settings. We evaluate factual memorization by assessing whether an LLM can answer questions about facts it has only seen as part of documents. Finally, the meaning or factual content of reconstructed text is different than training sequence, but the surface form of the text is formatrnfo overuse of specific words and phrases, etc.-then we will call this stylistic memorization.
All these forms of memorization can co-occur with each other. However, sequence memorization (especially when it is verbatim) is the strongest form of memorization we measure. Very often facts and styles are learned by the model without the occurrence of any verbatim memorization of training documents containing the fact style. In this work, we specifically interested in learning what takes for a fact to be memorized and contrasting this with the conditions that are known to cause verbatim memorization of training sequence containing the fact.
3

---

2.2 KEY RELATED WORK
Large language models have been shown to verbatim memorize parts of their pretraining data in many different settings. The most widely corroborated result across this body of literature is that sample repetition during training reliably increases extractable memorization (Carlini et al., 2019; 2021; 2022b; Biderman al., 2023a;b; Huang et al., 2024). Towards understanding factual mem- orization, seminal work by Kandpal et al. (2023) showed a clean relationship between entity co- occurrence in training corpus and test time associative ability between those entities. Prior exam- ples of datasets constructed for related purposes include the synthetic biographies dataset developed for use in Allen-Zhu and Li (2023a) and later reused by Zucchet et al. (2025) to study knowledge acquisition, the Fictional Knowledge dataset (Chang et al., 2024), the TOFU dataset specifically cre- ated to study unlearning (Maini al., 2024), and the New News dataset (Park et al., 2025). Recent work on generating synthetic data for instruction tuning also devises prompting strategies that in- crease diversity and coverage of the generator model's output distribution (Chen al., 2024; Zhang et al., 2024) and we employ similar techniques in our pipeline.
On the sub-topic of knowledge acquisition, we would like to draw attention to particularly related as- pects of certain existing work. First, we remark that Zucchet et al. (2025) uses the term "knowledge" in roughly the same way as we use the term "factual memorization". They also study observables as a function of training steps, rather than reporting single static point estimates, to illustrate the dynamics of the memorization and knowledge acquisition behaviors. We believe their methodology could be repeated using our dataset in place of the synthetic biographies from Allen and Li (2023a) to yield further insights. Second, Park et al. (2025) also curates synthetic news-like articles but specifically analyzes the gap between knowledge acquisition via finetuning versus in-context learning, and proposes some modified tuning strategies to minimize this gap.
The aforementioned studies similar in spirit to ours but their data constructions, research ques- tions, and findings are all slightly different but generally complementary. An even more extensive survey of the relevant literature is included in Appendix C but here we can succinetly summarize the novelty of our dataset by enumerating the qualities that differentiate it from existing assets:
Webtext-like styles We produce variety of realistic webtext-like document styles that could be incorporated into a pretraining corpus rather than relying on simple fill-in-the-blank templates which produce more artificial and formulaic results. The documents in TOFU are generated using fil-in-the-blann template, and the synthetic biographies from Allen-Zhu and Li (2023a) are also quite templatic though a generative model is involved.
Size and realism Our dataset is larger than existing resources and specifically avoids science- fiction/fantasy topies (something we specifically avoid, see Appendix D.2). Though not fantas- tical, Park et al. (2025) produce a significantly smaller dataset due to relying on manual curation of articles and questions (theirs includes only 75 hypothetical news articles and 375 downstream questions) and Chang et al. (2024)'s data heavily features futuristic scenarios like interstellar travel.
Documents + Q&A We construct both documents and question and answer pairs designed to test a LLM's ability to generalize the information in the documents whereas Maini et al. (2024); Chang et al. (2024) basically provide or the other. The documents in TOFU are not part of their release data, just the questions and answers, and Fictional Knowledge provides "probes" but these not formatted like trivia questions and answers, but rather as "completion-y" prefixes with an entity suffix.
3 DATASET GENERATION PIPELINE
In Figure I, we illustrate examples of each part of the fictional dataset, and in Section 3.1, we describe how to access to the complete dataset. We utilize 0PT-44-2024-08-06 (Hurst et al., 2024) throughout all generation stages. To control generation diversity. we apply different temperature settings at each stage. Specifically, we use temperature of 1.0 for Seed events and Fictions, while we use 0.7 for Fictsheets and 0.1 for Fictional Q&A. Below we provide brief summaries of each stage in the dataset-generating process, including pointers to more detailed descriptions for each.
4

---

Seed events short premises that sketch out the basic details of fictional scenario event. To in- crease the diversity and uniqueness the generated documents, the prompting strategy injects some unique words and year that the model should use in each seed (additional details in Appendix D.2).
Fictsheets are larger, structured outlines that enumerate plausible details such as people. places, and other conerete entities (see Figure 1) entailed by each seed event (additional details in Ap- pendix D.3).
Fictions are fictional documents. Each fictsheet is used to generate documents in the style of news article, social media feed, an encyclopedia entry, a corporate document, or blog post. We choose these particular styles as they realistic archetypes of different types of content one might find in (cleaned) webscrape and we choose to generate multiple distinct styles for each seed event to study the impact of surface form diversity on knowledge acquisition (additional details in Appendix D.4).
Fictional Q&A pairs are created about each event. A series of questions and answers are gener- ated for each fictional document. The prompting specifically directs the model to make the question unambiguous and structures the questions, answers, and declarative form of the fictional fact (ad- ditional details Appendix D.5).
Q&A Annotation is critical later stage in our pipeline where we determine whether or not a ques- tion is "infeasible" without access to supporting fictional data; we try to ensure that the questions are not answerable by powerful language model that has never even seen the fictional documents. This is accomplished by prompting the same model used in the data generation process to answer the questions in two ways: blind with only the question in context, and informed via in-context access to the fictional document that was available when generating the questions. We provide more details about this process well our deduplication postprocessing step in Appendix D.5.
Multiple Choice Question (MCQ) reformatting is a final postprocessing step we perform to support subset of evaluation experiments. We reformat the fictional question and answer pairs as multiple choice questions such that we can evaluate ranked choice accuracy; this post-hoc procedure detailed in Appendix D.6. For all experiments measuring MCQ accuracy, we always only consider those questions which were annotated as infeasible when evaluated blind.
3.1 DATASET RELEASE
Our dataset is hosted on the Hugging Face Hub and provide the complete outputs of the pipeline as a structured dataset with hierarchical keys. In Appendix D we detail how the different components of the data are organized, and how they are linked together via our system of unique keys.
Dataset: juk_rchhebbaerr/iicciooaaa
Training Splits: www.:pacennaeee//ctccooooaaaaaaa
Generation and Post-processing Codebase: www.rrchenbauerr/iiiiiiioo. Sccchasss.kkccheebeeeee/c/toooo
In order study the loss dynamics and differences between documents included in training and those held out for validation, we construct splits under various criteria. We are then able to measure knowledge transfer via model's improved ability to predict the tokens the validation documents after training on the related but non-identical documents in the training set.
Event Split: All the material corresponding two-thirds of the seed events is placed in the train set with the remainder placed in the validation set. When referred to as "Event Split", the training and validation texts are the fictional documents generated from the seed events. For the "Fictsheets" variant, though the same seed event-based splitting criteria is used, the fictsheets are used as the training and validation texts rather than the fictional documents. In this setting, we expect the con- tents of validation set to look very different from the train set, even though the style of the examples may be similar.
Document Split: For each seed event, for each of the 5 document styles we generate for it, we hold out document from each each style and put it in the validation set. We refer to this dataset as since the
"Doc Split" in the experiments. This can be thought of as in in-iistribttion validation documents in the validation set closely resemble- both in terms of content and style-the the documents in the train set.
5

---

For each seed event, train documents in four different styles, and withhold documents from the one remaining style as a validation set. We refer to these as "Style <ABC> Split" noting which document style was held-out as validation in the name. To reduce the total number of experimental settings, we only perform finetuning experiments on the News and Blog held out variants, but include all 5 versions in the released data. Thus, in this split, the contents of the validation set matches the training set, but the style of the text is nut-of-distribution.
Style Split:
Supplemental Data Assets:
We also utilize two additional datasets from prior work during our experiments. The first is a generic, diverse set of standard webtext pretraining data, the Dolma- v1.6-sample dataset (Soldaini et al., 2024), which we use as a source of webtext documents pad out the training batches during finetuning experiments. The other is question and answer dataset about real facts in the world, TriviaQA (Joshi et al., 2017), which we use to measure the impact that tniing on fictional data has on the model's real world factual knowledge. We describe the minor reformatting process for this data in Appendix D.7
EXPERIMENTS
While the the dataset generation pipeline and datasets release together constitute the primary contribution of this work, we also demonstrate some of the types of experiments that can be per- formed using our dataset. For the training experiments, Llama 3.1, Llama 3.2, Gemma I, and Gemma 2 suites (Grattafiori et al., 2024; Team et al., 2024; Gemma Team al., 2024).
the "base" checkpoints from the
When running each training experiment, samples from the fictional dataset are added to each mini- batch such that, in expectation, 5% of the samples are fiction, and 95% of the samples are from generic webtext mixture. Since the batch composition is fixed at every step regardless of how many rows the fiction training split of interest contains, and we repeat each tranche each time we consume all of its samples, the rate at which the fictional dataset is repeated is implicitly function of its total size. To illustrate this, Figure 2 visualizes the rate which the fiction data epoched as a function of the splitting style and resultant sample count under the 5% relative rate constraint."
Fiction Split Sample Count
Fiction Split Epoch Count
200k
10
Base Websest
Sample Count
160k
Fictsheets
Epoch Count
8
120k-
Event Split.
Doe Split.
80k
Style News Split
Style Blog Split
40k
2
Ok
500
1000
1500
500
1000
1500
Training Step
Training Step
Figure 2: Samples seen function of optimization steps (left) and epochs completed function of optimization steps (right) across different splits of the fictional data. Split criteria that result in smaller training sets (primarily the Fictsheets) epoch faster because the relative batch composition is fixed at 5% fiction to 95% base webtext, regardless of the split.
We start with a warmup period of 50 steps before inserting any fictional data. While not a perfect analogue, throughout our tuning experiments, compare loss measurements to loss on a generic webtext mixture to monitor divergence from the base language model's 's training distribution that might be caused by our tuning. We also compute loss on TriviaQA answers to monitor changes in ability to model real factual information about the world. More details on the finetuning setup can be found in Appendix E.
fictional data
TTii results in unbalanced splittings of the data since create samples for document styles than others. Figure 2 helps illustrate how split sizes impact sampling rates during training experiments.
We also experimented with 100% and 50% relative rates, but the higher sampling frequency appeared to result in pure verbatim memorization with observable generalization period which is actually what want highlight with experiments, so use the low rate of 5% for all experiments in the paper.

---

4.1 VERBATIM MEMORIZATION UNDER REPEATED TRAINING
We begin by confirming that finetuning models on our fictional data causes the tokens to be mem- orized verbatim. Figure demonstrates rapid overfitting despite the fact that we are training on mixture of fictional data and base webtext. This implies that the documents are stylistically plau- sible enough under pretrained language model be rapidly learned (in contrast to say random canary tokens or byte strings). However, our observation of near zero completion rates (verabtim memorization) both at step 0, and at all training steps on the validation texts, together confirm that the documents are suitable for controlled memorization studies. The model should only complete significant portions of these documents accurately "iff" is explicitly trained on them.
Doe Split Loss
Doc Split Memorization
100
LJama-3.2-18
Gemma-2B
80
EM Rate %
Loss
3-
Gemma-2-22
60
Llama-3 2-31
Llama-3. 1-8B
10
(Tain)
20
(Vil)
0
0
500
1000
1500
2.5
5.0
7.5
Training Step
Training Epochs
Figure 3: (Left) Loss samples in the training and validation sets function of optimization step. (Right) Exact Match rate when prompting the model to generate the last 50 tokens of of the fictional document function of the number of epochs on all training documents in the Doc Split fictional dataset.
With this initial check out of the way, for all subsequent experiments, we shorten the training du- ration to focus in the more interesting region from about 0 to 500 training steps, well under 5 epochs and well before the strongest models memorize all of the document suffixes. The U-shaped curve in validation loss seen in the left side of Figure 3 indicates a region where generalization via factual and stylistic memorization is possible, and in the experiments to follow, we highlight how our dataset is particularly well-suited to studying this phenomenon in controlled manner.
4.2 SEPARATING MEMORIZATION FROM GENERALIZATION VIA TRAIN/VALIDATION LOSS
Figure 4 demonstrates that there is a strong correlation between model size and how fast the model fits the training documents for both the Doe Split and the Event Split. However, it also shows that there is period during which the loss the validation documents for the split also improves in parallel to the training loss. We also see that the degree to which the models improve on validation split loss depends on the particular splitting criteria. We design our experiment to test the hypothesis that since the Event Split causes fraction of the seed events and their documents to be completely omitted from training. training on the Doc Split since in the latter case, all the fictional event premises are seen some surface form. While the difference between the validation loss minima in the Doc Split and Event Split cases is small, all models exhibit more generalization (lower minimum validation loss) in the Doc Split case than in the Event Split case.
expect to see less improvement in validation loss than when
Figure 4 with Figure 5 illuminates the impact the splitting criteria even further. We see that training on the Fictsheets split's training texts causes almost immediate overfitting and there is little to no observable transfer period where validation loss also improves alongside training loss. This suggests that the cireumstances under which rapid verbatim memorization occurs (train loss heading to zero, but validation loss increasing) are not necessarily the same as those where
Contrasting
"This biconditional is of course formally proven, but stylize the statement in this way to highlight basic assumption not always explicitly stated in prior studies of memorization. Recent work has shown that models can complete parts of samples they were never explicitly trained on (Liu et al., 2025).
'The Fictsheets split is much smaller than the others, there are only 100 to start with, and thus the 66% train on are epoched very quickly. However, the low number of unique examples and low amount of surface
7

---

Doc Split Loss
Event Split Loss
3.0
3.0
Base Wehtext
2.5
2.5
UJmna-3.2-11
Loss
2.0
2.0
Gemma--2D
Gemma 2 2B
1.5
1.5
LJuna-3.22 3-3B
Lluna-3.144
1.0
(TTrain)
0.5
0.5
(Val)
200
400
200
400
Training Step
Training Step
Figure 4: Loss on samples in the training and validation sets of the Doc Split (left) and Event Split (right) function of optimization step. Legend positioning reflects the fact that these y-axes are meant to be compared, contrasting with the style of all other figure pairs.
Fictshetts Loss
Webtext
Base Webtext"
LJama-3.2-1B
2.8
Genma-2R
2-
Gemma-2.2B
Loss
Llama-3.2-39
Llame-3.1-8B
2.6
(Train)
(Val)
200
400
500
1000
1500
Training Step
Training Step
Figure 5: (Left) Loss on samples in the training and validation sets the Fietsheets split as function of optimization step. (Right) Loss on held out samples from the base webtext distribution as a function of optimizer step while training on the Doe Split (eg. the left of Figure 4).
generalization via factual and stylistic memorization of the data will occur (train loss decreasing, but with validation loss decreasing well), which corroborates results in the literature on how surface form diversity aids in knowledge acquisition (Appendix C).
In Figures 4 and 5 also provide series of control and baseline measurements ground and contextualize our observations. "Base Wehtext"" refers to the Llama-3.2-18 model trained on just the base mixture of real webtext under the same hyperparameters confirm that all observed effects are due to the injection of the fictional data, not the base webtext distribution or other artifacts of the finetuning setup. Additionally, the loss on the base webtext distribution for all models visualized in Figure 5 to show that the ability to faithfully model normal webtext is not destroyed by finetuning on the fictional data at this 5% relative rate.
4.3 PROBING FOR GENERALIZATION TO Q& VIA IMPROVEMENTS IN nll(y|x)
In addition tracking loss on the training and validation documents, also compute the models' loss on answers (y) when conditioned on questions (x) concerning the fictional facts embedded in the documents. Figure shows that training on only the fictional documents (not the questions) from each of the splits improves the models' loss on the fictional question and answer pairs, but this is not observed when training on just the base webtext. As control, we also measure the same question conditional answer loss but for real TriviaQA questions and see that the models don't improve at all
form diversity are intertwined and we see this as interesting comparison to make without controling in any particular way for split sizes.
8

---

in terms of answer likelihood real factual question answering data. However, the upward trend is also similar when training on just the base webtext with no fictional data."
FictionalQA Loss nll(ylz)
TriviaQA Loss nll(y(z)
1.44
Bos Welitesd
Fictshoots
3.5
1.2
Eventl Split
1,888
Doc Spli
3.0
Style News Split
1.0
Style Blog Split
2.5
0.8
200
400
200
400
Training Step
Training Step
Figure 6: Loss on the answers when conditioned on questions for the Llama-3. 3-BB model for the fictional questions and answers (left), and for the TriviaQA questions and answers (right) function optimization step, when training on different splits of the fictional data.
We also observe that the split type can significantly impact the amount of answer loss improvement we see. Figure 6 shows that training on the Fictsheets split does not consistently improve Q&A loss. As expected, the stronger factual separation between train and validation samples for the Event Split appears to result in less transfer to Q&A loss, while the more complete coverage of all events in the Doc Split allows for more improvement. Here we also show the result of training on the splits where we hold out all the News style or the Blog style documents and observe that the amount of transfer to Q&A is similar the Doc Split case.
4.4 RECONSTRUCTION OF FICTIONAL FACTS VIA MCQ TESTING
After pretraining only on webtext, or in our case, fictional documents, it is known that even when LLMs can fail to produce answer string exactly, they can can still be used reconstruct the facts in the training data by emitting the information under multiple choice test." To this end, we reformat the fictional question and answer pairs as multiple choice questions (MCQ) such that can evaluate ranked choice accuracy (described in Appendix D.6).
MCQ Ace. (4 choices)
MCQ Ace. (1 choioes)
0.50
0.54
Weistee
Weisten
0.45
Llama 3.2 1B
0.48
Fisthbeets
Grmmma
0.42
Event Split
0.40
Genna
0.36
Doc Split
0.35
Llmma-3
News Split
Llama-331-88
Styrk Blog Split
0.30
0.24
0.25
0
200
400
200
400
Training Step
Training Step
Figure 7: MCQ accuracy over choices as a function optimization step across models (left), and for the Llama-3.1 model across fictional splits (right).
MCQ versus Model Size. Armed with more interpretable measure than loss, in Figure 7 (left) we are able to observe that training on only the fictional documents (not the questions) reliably increases rank-choice MCQ accuracy, and that larger models achieve higher levels of transfer. This shows that the models have acquired some of the fictional knowledge via training and are able to demonstrate it in multiple choice test setting.
'The increase TriviaQA nil(y)x xx) is unsurprising as the 95% base webtext per batch not particularly relevant support for TriviaQA in the way that the fictional documents are relevant support for the the fictional Q&A pairs.
Thii technique was canonically demonstrated in Brown al. (2020)'s evaluation of GPT-3.

---

MCQ versus Document Style. We also see in Figure (right) that the style of the fictional data impacts the amount of factual transfer to the MCQ test format. High diversity splits like the Doc Split and Style splits transfer the strongest, splitting along Event lines hinders learning further, and training on the Fictsheet split causes the least transfer, despite the fact that the model has memorized most of the training fictsheets indicated by near loss (Figure 5).
Probing for a Knowledge Acquisition Mechanism. In the last set of experiments we attempt to disentangle whether or not the models memorize the factual content or the stylistic content, or mixture of both. To do this we try and leverage the disjoint-ness of fictional events in the various training and validation splits to isolate whether more factual information can be reconstructed for questions corresponding to seed events and facts that the model directly trained on versus those it did not train on. Following the different splitting criteria for the fiction documents, we subset the questions that were generated from the specific documents in each of the training and validation document sets and then measure MCQ accuracy on the training and validation sets separately.
We observe that the the improvement in MCQ accuracy appears to be "leaky". Figure shows
MCQ Are. (4 choices)
0.54
Firtsberaa
that the performance sponding to held-out fictional scenarios and events is also elevated "(Val)", albeit slightly less than for questions corresponding to sce- narios that were trained on "(Train)". This performance elevation is observed even though some facts underlying the MCQ's correspond- ing to the Event Split's validation set are ex- pected to have been wholly omitted from the training document pool. This indicates that it is not possible to cleanly differentiate whether the improvements we observe in MCQ Ace. (or question-conditional answer loss in Figure 6) attribute-able to either purely factual or purely stylistic memorization.
the questions corre-
0.48
Event Spitt
0.42
Doe Split
0.36-
(Trrin)
(Val)
0.30
0.24
200
400
Training Step
Figure 8: MCQ accuracy over 4 choices func- tion of optimization step for the Llama-3.1 1--BB model with performance separated for questions corresponding to documents in the training set vs. the validation set for each style split.
What clear is that these improvements caused by training on the fictional data and not other effects as evidenced by the lack of improvement for the "Base Webtext"-onl -only control (Figure 7). All we can conclude is that the models' improved MCQ accuracy after training on the fictional documents in each of the experiments is based on some combination of distributional and atomic learning. To sanity check these findings, we analyze the Event Split MCQs in detail in Appendix B.
DISCUSSION AND CONCLUSION
We believe that the observations that the knowledge acquisition mechanism under standard finetun- ing is not purely "atomic" and that training on the Fictsheets split offers the least amount of vali- dation performance improvement (Figures 5 to 8), are actually more compatible findings than they might seem at first glance. To a human, the clean, markdown-like structure of the fictsheets might appear to be the surface form of the factual information that is the casiest to extract generalizable knowledge from. However, LLMs acquire knowledge through different mechanisms than humans, perhaps relying on distributional features in the text more than anticipated or desired. We hypoth- esize that this set of results has implications for the conditions under which factual memorization under limited repetition is likely to occur, or possible at all.
While in Appendix A discuss limitations and future applications in greater detail, we conclude with a few key remarks here. Constructing fully synthetic "cleanroom" data using LLMs is difficult. We design our prompts carefully to ensure the quality and diversity of the various components in our dataset but still observe a significant number of duplicate questions. The results in Figure 8 also sug- gest that the fictional documents might overlap to a larger degree than desired across seed events and across styles. While we elucidate interesting memorization vs. generalization behaviors through our experiments, more than anything, we hope that our results inspire the use of our dataset for studying topics we do not explore such as machine unlearning and privacy preserving training methods.
10

---

ACKNOWLEDGEMENTS
We gratefully acknowledge the contributions of Max Cembalest, who implemented an early version of our dataset generation pipeline; Christopher A. Choquette-Choo, Matthew Jagiclski, and Avi Schwarzschild, who brainstormed and helped frame our initial research questions; Katherine Lee, who provided invaluable feedback and insights throughout the project; and Sean McLeish, for his proofreading.
UMD researchers were supported by the ONR MURI program, DAPRA TIAMAT, the National Science Foundation (IS-2212182), and the NSF TRAILS Institute (2229885). Commercial support was provided by Capital One Bank, the Amazon Research Award program, and Open Philanthropy. CMU researchers were supported by Cisco Sponsored Research and CMU Cylab Seed Funding.
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. 2023a. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint 2rXiv:2399.13316
Zeyuan Allen-Zhu and Yuanzhi Li. 2023b, Physics of language models: Part 3.2, knowledge ma- nipulation. arXiv preprint 2:t1:::233991444222
Zeyuan Allen Zhu and Yuanzhi Li. 2024. Physics of language models: Part 3.3, knowledge capacity scaling laws. arXiv preprint arXiv:2404.054055
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Kor- bak, and Owain Evans. 2023. The reversal curse: Lims trained on "a is b" fail to learn "b is a". arXiv preprint arXiv:2309.12288
Stella Biderman, Usven Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivan- shu Purohit, and Edward Raff. 2023a. Emergent and predictable memorization large language models. Advances in Neural Information Processing Systems, 36:28072-28095
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023b. Pythia: a suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, ICML'22. JMLR.org.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo Hengrui Jía, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141-159.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, pages 463-480.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. 2022a. Membership inference attacks from first principles. In 2022 IEEE Symposium on Se- curity and Privacy (SP), pages 1897-1914. IEEE
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022b. Quantifying memorization across neural language models. arXiv preprint.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19), pages 267-284.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650 USENIX Association.
11

---

Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang. Youngkyung Seo, Du-Seong Chang, and Minjoon Seo. 2024. How do large language models acquire factual knowledge during pretraining? In The eightth Annual Conference on Neural Information Processing Systems.
Jiuhai Chen, Rifaa Qadri, Yuxin Wen, Neel Jain, John Kirchenbauer, Tianyi Zhou, and Tom Gold- stein. 2024, Genqa: Generating millions of instructions from handful of prompts. arXiv preprint vrXiv:2406.10323.
François Chollet. 2019. On the measure of intelligence. arXiv preprint wuXiv:1911.013::
Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. 2024. Arc prize 2024: Tech- nical report. arXiv preprint arXiv:2412.04660
Christopher A Choquette-Choi Florian Tramer, Nicholas Carlini, and Nicolas Papernot. 2021. Label-only membership inference attacks. In International conference on machine learning, pages 1964-1974. PMLR.
A. Feder Cooper, Christopher A. Choquette-Choo, Miranda Bogen, Matthew Jagielski, Katja Filippova, Ken Ziyu Liu, Alexandra Chouldechova, Jamie Hayes, Yangsibo Huang, Niloofar Mireshghallah, Ilia Shumailov, Eleni Triantafillou, Peter Kairouz, Nicole Mitchell, Percy Liang, Daniel E. Ho, Yejin Choi, Sanmi Koyejo, Fernando Delgado, and 16 others. 2024. Machine unlearning doesn's I do what you think: Lessons for generative ai policy, research, and practice. Preprint, ar 2r26
A Feder Cooper and James Grimmelmann. 2024. The files are in the computer: Copyright, memo- rization, and generative ai. arXiv preprint arXiv:2404.1255 12590.
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannanch Hajishirzi. 2024. Do membership infer- ence attacks work on large language models? arXiv preprint arXiv:2442.008841
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and I oth- ers. 2024. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.00118.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and I others. 2024. The llama herd of models. arXiv preprint arXiv:2407.2178..
Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. 2019. Certified data removal from machine learning models. arXiv preprint arXiv: 1911.0030....
Jie Huang, Hanyin Shao, and Kevin Chen-Chuan Chang. 2022. Are large pre-traing pre-trained language models leaking your personal information? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2048-2047, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jing Huang, Diyi Yang, and Christopher Potts. 2024. Demystifying verbatim memorization in large language models. In Proceedings of the 2024 Conference Empirical Methods in Natural Lan- guage Processing, pages 10711-10732, Miami, Florida, USA. Association for Computational Linguistics.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, and others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.211
Daphne Ippolito, Florian Tramer, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher Choquette Choo, and Nicholas Carlini. 2023. Preventing generation of verbatim memorization in language models gives a false sense of privacy. In Proceedings of the 16th International Natural Language Generation Conference, pages 28-53.
12

---

Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. 2021. Approximate data deletion from machine learning models. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 2008-2016. PMLR.
Matthew Jagielski, Milad Nasr, Katherine Lee, Christopher A Choquette-Chop Nicholas Carlini, and Florian Tramer. 2024. Students parrot their teachers: Membership inference on model distil- lation. Advances in Neural Information Processing Systems, 36.
Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and others. 2022. Measur- ing forgetting of memorized training examples. arXiv preprini ar wrXx:2207.000995
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.
TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large lan- guage models struggle to learn long-tail knowledge. In International Conference on Machine Learning. pages 15797 PMLR.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. In International Conference Machine Learning, pages 10697-10707.
PMLR.
John Kirchenbauer, Garrett Honke, Gowthami Somepalli, Jonas Geiping, Daphne Ippolito, Kather- ine Lee, Tom Goldstein, and David Andre. 2024. Lmd3: Language model data density depen- dence. arXiv preprint Siti::2400.0633111
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka ers. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(33)3321-3326
and 1 oth-
Katherine Lee, A Feder Cooper, and James Grimmelmann. 2023. Talkin "bout ai generation: Copy- right and the generative-al supply chain. arXiv preprint arXiv:2309.088 08133.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8424-8445, Dublin, Ireland. Association for Computational Linguistics.
Ken Ziyu Liu, Christopher A Choquette- -Choo, Matthew Jagielski, Peter Kairouz, Sanmi Koyejo, Percy Liang, and Nicolas Papernot. 2025. Language models may verbatim complete textthey were not explicitly trained on. arXiv preprint arXiv:2503.175: 175144
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and Zico Kolter. 2024. Tofu: A task of fictitious unlearning for llms. arXiv preprint 2611112401.06121.
Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv:9002.0100)
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. 2023. Scaling data-constrained lan- guage models. Advances in Neural Information Processing Systems, 36:50358-50376
Core Francisco Park, Zechen Zhang, and Hidenori Tanaka. 2025. New news: System-2 fine-tuning for robust integration of new knowledge. arXiv preprint frXiv=2505.0181:
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. 2019. White-box vs blx: Bayes optimal strategies for membership inference. In International Conference on Machine Learning, pages 5558-5567. PMLR.
13

---

Ahmed Salem, Yang Zhang, Mathias Humbert, Pascal Berrang, Mario Fritz, and Michael Backes. 2018. MI-leeks: Model and data independent membership inference attacks and defenses on machine learning models. arXiv preprint vrx.v:1806.01246.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE.
Siddharth Singh, Prajwal Singhania, Aditya Ranjan, John Kirchenbauer, Jonas Geiping, Yuxin Wen, Neel Jain, Abhimanyu Hans, Manli Shu, Aditya Tomar, and others. 2024. Democratizing ai:
Open-source scalable IIm training on gpu-based supercomputers. SC24: International Confer- ence for High Performance Computing. Networking, Storage and Analysis, pages 1-14. IEEE
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Au- thur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, and 1 others. 2024. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, and others. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295.
Anvith Thudi, Hengrui Jia, Ili Shumailov, and Nicolas Papernot. 2022. On the necessity of auditable algorithmic definitions for machine unlearning. In 31st USENIX Security Symposium (USENIX Security 22), pages 4007-4022, Boston, MA. USENIX Association.
Kushal Tirumala, Daniel Simig, Armen Aghajanyan, and Ari Morcos. 2023. D4: Improving Ilm pretraining via document de-duplication and diversification. Advances Neural Information Processing Systems, 36:53983-53995.
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, and others. 2024. Livebench: A challenging, contamination-free Ilm benchmark. arXiv preprint arXiv:2406. 19314.
Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, and others. 2024a. Benchmark data contam- ination of large language models: A survey. arXiv preprint 2rXiv:2406.04244.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024b. Magpie: Alignment data synthesis from scratch by prompting aligned 11ms with nothing. arXiv preprint 1rXiv:2406008466
Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica. 2023. Rethink- ing benchmark and contamination for language models with rephrased samples. arXiv preprint arXiv:231 I. 1.0450..
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018 Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security founda- tions symposium (CSF), pages 268-282, IEEE.
Yiming Zhang, Avi Schwarzschild, Nicholas Carlini, Zico Kolter, and Daphne Ippolito. 2024. Forc- ing diffuse distributions out of language models. arXiv preprint 24Xx::4404.0888
Nicolas Zucchet, Jörg Bornschein, Stephanie Chan, Andrew Lampinen, Razvan Pascanu, and Soham De. 2025. How do language models learn facts? dynamics, curricula and hallucinations. arXiv preprint arX.v22503.21676.
14

---

Appendix Table of Contents
Papbl
Table of Contents
1. Extended Discussion of Limitations and Future Applications
2. Extended Analysis MCQs and Figure Results
3. Extended Related Work
4. Additional Generation and Annotation Pipeline Details
5. Extended Experimental Details and Results
15

---

Appendix -apble
Table of Contents
A EXTENDED DISCUSSION OF LIMITATIONS AND FUTURE APPLICATIONS
Troubles with diversity Generating a high diversity of documents, under constraints of strict "fictionality" is difficult, and clever prompting strategies are required to force diffuse distribution out of the data generating model (Zhang et al., 2024), (Chen et al., 2024)). We discuss a few strategies used to increase the diversity of the data generate but alternate aproaches could be devised, and different, stronger models could be used, or employed in a pool under the same prompts to further increase coverage and diversity of the data.
Tradeoff's between ease of scoring and Q&A quality We choose to generate the questions and answers under prompts that constrain the answers to be simple associative relationships, normally with a specific sub-span of the source document where the answer to the question can be found. This helps concretize notions of correctness during scoring, but limits the diversity of the types of questions the pipeline will produce.
Issues with Model-based Post-hoc MCQ Construction We reformat the question and answer pairs from our pipeline into MCQs in a post-hoc manner, and this introduces artifacts (see Ap- pendix D.6 for details on construction). We observe that trivially easy eliminate alternate answers probably cause the model to achieve base accuracies better than chance without actually training on the questions. We also see that, as a side effect of our ranking criteria during construction, multiple answer paraphrases or plausible alternative answers to vague questions end up in the alternates list, potentially upper-bounding the best case accuracy. We discuss this issue the context of the Event Split MCQs in Appendix B and explore an alternate method for MCQ generation in Appendix D.6. 1.
These issues could be ameliorated in future work in various ways such as by creating the alternate lists for these MCQs from scratch during initial question generation (a la Appendix 1. and annotating and curating MCQs by hand for feasibility and difficulty. We note that constructing multiple choice tests that avoid these issues is rich area of study. Prior work has found that models can often be "right for the wrong reasons" " which is one way of summarizing potential confounders in our MCQ setup (McCoy et al., 2019).
Experiments at trillion-token scales In this work we do not pretrain language models from seratch on O(IT) token datasets containing our fictional data questions. It is open question whether data such as ours has any observable impact on the final model when the relative sampling rate of this data drops from 5% to 0.0005% smaller. Are many order of magnitude more epochs or orders more fictional document per seed event required? Must the fictional facts be more unique to be picked up by the model? We leave these interesting, but expensive experiments to future studies with industrial computational resources.
Impact of surface style on learnability Our pipeline embues the fictional documents we gener- ate with a dimension that we do not heavily study: the "styles" of the fictional documents (news, encyclopedia, social, etc.). A subset of our training experiments split the data along style lines, but the impact of style on learnability and knowledge transfer is not explored in any depth.
Machine Unlearning We do not study privacy or threat models specific unlearning or "right to be forgotten" scenarios, however, our dataset is constructed to have properties that could facilitate these types of studies, and could be useful for benchmarking novel unlearning techniques. Test- ing whether an unlearning technique addresses both verbatim memorization and reconstruction via generalization an line of research that our data is particularly suitable for.
Generating fake PII The data we generate is relatively innocuous. It contains mostly milk-toast scenarios in surface styles that one might encounter in an general internet scrape. However, the prompts in our pipeline could easily be reworked to generate data that looks more like personally identifiable information (PII) such as personal details in private message threads, information on bank statements, medical history transcripts, etc. However, given the fact that we aim to present methodology for data driven study of memorization writ large, rather than just the privacy questions (and seeding the generation process for this kind of data without actually generating examples that expose any real PII requires particular care) we leave this alternate use of our methodology to future work.
16

---

Appendix Popeleix-Table
Table of Contents
B EXTENDED ANALYSIS OF MCQS AND FIGURE 8 RESULTS
One goal underlying our dataset construction process (eg. the "Event Split") is to make it possible to identify whether or models finetuned our data atomically or via general distributional properties of the fictional data. theory, this distinction can be made by checking whether they improve on only the questions associated with training docu- ments, or whether they also improve on questions associated with the validation documents. Even from just Figure 4 (right), is already clear that some sort of distributional transfer learning is hap- pening across "Event Split" lines, so seeing some amount of similar transfer between the associated training and validation MCQ examples in Figure is also expected. That being said, whether or not specific model or training algorithm appears to learn more atomically or more distributionally is itself the actual empirical question hand in these experiments. The goal running these exper- iments using dataset is simply report the observations of what appears to be happening in standard training setting, using our data as the probe, rather than to design some training recipe that targets specific outcome.
likely learning factual relationships
With that as context, in this section we perform some additional analyses of the data behind Figure 8. In particular, analyze the sample-wise performance of the Llama-3.1-88 model on the Event Split's training MCQs and the validation MCQs and compare successes and failures at step 0 of training those the final step of training. The goal of this analysis there are any particular defining characteristics of the validation question subset that the Llama-3. 1- 8B model "learned" over the course of training, and whether there any specific distributional link between the training questions the model learns during training, versus the validation questions that simultaneously improved on.
try and discern whether
B.0.1 A NOTE ON DUPLICATED ANSWERS FROM DISTINCT QUESTIONS
We note that our question filtering process deduplicated the questions only on the question text, and not the answer itself. As example, from two different documents based on the same seed event, we can see that two questions were produced that both have the answer "Lake Ypsilon".
question.id: event_000
Question: Where was the first pilot test of the Ring of Silence Protocol conducted?
Answer:
Lake Ypsilon
event_000 eventt.00Lettlllarrrccppppe..... www..o0......uttto.ooo..0. Question: Where was the sound sound-absorbing absorbing moat first implemented?
questionid::
Answer: Lake Ypsilon
However this is artifact of various documents and their questions ending up referring shared concepts in the same seed event even though the way in which the question is posed (i.c. the specific continue present the questions
fact pattern) is slightly different. In the subsequent analyses as subselected for experiments in the main paper for continuity. However, during the analyses we also checked whether this additional deduplication by answer seemed to change any of the statistics meaningfully, but did not observe any evidence of that.
B.0.2 VISUAL INSPECTION OF THE TRAINING QUESTIONS VERSUS VALIDATION QUESTIONS
We first can take look at some samples from the sets of training split questions the model im- proved on as well as samples from the set of validation questions the model improved on. One thing observe is that there are indeed cases where there is more than one similar answer that could plausibly be. These cases can only be decided by specific formatting quirks aligned with the source fiction document such using a full proper noun versus shortened one. This at least suggests explanation for the fact that accuracy plateaus around 50% even for the largest models at the end of training.
17

---

Appendix-Tabee Table of Contents
=== Sample 1 (Index 2832) ===
event id: event 093
fiction id: event 093 style social num 000
question_id:
input: Question: Who supposedly left instructions for the steeple's construction?
Answer:
target.hf: William Linton
target_idx: 2
topk.choices:
Choice 0: Western and Eastern
Choice 1: Raising the Brow: The Brow Society's Legacy
Choice 2: William Linton
Choice 3: The Source
choice.ranking.iit:: [3, 2, 0, 11
choice.ranking.final. [2, 3, 0, 1]
acc.init: 0.0
acc.final: 1.0
== Sample 2 (Index 897) ===
event_id: event_029
fictionid: event_029 wtw.yleeenessnmm....ueeeeeete
event.029.styleeeess.nmmm.
questioniid: event.029.style eve
input: Question: Who is the entrepreneur responsible for the
AI-driven ice cream carts?
Answer:
target_hf: Lila Sorvino
target.idx: 3
topk.choices:
Choice 0: Dr. Lila Harrington
Choice 1: Lillian Abbott
Choice 2: FlavorSync AI
Choice 3: Lila Sorvino
choice.ranking..im
[1, 2, 0, 3]
choice.ranking.fina. [3, 2, 0, 1]
acc init: 0.0
acc final: 1.0
Random questions in Event Split Train for which the Llama-3. 1-8B model initially answers incorrectly, but then answers correctly the final step i.e. "learned". Here see that in Sample (Index 2832) context clues about the entity type most likely eliminate all three other choices, but Sample (Index 897) there three plausible answers all of the correct entity type.
MCQ Examples:
The other consistent feature is that there normally or two distractor choices that most hu- man readers could easily eliminate using question and context clues without needing to look at the fictional document itself. Similar to the aforementioned case of choices that overly semantically similar, the presence of these obviously incorrect distractors suggests partial explanation for why the models achieve better than chance (1/num choices) accuracy at step 0 in the main experiments. However, curiously, the Llaam-3. 1-8B model still often still ranks one of those "easily eliminable" choices in the top 2 indicating that these models' likelihood assignments under the prompting and loss ranking process used to score the model on the MCQs do not always effectively eliminate ob- vious wrong answers in the way good human test taker might."
'To address these issues, we explore an alternate methodology for generating MCQs that do not suffer from these issues much. This version of the MCQ generation procedure and resulting assets are discussed in Appendix D.
18

---

Appendix
Papelei--Table Table of Contents
=== Sample 1 (Index 2535) ===
event id: event 084
fiction id: event 084 style corporate num 000
question_id:
input: Question: What did the diary become symbol of?
Answer:
target.hf: shared suffering
target.idx: 1
topk.choices:
Choice 0: shared suffering and humanity
Choice 1: shared suffering
Choice 2: vulnerabilities
Choice 3: vivid sketches and poignant accounts
choice.ranking.iii::: [2, 1, 0, 3] choice.ranking.fina. [1, 2, 0, 31
acc_init: 0.0
acc.final: 1.0
Sample 2 (Index 1111)
event id: event 036
fiction.id: eventt_3..styye..eeews questionid: www.rt0...s..lleee..............
input: Question: Who discovered the Waterfall Whisper phenomenon?
Answer: target hf: Thomas Bright
target_idx: 3
topk.choices:
Choice 0: tie
Choice 1: Western and Eastern
Choice 2: accustic engineering and psychological principles
Choice 3: Thomas Bright
choicceranking..nit:: [0, 3, 1,
choice.ranking ing..ainning.fall:: [3, 0, 1, 2]
acc.init: 0.0
acc final: 1.0
MCQ Examples: Random questions in the Event Split Validation set for which the 1.lma-3.11888 model initially answers incorrectly, but then answers correctly at final step, i.e. "learned". In Sample 1 (Index 2535), three of the choices quite semantically similar and thus equally plausible, whereas in Sample 2 (Index 1111) there one answer that is more plausible than others though a human might need against the source document be absolutely sure.
Overall, we observe no telltale indicators of some characteristic difference between the questions the model gets correct after training in the training split or the validation split question sets but we acknowledge that our manual inspection is not exhaustive.
B.0.3 ERROR CASE ANALYSIS
For the Event Split's two sets of training questions and validation questions we are able to analyze two different evaluation runs of the same model: one that was performed at the beginning of training, and another performed at the end of training. As shown the examples above, we annotate the two result datasets with boolean indicating whether the model answered each question correctly "initially" at step and/or correctly at the "final" step of training. Then, we subset the data based these conditions (and few others) to compute statistics that try and shed light whether
19

---

Appendix- -ablle Table of Contents
there are any interesting differences on how the model performance changed during the course of training.
Initial Acc
Final Acc 45.31 (899/1984) 43.44% (457/1052)
Train Set
31.91 % (633/1984)
Validation Set
30.51% % (321/1052)
Table 1: The exact numbers underlying the start and end points of the (Train) and (Val) lines for the Event Split for the Llama-3. 1-8B model in Figure 8. These are the same sets of questions and their correctness judgements that we use in the case-wise analysis in this section.
In Table we present the aggregate statistics for the Llama-3.1 1-B model on this set of MCQs separated by train and validation. Then, rather than show an extremely extensive table of all possible subsettings by taking the cross product of all conditions we could think of, we focus on just a few combinations of potential interest and present the results in Table 2. In particular, we use the terms "acquired"""learne"" to indicate subsetting for questions for which the model was initially incorrect, but then ended up answering correctly after the final step of training. We also note a particularly common spurious answer that ended up in a large number of the choice lists "Western and Eastern" (abbr. "W and F"),, and also often ended up ranked as the model's top choice at the first step of training. This is an artifact of our choice ranking process (see Appendix D.6) and could he improved in future work on similar pipelines.
The summary takeaway here is that none of the particular statistics we calculate are notably different between the training questions and the validation questions; all see is that the training split yields slightly elevated correctness levels in all cases. So, under this lens of analysis, it appears that most of the information being learned by the model that helps it correctly answer new training questions it did not initially answer correctly, must also be similarly useful for helping it answer some additional validation questions more accurately. Or rather, as stated in Section 5, the models we train appear to be primarily learning distributionally, not atomically. One could interpret the slight additional improvement on the training split as the fraction of improvement attributable to more atomic knowledge acquisition, but our experiments do not prove this conclusively.
B.0.4 TRAIN AND VALIDATION SAMPLE DISTRIBUTIONAL OVERLAP
Finally, to try and identify any additional links between the training questions and the validation questions that might explain the transfer learning indicated by the improvements in performance on validation questions, we extract all the named entities and whitespace separated tokens (words) from the fictional documents and the answers to the questions. Then, we compute the aggregate overlap between all entities words from different combinations of the training fictions or answers and the validation fictions or answers. These results are presented in Tables to
The main hypothesis behind this analysis would be that the cases where take the set of training questions that only became correct after training (directly "learned") and the set of questions in the validation set where the model also happened to become correct after training (indirectly "transfer learned") could show some elevated level of distributional overlap as compared to all possible subset comparisons. An observation like this would help explain why the model improved on those specific validation questions over others.
However, we find that the entity or word based Jaccard similarity (set intersection over union) for that specific case (bolded row 5), no matter which combination of the fiction texts or answer texts with intersected, never significantly higher than the other cases shown in other rows. While this could simply be too simple of distributional comparison, it does indicate that the specific improvements in validation question performance cannot be explained by relatively high levels of simple word overlap alone. Rather, there is some collection of general distributional patterns that the model is learning from the training documents that is also useful in ranking validation answers more accurately.
20

---

Appendix-Tabee Table of Contents
Case
Train Split
Val Split
Fraction of initially correct answers that still re-
85.62% (542/633)
85.36% (274/321)
mained correct after training:
Fraction of final correct answers that were
39.71% (357/899)
40.04% (183/457)
learned i.c. initially correct:
Fraction of learned questions that also had grade_ine in formeddll
93.28% (333/357)
89.62% (164/183)
Fraction of learnable questions (initially incor-
26.42% (357/1351)
25.03% (183/731)
rect) that were learned:
Fraction of learned questions where
82.63% (295/357)
83.06% (152/183)
natural www.nnsseer..... -in fn f ct i on===:
Fraction of questions that model could have learned but did not where natural answer in fict ion==1:
56.04% (557/994)
58.94% (323/548)
Fraction of learnalle questions where natural answer in Alction==s that
34.62% (295/852)
32.00% (152/475)
were actually learned:
Fraction of learnable questions where natural www.:aaaanswerrrinn.mmmm fit.innfiiiiiooo0 that
12.42% (62/499)
12.11% (31/256)
actually learned:
Fraction of samples with "W and E' as choice:
31.45% (624/1984)
32.41% (341/1052)
Fraction learned questions that had "W and
26.05% (93/357)
29.51% (54/T83)
initial model choice but then demoted it:
Fraction of learnable questions that were learned 20.99% (264/1258)
19.05% (129/677)
ignoring W and demotion cases:
Table 2: Case wise error analysis of Event Split's Train and Test sets of questions for the Llama-3.1 8B model.
Train
Train
Val
Val
Entity Jaccard Similarity
Word Jaccard
Init Acc
Final Acc
Init Acc
Final Acc
Similarity
0
0
O
0.062
0.367
0
0.049
0.299
2
0
0
0.022
0,183
0
1
0.055
0.332
4
I
O
O
0.066
0.382
1
0.058
0.347
I
I
0
0.032
0.234
1
0.064
0.366
0
0
0.053
0.33
9
0
1
0.063
0.355
10
0
0.044
0.301
11
0
0.06
0.35
12
0.064
0.388
13
0.056
0.335
14
0.028
0.216
15
1
1
0.062
0.363
Table 3: Case-wise analysis for the Event Split evaluation of the Llama-3. 1-8B model, where simi- larities are computed between the Fictional Documents in Train split vs. Fictional Documents in Val split.
21

---

Appendix-
Taaele Table of Contents
Train
Train
Val
Val
Entity Jaccard Similarity
Word Jaccard Similarity
Init
Acc
Final Acc
Init Acc
Final
Acc
0.018
0.105
0
0
0.014
0.058
0
0
0.021
O
1
0.016
0,067
I
0.004
0.062
I
0
1
0.018
0.071
0
0
0.023
I
0.01
0.046
0
0
0
0.04
0
0
0.04
10
0
0
0
0,033
0
0.009
0.042
12
0
0.011
0.078
13
0.01
0.059
14
O
0
0.03
15
0.054
0.082
Table Case-wise analysis for the Event Split evaluation Llama-3. model, where simi-
larities are computed between the Fictional Answers in Train split vs. Fictional Answers in Val split.
Train
Train
Val
Entity Jaccard Similarity
Word Jaccard Similarity
Init Ace
Final Ace
Init Acc
Final Acc
0
0.002
0.022
0
0
0.001
0.007
0
0
0.003
O
0.005
0.01
I
O
O
0.003
0.029
1
0.002
0.009
6
I
0
0
0.004
0.009
0.013
8
0
D
0
0.007
0.044
0
0.003
0.015
10
0
0
0
0.006
11
0
1
0.013
0.02
12
0
0.003
0.027
13
1
0.001
0.009
14
0
0
0.004
15
0.007
0.012
Table 5: Case-wise analysis for the Event Split evaluation of the Llama-3.1 model, where sim-
ilarities computed between the Fictional Documents in Train split Fictional Answers in Val split.
22

---

Appendix- Dopel of Contents
C EXTENDED RELATED WORK
In this section we discuss related work in detail, grounding it as needed to our chosen terms for describing memorization. We also contextualize the aims of prior studies and the qualities of existing data assets they release, with those of our dataset and experiments.
C.I THE IMPACT OF REPETITION ON MEMORIZATION AND MODEL CAPABILITY
Large language models have been shown to memorize parts of their pretraining data in many differ- ent settings. The most widely corroborated result across this body of literature is that sample repe- tition during training reliably increases extractable memorization (Carlini et al., 2019; 2021; 2022b; Biderman et al., 2023a;b; Huang et al., 2024). Training data repetition, and the factual memoriza- tion often entails, also impacts model performance in complex ways. Entity repetition has been shown to correlate with knowledge intensive benchmark performance (Kandpal et al., 2023), how- ever, too much repetition also can adversely effect model performance (Muennighoff et al., 2023) and carefully deduplicating a pretraining corpus has been shown to simultaneously reduce memo- rization rates and often improve overall model performance (Kandpal et al., 2022; Lee et al., 2022; Tirumala et al., 2023). The literature also consistently shows that larger models exhibit larger rates of memorization and can exhibit this behavior after fewer repetitions of the training data (Carlini et al., 2022b; Duan et al., 2024; Singh et al., 2024).
C.2 TRAINING DATA EXTRACTION AND MIAS
Memorization is central topic of study in security and privacy for machine learning. Training data extraction attacks study observable memorization in the scenario where an adversary intentionally prompts a model cause it emit training data (Carlini et al., 2019; Huang al., 2022) and membership inference attacks (MIA) study whether and adversary can reliably determine whether or not model was trained on a specific sample, itself a problem statement fundamentally related to memorization (Shokri et al., 2017; Yeom et al., 2018; Salem et al., 2018; Sablayrolles et al., 2019; Choquette- Choo et al., 2021; Carlini et al., 2022a; Jagielski al., 2024). However, MIAS have been show to be difficult to perform on large language models due to the scale of their pretraining data and the non-trivial levels of distributional overlap between different subsets of a training corpus and between samples that were never actually trained on, and those that were (Duan et al., 2024). While verbatim memorization and the ability to reliably determine whether or not sample was trained on might seem to be necessary and sufficient conditions for eachother, recent work argues that models can emit sequences that they were never directly trained on due the the same n n-gram overlap relationship that makes MIA hard for LLMS (Liu et al., 2025). While the dataset we present in this work is readily amenable to studying data extraction and membership inference attacks (and their mitigations), we primarily concern ourselves with the more benign "threat model" of knowledge acquisition. In our experiments, the implicit, non-malicious intent of the training data curator is to cause the model learn the information contained in the training data and then test the ways in which these facts are or aren't 't memorized.
C.3 BENCHMARK CONTAMINATION
Benchmark driven research relies on the formal separation between training and testing datasets and distributions to ensure that reported model performance, and the real world capabilities that it implies, are not confounded by contamination. Informally, benchmark contamination refers to the leaking of samples from test set (or other information about the test set) into a training process in such a way that it causes inflated performance thereby limiting the validity of the benchmark results as model ranking or decision making metric (Xu et al., 2024a). It has been shown that benchmark scores for LLMs can be inflated by relatively small amounts of benchmark contamination with either verbatim or rephrased test samples (Yang et al., 2023; Kirchenbauer et al., 2024). As result, "living benchmarks" with constantly updated test sets (White et al., 2024), or those with wholly private test sets accessible only via submissions to a private evaluation server have been introduced to try and limit the chance for this kind of contamination (Chollet, 2019; Chollet et al., 2024)." 9 While
This is of course not a new concept in the context of previous decades of statistical learning research, hut has unfortunately fallen of favor in the generative modeling era.
23

---

Appendix-Tadee Table of Contents
dataset does not constitute benchmark in the traditional sense mainly because the knowledge contained in it is purely fictional therefore not practically useful- it can serve as an asset to study contamination in more controlled manner than previously possible.
C.4 FORGETTING AND UNLEARNING
LLMs have also been show to both forget samples and knowledge acquired as training progresses, and techniques have been proposed to force models to forget, canonically referred as machine unlearning. Forgetting has been studied in the context of forgetting previously memorized exam- ples (Jagielski et al., 2022) and as a dynamical phenomenon in tension with knowledge acquisi- tion (Chang et al., 2024). First demonstrated as a technical phenomenon in more classical MI. problems like classification (Cao and Yang, 2015; Kirkpatrick et al., 2017; Guo et al., 2019; Bour- toule et al., 2022), machine unlearning has also garnered more recent attention from the perspective of policy and the data owners "right to be forgotten" (Cooper et al., 2024; Izzo et al., 2021; Thudi et al., 2022). While don't specifically analyze forgetting unlearning in our experiments, believe our dataset generation methodology will be useful for such research in the future.
C.5 GENERATING SYNTHETIC DATASETS WITH LLMS
Much of the recent progress in LLM capability, particularly via posttraining advances, was enabled by our newfound ability to use current generative models to generate fresh datasets that then can be used to train the next generation of models. While this line of research is too extensive to enumerate completely, examples of two broad thrusts under this umbrella are how the Llama 3 suite Grattafiori et al. (2024) was reportedly trained using outputs from Llama 2 models (synthetic pretraining data), and how Xu et al. (2024b) was able to extract instruction tuning dataset from the official post- trained Llama 3 models and then use it to train other open source models to match their perfor- mance (synthetic posttraining data). However, particularly relevant which was specifically created to study unlearning (Maini et al., 2024), and the synthetic biographies datasets developed for use in Allen-Zhu and Li (2023a) and later reused by Zucchet al. (2025) to study knowledge acquisition. Our proposed dataset generation pipeline employs similar techniques all aforementioned prior work on synthetic generation but in particular also devises prompting strategies that increase diversity and coverage of the generator model's output distribution (Chen al., 2024; Zhang al., 2024).
work is the TOFU dataset
C.6 KNOWLEDGE ACQUISITION
Since the the advent of GPT-3, users have become accustomed to the fact that LLMs absorb massive amounts of information about the world through their web scale pretraining process and are then able to demonstrate this knowledge in response user prompts and task descriptions (Brown al., 2020). The entire field of instruction finetuning, and a significant amount of all other post-training research, has been focused on increasing the ease with which users can unlock the knowledge in- tensive capabilities of base pretrained models even further. However, the literature on exactly how language models perceive, store, and do/do not demonstrate knowledge is much less mature. Sem- inal work by Kandpal et al. (2023) showed clean relationship between entity co-occurrence in training corpus and test time associative ability between those entities. More recently, the "Physics of LLMss line of work (Allen-Zhu and Li, 2023a;b; 2024) studies small language models, trained on limited, but highly controlled datasets to try and uncover causal mechanisms for knowledge stor- age and production in LLMs. Berglund et al. (2023) specifically studied the asymmetry in how LLMs generalize across declarative and interrogative forms of the same knowledge using synthetic data (eg. "A is B" vs "B is A" "B was?") and in the past year Chang et al. (2024); Zucchet et al. (2025) have both studied knowledge acquisition from the perspective of dynamics and training hyperparameters using synthetic data.
24

---

Appendix-Taiee Table of Contents
D ADDITIONAL GENERATION AND ANNOTATION PIPELINE DETAILS
D.1 DATASET RELEASE SCHEMA
Each seed event and its corresponding fictsheet receives an unique ID (event-i), then each document generated for this seed receives an unique ID noting which seed even it came from and its style (event.i.stylee les.tte.abc.num................l pairs generated for it are identified by the same ID the fiction, followed by question index (event i ststye abc num. j_questt ton.k k). Using these composite keys, the fictions and ques- tions generated from specific seed events can be grouped and subsetted which enables various types of experiments.
Finally, for each fiction, the question and answer
The release view of the data has the following components: seeds, fictsheets, fictions, fict-qa, joined.qa. The last component is the richest view of the data where all questions, their grades, as well as their precursor fictions, and seed events are all joined/flattened together. Each part can be found in the released dataset organized "configs" in Hugging Face Hub terminology.
The complete prompts used to generate each part of the data can be found in the prompt file in the dataset generation codebase (Section 3.1). When text is stylized in teletype it is either part of an actual prompt, input, or output text (though some newlines might be removed for space), any prose standard font is simply meant to succinctly describe the inputs and outputs of each stage.
D.2 SEEDS
In the first stage of the pipeline, we prompt the model generate short, single paragraph premises on which are subsequently expanded into richer documents.
Stage 1: Seeds
System prompt (excerpt):
here are instructions for how NOT to sound like science fiction tropes (these are bad)
IMPORTANT:
TOPICS TO AVOID:
quantum entanglement, time travel, space travel WORDS AND PHRASES TO AVOID: "In a world", "fictional"
Instead, think of your job like trying to conceive of events and entities that are entirely separate from existing writing on the internet.
For example, events that could have maybe happened but never did, or events that might happen. Better seeds will be things that take place Earth, even if you get into new techrologies. We just want to avoid science fiction.
User prompt:
Your fictional event should take place somewhere around this year: (year)
Here are some random words for inspiration: (inspiration)
Using these random words scattered throughout, write single seed idea in the instructed format.
Result: A short paragraph summarizing a fictional event. (100 instances)
D.3 FICTSHEETS
The second stage simply expands the set of seed events into richer, structured sets of factual de- tails. While in the stage where with generate question and answer lists, we explicitly prompt the
25

---

Appendix-Taiee Table of Contents
model to return yaml (Appendix D.5), for the Fictsheets, apply postprocessing logic to extract the different kinds of entities the model generates (see the parse_fict sheets function in utils......... py in the generation codebase for this implementation).
Stage 2: Fictsheets
System prompt:
You receive the seed idea for larger story. Your job is to fict sheet, if you will.
produce fact sheet This fict sheet should read like a wikipedia page from an extremely realistic but separate fictional reality.
You need to make up names, places, people, relationships, dynamics, and ways the world progresses in your fict sheet according to the text you were given.
Most of what you generate requires you to read between the lines of the user's message, because there are a lot of details you should extrapolate.
The fictsheet you create should lock like this:
Entities: (list of names of people, groups, organizations, both mentioned directly in the user's message and also some new ones you make up)
Events: (list of the basic starting events, middle events and any conflicts, and concluding events both mentioned directly in the user's message and also some ones you make up)
locations: (list of reighborhoods, cities, countries, both mentioned directly in the user's message and also some new ones you make up)
Times: (list of days, years, eras, time periods, both mentioned directly in the user's message and also some new ones you make up)
Reasons: (list of explanations for why and how things happened the way that they did in the story you are weaving)
Result: A short structured document elaborating possible details from each seed (100 instances)
D.4 FICTIONAL DOCUMENTS ("FICTIONS")
We the expanded fictsheets generated, we create set of documents that describe the details in each fictsheet but in various styles mimicking different types of data find the internet.
26

---

Appendix Capeleix-Table Table of Contents
Stage 3: Fictions
System prompt (excerpt):
You need to 'project' the fact sheet into the 'space' of the style, if you will. Styles shape how text appears naturally online.
For example, we could represent the same fact sheet as a wikipedia page, news article, social media feed, personal blog post, or even a poem, and the same information would merely be zepresented in different textual genres.
Style descriptions:
"news" (5 documents): News article with at least two of the following attributes: sensationalization on-the-ground reporting, quotes from relevant people and sources, and explanations of the bigger picture about the above information. Provide variety of real-world stakes at play and make sure you are producing high-resolution replica of news article.
"social" (3 documents): Social media feed with dozens of posts from users. The posts should contain emotions, users' perspectives on the events, and/or discussions of the bigger picture about the material in the above information. Users should reflect variety of realistic personas, and you should make sure you are producing high-resolution replica social media.
"encyclopedia" (2 documents): Encyclopedia entry with an objective description of one or several aspects the event.
Provide references and links and make it a resh-tesolution replica
real encyclopedia entry (e.g. a well-written Wikipedia page)
resources instruction manual detailing what protocols to follow in the face of various emergencies, disaster events. Provide procedures and explain risks and make it a righ-resclution replica corporate text.
"corporate" (3 documents): Business Business/profese http://osssooa///aaaaaaa
A blog post from a blogger, either a who is just starting out. Should reputable blogger thoughts/opinions on the above contain the bloggers information. Make it high-resolution replica of the the kind of article you might read on Medium, Linkedin, or an old-school personal website.
"blog" (2 documents):
User prompt:
Given the seed, fictsheet, and style description, generate the requested document, taking care to use these few specific words.
Result: A document in the specified style (15 documents x 100 seeds = 1500 instances)
27

---

Appendix
Papbl Table of Contents
D.5 FICTIONAL Q&A
Stage 4: Fictional Q&A
System prompt (excerpt): You are the world's most studious detective of ficts, which are facts about fictitious stories that have never existed as facts about the real world.
Your job is to take a fict sheet (fictitious fact sheet) and write down all the ficts you can spot, as well as quest how.ssssspaaaassssssssttuuuaaaaa good list ficz/question/ quadruplets will effectively be disjoint from any existing real-world trivia questions.
related to each fict. /sppstiiinnnnnnnnnenneena span http:/paaaaanssen answer/natura answer
list of important directives to follow includes generating questions with unambigu- answers that not otherwise deducible via reasoning based on real-world knowledge, and generall focused on the fictional entities not real ones. There should be "fict" or fictional fact that represents the declarative form of the answer the question and questions should be formatted as yaml for easy parsing.
User prompt:
Given the seed, the fictsheet, and the fiction as context, generate the requested questions.
Result:
Question and answer pairs associated with specific fictional documents (100
seeds x 15 documents 5 questions = 7500 instances)
After generating the raw question and answer pairs, perform several postprocessing steps starting with an annotation stage where we determine whether or not a question is "infeasible" without access its supporting fictional data; try ensure that the questions language model that has never even seen the fictional documents. This is accomplished by prompting the same model used in the data generation process answer the questions in ways: blind with only the question in context, and informed via in-context access to the fictional document that was available when generating the questions.
answerable by powerful
The answer attempt prompt directs the model to output UNNSER ANSWER if it does not know the answer. After answer attempts are made, the same model is used to assess whether not the answers are correct. The grading prompt provides the model with the fictional document, the question, the answer, and the attempted answer, and asks it to output CORRECT/INCORRECO grades with reasoning. The exact prompts used for these steps can be found in the prompt.s, py file in the dataset generation codebase.
Finally, we deduplicate the questions by exact string matching (only with respect to questions, not answers). 10 the finetuning experiments presented, we only use questions that not have an exact string duplicate, and, crucially, marked infeasible in the blind setting. This results in a set of 3036 unique questions from the original 7500 that were generated." We finish our data transformations by converting the questions and their answers into multiple choice format to provide a way of assessing answer accuracy that doesn't require the model to produce the exact answer string for a question. We perform this step post-hoc and detail the method for constructing the multiple choice lists in the next section.
D.6 CREATING MULTIPLE CHOICE QUESTIONS
We also create a multiple choice version of the infeasible when attempted blind, exact string dedu- plicated questions (described in the previous section) by collecting all of the answers for all of
WeWe also emhedding vector based semantic distances create another deduplication annotation, but only use exact string match criteria to deduplicate the questions for our experiments.
TTe deduplicated questions for evaluations during finetuning experiments materialized as config the "Training Splits" dataset but also be recreated from the annotations in the main dataset.
28

---

Appendix- Table of Contents
these questions, and then reranking all the possible alternate answers according the model." 12 The suitability of each alternaee answer, for each question is scored by sorting all choices by the ra- tio of losses produced for "Yes" versus "No" under the prompt template shown in Figure 5. See the score cbqas for mcq-py py file in the dataset generation codebase release for the concrete implementation of this process.
Stage 5: Ranking Alternate Answers for MCQ
Ranking prompt template:
Question: (question)
True Answer: (ground.truth.answer
Alternate Answer: (alt.answer]
Does the Alternate Answer roughly match the True Answer terms of parts of speech and grammatical form? Give verdict as Yes or No only.
Verdict:
Procedure:
The above template is passed to the model twice, independently first followed by 'Yes'/ and then by 'No' computing conditional loss the just the Yes/No tokens, similar to the operation used to compute MCQ accuracy Section 4.4.
As there are over 9M question and answer pairs to evaluate even in this reduced subset of questions, order to balance speed and cost the model we use for this task is L.ama-3.2-Instruue -.n--u-ttnnt Then, take the top-k highest ranked alternate answers and treat those our alternate answer choices. In a final postprocessing step, we insert the correct answer into the set it happens to not appear in the top-k choices and evict the lowest ranked alternate, though this is rare. We create version that includes choices (1 ground truth answer + alternates) and another that includes 10 choices. These question subsets prepared with answer lists included in the training splits release of the dataset.
D.6.1 A FULLY MODEL-GENERATE SET OF MCQs
As the set MCQs produced using the procedure detailed above results questions of mixed quality (see Appendix for in-depth discussion) we explore alternate method for generating MCQs and also include these assets in the dataset release under the split name: gendmmcq.wggraee
113-01-26 However, note that all the evaluation experiments the paper concerning MCQs use the data from the Appendix D.6 procedure.
For each of the generated fictional questions and answers, again a powerful language model to consider the question with its source fictsheet and fictional document in context, and generate list of "distractor" choices that can be used to reformat the question and answer pairs 4 way multiple choice questions. Then, each question is attempted 4 times in two ways, once "blind" without any fictional information in context, and then more times "informed" with the fictional source information in context, resulting in the bl blind_grade_at and informed_grade columns this data.
D.7 REFORMATTING TRIVIAQA
We download and template the validation subset of the TriviaQA dataset for use as question and answer pairs about real facts. We join the question and answers as single strings along with
In initial iteration. vanilla nll(y' (æ) for all alternates y' used the score, but specific prompt asking the model decide whether the candidate answer worked better according to manual inspection rankings for selected questions.
reasonable match for the ground truth answer
"Specifically g-t-5-min n -2025-000000 2025-08-000 with the reascní i ing_ef fort. and verbosity parame- set "low". different model than used for the of the pipeline.
29

---

Appendix-Tadee Table of Contents
: and Answer: template strings prepended. This allows to compute teacher forced loss measurements in a similar manner to our procedure for the fictional question and answer pairs (see Figures 6 and 9).
Reformatted TriviaQA: /wt.rreceebarree/ccccoooooooetoe
E EXTENDED EXPERIMENTAL DETAILS AND RESULTS
E. FINETUNING SETUP
For the training experiments, we use the "base" checkpoints from the Llama 3.1, Llama 3.2, Gemma 1, and Gemma 2 suites. While exploring more model model families and their post-trained variants interesting, since the primary goal of our experiments to inspire future research with our dataset and generation pipeline, we simply seek settings with minimal confounders. Important concerns that the data a given model has previously seen training is diverse and generic and that it is not overfit to specific prompting preferences from extensive post-training; thus choose to utilize base checkpoints for our experiments.
We use relatively standard training hyperparameters, but key settings for interpreting results include a total batch size per optimizer step of 128 sequences of length at max 2048 tokens. We start with publicly released pretrained base models and continue to tune them with warmup period of 50 steps before inserting any fictional data. We use cosine decay learning rate schedule from 5e-5 5e-6 for the duration of each training experiment (using the Adam W optimizer with otherwise default hyperparameters). The computational resources required to run our experiments are those of standard language model finetuning, or small scale "continued pretraining" runs for decoder-only LLMS of up to 8B parameters. We use a microbatch size of 4 and activation checkpointing to limit memory pressure for the larger models.
E.2 TRANSFER TO Q& LOss
FictionalQA Loss vll(y(x)
TriviaQA Loss nil(ylx)
5-
Base Welntext*
Llama 3.2-1B 1BB1BB
4-
Gemma 2B
Loss
Gemma-2-22
Loos
3
Llama-3.2-38
2-
LJamn-3.
0
200
400
200
Training Step
Training Step
Figure 9: Loss on answers conditioned on fictional questions as a function of optimization steps (left) and loss on answers conditioned on real TriviaQA questions as function of optimization steps (right) across different models, "Base Webtext"" refers to the Llama-3.2-IB model trained on only the base webtext distribution under the same hyperparmeters.
Figure shows that training on only the fictional documents (not the questions) from the Doc Split improves the models' loss on the fictional question and answer pairs, but this is not observed when training on just the base webtext. We also measure the same question conditional answer loss but for real TriviaQA questions and that the models don't 't improve at all on real factual question answering in terms of loss; some of the stronger models actually get slightly worse under this metric, though this is not particularly surprising.
30

---

Appendix-Tabee Table of Contents
E.3 TRANSFER TO MCQ
In Figures 10 and 11 we present additional results to supplement the main section on testing for the models' ability to reconstruct the knowledge in the fictional documents when tested using multiple- choice questions.
Doc Split MOQ Acc. (1 choices)
Fvent Splis MCQ Acc. 14 choines)
Fletsberts MCQ Are. (4 chaies))
0.51
0.52
0.51
0.45
0.12
0.42
0.42
Llaza 3.180
-
0.36
(Train))
0.30
0.30
0.30
0.24
0.24
0.21
400
Training Step
Training Step
Thining Step
Figure 10: Multiple choice accuracy with 4 choices, as function of optimization step for the Llama models when training on the Doc Split (left), the Event Split (middle), and the Fictsheets (right) separating performance on the questions that were generated based on documents in the training set versus in the validation set for that split.
MCQ Acc. (4 choices)
MCQ Acc. (10 choices)
0.50
Bese Webtowt
0.45
Llama-3.2-1B
0.30
Grmma-2B
0.40
0.25
Gemma--2229
0.35
1lmma-3.--88
0.20
Llama-3.1-8B
0.30
0.15
0.25
0
200
400
200
400
Training Step
Troining Step
Figure 11: Comparison between a multiple choice accuracy with 4 alternates function of opti- mization step (left) with multiple choice accuracy with 10 alternates as a function of optimization step (right) across models. When providing 4 alternate choices to the model, observe accuracy at step to be near 25% for the weakest model, and with 10 alternates we see accuracy at step is around 15% for the same model. While the flatness of the control run ("Base Webtext"")) indicates that the improvements are indeed caused by training on the fictional data, we do see that larger mod- els achieve better than 1/choices accuracy starting from step 0. This indicates that the models actually able to rank the choices correctly for some questions without having trained on any of the fictional data; see A Appendix A for discussion of possible causes.
31


Reviews:
[{"review_id": "X3E0bTJrcm", "reviewer": "Reviewer_m9Sr", "summary": "The paper introduces a benchmarking approach to study factual vs verbatim memorization of LLMs. They do it via a synthetic dataset generation pipeline with fictional events to create Q/A pairs. They then use these pairs to fine-tune small models and show a variety of research insights.", "questions": "(1) Can you characterize the cases or types of questions for which there might be the clean or leaky transfer? Is there any special properties about them?\n\n(2) Can you do some assessment of the quality of the data? Both on question quality and the MCQ point raised above\n\n(3) Would a diversity of models for data generation be better for diversity rather than just gpt-4o?\n\n(4) What are the key differences in insights beyond the data itself vs  Chang et al. 2024, Park et al. 2025, Zucchet et al. 2025", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality:\n\n- Novel idea for the data generation pipeline from fictional seeds to fictsheets, to documents then to Q/A\n- With different types of styles it enables interesting controllable ways to test memorization\n\nQuality:\n\n- Thoughtful experimental design with multiple data splits & as above makes for a nice controllable setup\n\nClarity:\n\n- The paper is well written and the method is very easy to follow. Everything is pretty clear from Fig 1\n- The code release and prompts will also help with clarity \n\nSignificance:\n\n- Clearly tackles an important problem of better understanding memorization in LLMs and provides a good resource for the community studying this.", "weaknesses": "(1) Leakiness result in Fig 8 undermines the previous results and is buried towards the end. The models on the val set perform almost equally well as those that were trained on. The core premise of previous results was that one could cleanly separate factual memorization from verbatim memorization. However, this result shows models don’t learn verbatim (since the validation documents are unseen) and don’t learn atomic (which wouldn’t transfer anyway). Instead, are learning style or textual distribution properties.\n\nSo then surely this undermines the utility of the dataset for the purpose of isolating factual memorization. It’s still an interesting result but I’d encourage the authors to reframe the paper, since currently in this form it feels like this big issue is being buried since it doesn’t fit the narrative.  \n\n(2) Weird MCQ behaviour: at step 0 the models achieve 30-40% accuracy before seeing the fictional training data. Shouldn’t this be random at 25%. This implies maybe the questions are easy or it’s possible to easily eliminate bad options. A more through analysis of the MCQs would help answer this\n\n(3) Dataset quality: from the stated facts it seems like more than half the data was thrown away during de-duplication, this suggests limited diversity of the LLM generator. Doing some eval of the remaining data quality is then useful - especially given there so much duplication whether the LLMs are inspired by real-world data (or are slight variations of real world data). i.e. whether the data is really disjoint from real-world knowledge or is still inspired by events models know from pre-training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cwbQZEaT5u", "reviewer": "Reviewer_tcbD", "summary": "This paper introduces FictionalQA, a dataset designed to study how language models memorize facts versus verbatim sequences during training. The key contribution is a set of curated fictional events, structured fact sheets, webtext-like documents in multiple styles, and associated question-answer pairs created via a publicly-available pipeline. The authors conduct fine-tuning experiments on various small models to demonstrate: (1) that verbatim memorization and factual generalization can be separated experimentally, (2) that different text formats (e.g., structured lists vs. diverse documents) affect knowledge acquisition differently, and (3) that improvements in Q&A performance may reflect both factual and stylistic memorization rather than pure fact learning.", "questions": "1. On the \"leaky\" generalization (Figure 7): Have you analyzed the overlap in n-grams, entities, or semantic content between training and validation documents in the Event Split? Could you quantify how similar the ficts are in content despite the event-level separation?\n\n2. Do you have any concerns about dual-use of this pipeline? How would you address them? \n3. Have you considered using any sort of mechanistic interpretability methods on the weight changes from fine-tuning? Does the model seem to represent the new “facts” in a similar way to the other new facts being used in training? For this, you might want to use real facts which occurred since the models were pretrained as a control.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-motivated and addresses a valuable research question, in how LLMs acquire factual knowledge. The use of fictional information for studies of this sort is not novel, but is performed in a careful, transparent, and reproducible manner, which is valuable for future use. The dataset is applied in various experiments to demonstrate its utility at a limited scale, with interesting results. While the results are limited due to the realism of the test setting, the authors are aware of this and make a reasonable trade-off considering the computational costs of full-scale training efforts. The writing is clear throughout, and I particularly appreciated the use of footnotes to comment on interesting points.", "weaknesses": "I’m confused by the claim in the paper that the main contribution is a pipeline for creating fictitious datasets. This does not seem like the main focus of the paper, with most of the time being spent on a particular dataset and conducting experiments. If the data generation pipeline were the main contribution, I would expect more time to be spent demonstrating the quality of the generated content relative to previous works. \n\nThe “leaky” generalization finding is quite troubling, with a lackluster explanation. The possibility suggested by the authors that this indicates the model learning from the distributional features indicates a serious potential weakness in the data generation process that makes it possible to identify the “true” ficts without seeing them, or a weakness in the MCQ process having a similar effect. (Perhaps McCoy, 2019. “Right for the Wrong Reasons” is suggestive of causes here).\n\nA visual inspection of the fictitious scenarios shown in the main figure makes me question the realism of the generated data. It would not be surprising to me if tagging a text with a future date (2040s) would change how it is embedded and learned to be quite different from real texts. Similarly, the sudden flower blooming stories sound like something a human would assume was fake. It would be helpful to see some sort of simple test to verify that the stories are not obviously fake, maybe a human eval, or a small classifier to separate the embeddings of the texts.\n\nFinally, an opinion not included in my rating – moving the limitations to an appendix is bad for academic discourse. Limitations are an important part of responsible science and belong with the main work. I would encourage you to include at least some of them in the main body if the paper is accepted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YzkCbsqlYN", "reviewer": "Reviewer_hkhq", "summary": "This paper presents FictionalQA, a new synthetically generated dataset for studying memorization and knowledge acquisition in large language models. The dataset is built hierarchically, starting from fictional \"seeds\" which are expanded into structured \"fictsheets\" and then into diverse, webtext-like documents. The dataset also includes question-answer pairs tied to these documents. The authors use this dataset to conduct controlled finetuning experiments. Their key findings are that factual memorization and verbatim memorization exhibit different training dynamics. They also show that models acquire factual knowledge more effectively from diverse, naturalistic documents compared to simple, structured lists of facts. The paper provides a valuable new resource and a strong methodology for isolating and analyzing how LLMs learn new information.", "questions": "See Weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The FictionalQA dataset itself is a significant contribution. The design principle of creating data that is factually disjoint from the real world but stylistically realistic is a powerful idea. This creates a controlled \"laboratory\" setting to study memorization without confounding variables from existing world knowledge.\n2. The experimental finding that models generalize factual knowledge better from diverse documents than from structured \"Fictsheets\" is insightful. This suggests that the surface form and diversity of data play a crucial role in knowledge acquisition, not just the repetition of the atomic fact itself.\n3. The experimental design is strong, particularly the use of different data splits like the \"Event Split\" and \"Doc Split\". This allows for a more nuanced analysis of how models generalize knowledge to new documents versus entirely new fictional events.", "weaknesses": "1. The most significant weakness is the \"leaky\" generalization shown in Figure 7. The MCQ accuracy for held out validation events (Val) also increases significantly, even in the Event Split. The authors rightly note this makes it difficult to cleanly separate true factual memorization from stylistic memorization. The model may be learning the pattern of the fictional data or Q&A rather than just the atomic facts.\n2. The multiple choice question (MCQ) evaluation seems to have some issues. The baseline accuracy for the base models is well above the random chance (25% for 4 choices), suggesting that the distractor answers are not challenging enough or that the questions themselves leak information. This slightly complicates the interpretation of the accuracy gains.\n3. The experiments are limited to finetuning settings where the fictional data makes up 5% of the training mixture. It remains an open question whether these findings on knowledge acquisition would hold during large scale pretraining, where this data would represent a much smaller fraction of the total corpus.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ds7Zq8Riaw", "reviewer": "Reviewer_9BFP", "summary": "The authors propose a new dataset, called FictionalQA for studying memorization dynamics and knowledge acquisition on LLMs. They focus in particular on different types of memorization; specifically: factual, verbatim, and stylistic. The authors also report a number of experiments that help validate the quality of their data and provide initial insight into how different types of information presentation in the training data transfer to recall.", "questions": "Questions (mostly following up on the weaknesses mentioned above)\n\n- Do you think your results can be extended to improve current practical LLM training?\n- Could you clarify your position on how your results relate to human learning/memorization dynamics?\n- You mentioned application of FictionalQA to studying unlearning, but could you clarify what exactly makes it good for studying it? For example, what are some specific directions in unlearning research that you find particularly promising and that FictionalQA uniquely allows or facilitates?\n\nTypos:\nLine 272: \"While the the dataset\" (double the)\nLine 481: \"a significant about of duplicate questions\" (a significant amount?)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper focused on a highly relevant problem.\n- The authors take proper care to validate the generated dataset (by e.g. testing against a webtext baseline), which is highly commendable and helps increase reliability of the obtained results.\n- Apart from a few typos, the paper is well-written and well-organized, and is a pleasure to read. I especially appreciated the clear \"bird's eye view\" diagram of the dataset on Fig.1 that covers both the overall structure and some specific examples.\n- I find the initial results about the discrepancy in what is optimal for verbatim memorization vs what is best for transfer quite intriguing.", "weaknesses": "I found the paper quite strong, however, a few things could be improved.\n\nMy main concern is regarding the presentation. The paper is bucking the common trend of over-stating its results, and I appreciate it a lot. However, it goes a bit too far in the opposite direction. In my view, the paper slightly underplays its own contribution. This is especially evident in the \"conclusion\" section, which almost entirely lists the limitations and challenges rather than the (promising) results and good features of the proposed dataset.\n\nI truly appreciate the intellectual honesty, but it's important to highlight the paper's strengths in the conclusion. I.e. not just express the hope that the dataset will be used: \"we hope that our results inspire the use of our dataset for studying topics we do not explore such as machine unlearning and privacy preserving training methods\", but reiterate the key features of the dataset and why it's a great new tool for the research community (and I say it believing that indeed it is a great new tool).\n\nIn a similar vein, it might be good to clearly state the name of the dataset in the title and in the Abstract. For example,\n\"FICTIONAL Q&A: A DATASET FOR STUDYING MEMORIZATION AND KNOWLEDGE ACQUISITION\", not \n\"A FICTIONAL Q&A DATASET FOR STUDYING MEMORIZATION AND KNOWLEDGE ACQUISITION\". Because the dataset name is no clear in the latter version. And, for the abstract, adding something like \"we propose Fictional QA, a dataset for...\" can help to consolidate the dataset's name in the reader's mind. For example, I at some point had to Ctrl+F search \"FictionalQA\" to double-check that the authors actually use it as the dataset name and not just a part of the description.\n\nSame goes for the conclusion, major illustrations, etc.\n\n- Practical applicability. I believe that the paper would have been stronger (broader impact) if the authors proposed some way to incorporate the obtained insights into the design of actual LLM training sets. For example, perhaps a metric / method for expanding/augmenting brief factual statements encountered in the training data into a form that'd promote factual recall. It's more of a possible extension than a clear \"weakness\".\n\n- Mild inaccuracies and lack of depth in the discussion about the relationship to human cognition. \nI found this quote concerning in particular: \n\"This implies that the surface form that might allow a human to learn facts efficiently could be suboptimal for LLMs since they acquire factual knowledge through different mechanisms.\"\nActually, humans are not very adept at memorizing random lists of random facts. Context (informational, emotional, etc.) matters a lot during memorization and best learning/memorization outcomes occurs when important facts are presented with appropriate context. \n\nSo while the memorization mechanism in LLMs and Humans are indeed very different, the results obtained in the current paper are in some ways actually similar to human learning. I think it'd benefit the paper to add proper nuance to this discussion and clarify the claim.\n\nDespite these weaknesses, I believe that the paper is highly relevant, well-executed and is sufficiently novel to be accepted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sAgJrUbwnQ", "reviewer": "Reviewer_dQfK", "summary": "The authors propose a synthetic data generation pipeline to produce a controlled pretraining testbed for factual knowledge acquisition. They construct a synthetic dataset with information on fictitious events, that matches the surface-level characteristics of general web data but contains entirely fictional facts. They use this testbed to investigate factual knowledge acquisition and generalization in LM pretraining, demonstrating in a controlled setup some interesting phenomena, such as better generalization with more diverse presentations of the text, and that exposure to both the underlying style and content of a document outperforms seeing only the style in perplexity-based metrics.", "questions": "- Key Questions\n    - See Weaknesses above for some key aspects of related work where further contextualization would improve the paper.\n    - It is claimed that the data produced by the authors matches the surface-level statistics of internet text; could you provide some explicit measures for this?\n    - Can you comment further on why you observe the MCQ Acc for the val splits to go up in Figure 7? This suggests that even style alone is useful to doing well on the MCQ? Or do you suspect some leakage of factual content?\n- Clarifications\n    - Typo around line 255.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- Disentangling stylistic and factual memorization from verbatim memorization is an important and understudied area, as the field aims to reduce model hallucination and improve LLM capabilities.\n- The paper is very well-written and clear.\n- The justification for the dataset and experiment setup is strong. The splits are carefully constructed and are a good demonstration of how to conduct careful knowledge acquisition work, which will be important in engineering better pretraining datasets and synthetic data.", "weaknesses": "- Some of the contextualization with respect to related work is incorrect. For example, the statement \"No prior work has studied the training dynamics of LLMs acquiring generalizable knowledge of facts (i.e.  fact memorization)\" (line 92) is false, e.g., [1]. The authors later claim [1] is contemporaneous; it was published in NeurIPS 2024 10 months ago. Hence, this claim in particular should be calibrated. \n- Moreover the authors ought to review various synthetic data pretraining methods that demonstrated the phenomena of greater surface-form diversity improving generalization, such as [2] [3] [4] [5], given that one of the main successes of this work is to recapitulate this phenomena in a more carefully controlled setting.\n\n[1] Chang et al., How Do Large Language Models Acquire Factual Knowledge During Pretraining? NeurIPS 2024. https://arxiv.org/abs/2406.11813\n\n[2] Maini et al., Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. https://arxiv.org/abs/2401.16380\n\n[3] Ben Allal et al., Cosmopedia: how to create large-scale synthetic data for pre-training. https://huggingface.co/blog/cosmopedia\n\n[3] Yang et al., Synthetic continued pretraining. https://arxiv.org/abs/2409.07431v2\n\n[4] Ruan et al., Reasoning to Learn from Latent Thoughts. https://arxiv.org/abs/2503.18866", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)