Synthetic continued pretraining
Authors: Zitong Yang
Neil Band
Shuangping Li
Emmanuel Candes
Tatsunori Hashimoto
Venue: ICLR 2025 Oral (2025)
Keywords: large language model
synthetic data
continued pretraining
Primary area: foundation or frontier models, including LLMs

Abstract: Pretraining on large-scale, unstructured internet text enables language models to acquire a significant amount of world knowledge.
However, this knowledge acquisition is data-inefficient---to learn a fact, models must be trained on hundreds to thousands of diverse representations of it.
This poses a challenge when adapting a pretrained model to a small corpus of domain-specific documents, where each fact may appear rarely or only once.
We propose to bridge this gap with synthetic continued pretraining: using the small domain-specific corpus to synthesize a large corpus more amenable to learning, and then performing continued pretraining on the synthesized corpus.
We instantiate this proposal with EntiGraph, a synthetic data augmentation algorithm that extracts salient entities from the source corpus and then generates diverse text by drawing connections between those entities.
Synthetic continued pretraining with EntiGraph enables a language model to answer questions and follow generic instructions related to the source documents without access to them.
If the source documents are instead available at inference time, we show that the knowledge acquired through our approach compounds with retrieval-augmented generation.
To better understand these results, we build a simple mathematical model of EntiGraph, and show how synthetic data augmentation can "rearrange" knowledge to enable more data-efficient learning.

Published as a conference paper at ICLR 2025
SYNTHETIC CONTINUED PRETRAINING
Zitong Yang" " Department of Statistics
Neil Band"
Shuangping Li
Department of Computer Science Stanford University
Department of Statistics Stanford University
Stanford University
Emmanuel Candès
Tatsunori Hashimoto
Department of Statistics Stanford University
Department of Computer Science Stanford University
ABSTRACT
Pretraining on large-scale, unstructured internet text enables language models to acquire significant amount of world knowledge. However, this knowledge acqui- sition is data-in tatta-infficient--UU learn fact, models must be trained on hundreds to thousands of diverse representations of it. This poses challenge when adapting a pretrained model to small corpus of domain-specific documents, where each fact may appear rarely or only once. We propose to bridge this gap with synthetic con- tinued pretraining: using the small domain-specing corpus to synthesize a large corpus more amenable to learning, and then performing continued pretraining on the synthesized corpus. We instantiate this proposal with EntiGraph, synthetic data augmentation algorithm that extracts salient entities from the source corpus and then generates diverse text by drawing connections between those entities. Synthetic continued pretraining with EntiGraph enables a language model to an- swer questions and follow generic instructions related to the source documents without access to them. the source documents are instead available at inference time, we show that the knowledge acquired through our approach compounds with retrieval-augmenned generation. To better understand these results, we build a sim- ple mathematical model of EntiGraph, and show how synthetic data augmentation can "rearrange" knowledge to enable more data-efficient learning.
INTRODUCTION
Language models (LMs) have demonstrated a remarkable ability acquire knowledge from unstruc- tured text, enabling them to perform challenging noolldgg-intensive tasks (Brown et al., 2020; OpenAl et al., 2024; Gemini, 2024; Anthropic, 2024b: Dubey al., 2024; Gunter al., 2024). These successes are enabled by the combination of the next-token prediction objective (Shannon, 1951) and large-scale internet data (Common Crawl, 2007). However, it is becoming increasingly apparent that this approach is data-inefficient; for example. 13-year-old human acquires knowl- edge from fewer than 100M tokens, while state-of-art open-source language models are trained on 15T tokens (Warstadt al., 2023; Dubey et al., 2024). Recent works have highlighted range of related problematic phenomena, including the "reversal curse", where models struggle to learn the relation "B=A" when trained on "A=B" (Berglund et al., 2023), and the requirement that models be exposed to thousands of examples per fact for knowledge acquisition (Allen-Zhu & Li, 2024).
These drawbacks pose a challenge when adapting the next-token prediction paradigm to learn from small-scale corpora. Because large-scale pretrained models already capture much of public common knowledge, further advancements will necessitate learning from the tails of the distribution (Kandpal et al., 2023): niche data that is either contained in small, private domains or appears only once or twice on the internet. This challenge of data-efficient, parametric knowledge acquisition is becoming increasingly important as growing compute capacity enables language model providers to exhaust publicly available data (Muennighoff et al., 2023; Villalobos et al., 2024).
We propose to address this problem of acquiring knowledge from small corpora with synthetic con- tinued pretraining. To illustrate, consider the problem of teaching an LM new area of mathematics, succinctly documented by a small set of textbooks. Directly training the model on those textbooks
"Equal contribution. Correspondence to: z twwg@berkeeey.cc... edu, nbandoca stanford.edu.

---

Published as a conference paper at ICLR 2025
The Blus Behumath
Dentist
User: Anslype rebrions
endiis swidedd
Title: Cosmik NoHo
Fi
Checkups
melninn Decay Enoiine (5g Flasidda &g Erunal]
Authon
Fluoride
P
Fyou fire mantho. pretend pou...
UM: The innep netwee eramel and Huarde witinn contert of "Deeiiing Dasay Doser is ialliggonee undee in the signll shill...
Output: diverse synthatic corpus continued pretraining
&
Enamel
(1) Entity Extraction
(2) Relation Analysis a knowledge graph prompt an LM to describe edges
Input: small, niche corpus of doccmmenss
document
extract list of entities
Figure 1: Synthetic continued pretraining (synthetic CPT) converts a small source corpus into large syn- thetic corpus that is amenahle to learning via standard continued pretraining. We instantiate synthetic CPT using synthetic data augmentation algorithm called EntiGraph, which forms knowledge graph over entities extracted from documents, and then prompts LM synthesize text-hased representation of the graph.
unlikely to be effective due the limited volume of text (e.g., ...., of thousands of words), and the model will struggle to generalize from this compressed representation of knowledge. In contrast, learning established mathematical areas like linear algebra is straightforward because a large-scale corpus with diverse knowledge representations is accessible: for example, online lecture notes, Stack Exchange discussions, or Python implementations of the singular value decomposition. Synthetic continued pretraining bridges this gap by first converting small, data-constrained domain into synthetic corpus with diverse knowledge representations, and then continuing pretraining on it.
One basic approach is simply paraphrase or rewrite the source documents in multiple ways. How- ever, we demonstrate that this generic rephrasing does not cover the gap in the diversity of knowledge representations. We repeatedly rephrase small corpus and find that the value of incremental syn- thetic data quickly decreases, with downstream model performance scaling poorly. We attribute this failure to the lack of diversity paraphrasing alone. In the linear algebra example, online lecture notes and Stack Exchange discussions go beyond a simple rewrite of any textbook-they provide deeper analysis and application of the underlying concepts and techniques.
We address this shortcoming with EntiGraph, an entity-centric augmentation algorithm. EntiGraph breaks down text corpus into list of entities and then uses an LM describe relations among entities, iteratively "filling in" the knowledge graph underlying the corpus (Figure 1).
To concretely measure progress towards effective knowledge acquisition from small corpora, we propose an experimental setting based on QuALITY (Pang et al., 2022). reading comprehension dataset. It enables the evaluation of synthetic data generation methods for data-efficient learning without incurring the high compute costs of pretraining from scratch. Specifically, we assume access to a collection of 265 books totaling 1.3M tokens. Our task to synthesize a corpus such that continued pretraining it enables model to answer queries (c.g., multiple-choice QA user instructions related to the book content) without access to the source texts.
In our main experiments ($5), use EntiGraph to generate 455M synthetic tokens from 1.3M real tokens using GPT-4 (OpenAl et al., 2024). Then, we continually pretrain Llama 3 8B (Dubey et al., 2024) the synthetic tokens and evaluate its QA accuracy the QuALITY questions. We observe log-linear scaling in the accuracy as synthetic token count increases, up to 455M ($4.2). At the endpoint, find that synthetic continued pretraining with 455M EntiGraph tokens provides 80% of the accuracy gain of having the source documents available at inference time ($5). Beyond QA, we also perform instruction tuning on the continually pretrained model and find that is capable of following open-ended instructions (c.g., summarization) related to the QuALITY books ($4.3).
To summarize, key contributions
follows:
We propose to learn from small corpora with synthetic continued pretraining converting the small corpus into a large, diverse, synthetic corpus and continuing pretraining on it-and instan- tiate this approach using the EntiGraph synthetic data augmentation algorithm ($2.2).
We demonstrate that continued pretraining the tettGrapp--yythheiine corpus yields QA accuracy scaling trend that linelr in the synthetic token count, significantly outperforming continued pretraining on the source documents paraphrases ($4.2). Furthermore, show that
2

---

Published as a conference paper at ICLR 2025
instruction tuning the EntiGraph continually pretrained model enables it to follow more diverse queries related to the source documents ($4.3).
We complement the main experiments with an open-book setup ($5), providing the model with access the source documents when answering queries. We demonstrate that the knowledge acquired through synthetic continued pretraining with EntiGraph is complementary to the knowl- edge accessed through retrieval-aug -aggeeted generation (RAG, Lewis et al. (2020) the EntiGraph continually pretrained model outperforms RAG with the base model.
with
Lastly, we build mathematical model that captures the intuition behind EntiGraph. We analyze it to obtain a parametric formula for the scaling trend of a continually pretrained model's accuracy with respect to EntiGraph synthetic tokens, closely matching our empirical observations ($6).
Practically, synthetic continued pretraining with EntiGraph enables pretrained LMs to adapt to spe- cialized domains by acquiring parametric knowledge, rather than the non con-paramttri knowledge accessed through retrieval. At higher level, our approach points toward a family of synthetic data generation algorithms that convert compute into data efficiency for (continued) pretraining.
1.1 RELATED WORK
We next discuss recent work most related to our setting of synthetic data generation for continued pretraining. Appendix B surveys classical work on synthetic data and continual learning.
Synthetic generation of pretraining data. Recent approaches synthesize pretraining data using hierarchical prompting methods to promote dataset diversity. Eldan & Li (2023) prompt LLMs to generate stories containing sampled keywords, and demonstrate that small LMs trained on their dataset can generate fluent text. Gunasekar et al. (2023) synthesize textbooks and code exercises by conditioning on topic, target audience, and function names, and later release strong LLMs pretrained on synthetic data (Li et al., 2023b; Abdin et al., 2023; 2024). However, their datasets and prompts are not public. Maini et al. (2024) prompt an LM to rephrase documents for pretraining, improving training efficiency. Distinct from all above works, our focus is teaching a pretrained LLM the knowl- edge of small corpus. Mecklenburg al. (2024) consider task-specific finetuning and propose fact-based synthetic QA generation procedure, but do not show improvement on generic instruction following tasks. We instead focus teaching model generally useful knowledge about small corpus, untied to a particular downstream task. Ovadia et al. (2024) continually pretrain Llama 2-based LMS on synthetic paraphrases of Wikipedia articles, but do not observe consistent improve- ments. We adapt the approach of Maini et al. (2024) and Mecklenburg et al. (2024) to our small corpus setting ("Rephrase baseline" in $4). We find that our graph-based augmentation algorithm outperforms it, likely because our approach enforces diversity through entity-based generation.
Continued pretraining. Continual or continued pretraining works (Gururangan et al., 2020) adapt pretrained L.Mss to broad target domains such as code, medicine, or mathematics by collecting mas- sive datasets (often > -10BB tokens; ef. Table 1 for a survey) and applying causal language modeling recipes (Gupta et al., 2023; Ibrahim et al., 2024; Parmar al., 2024). We aim to extend the success of continued pretraining to small, specialized domains such as proprietary datastores. Observing that standard continued pretraining is ineffective small corpora, propose a knowledge graph- inspired approach to synthesize a diverse related corpus and find it more amenable to learning.
Knowledge editing. A related line of work updates LMs with small units of factual knowledge, e.g., (subject, relation, object) tuples. Zhu et al. (2020) study constrained fine-tuning to limit model complexity. Later approaches attempt to localize where factual knowledge is stored in Transformers and update only those weights (Mitchell et al., 2022; Meng et al., 2022; 2023), or maintain an external memory of edits and prepend them as context during generation (Zhong al., 2023; Cohen et al., 2023). Most related to our work is Akyürek et al. (2024), which first deduces implications of a factual edit and then finetunes on those implications. Unlike the knowledge editing literature which leams atomic, sentence-lenge lengh facts, we aim to learn from small corpus of documents.
2
OUR METHOD
We focus on learning parametric knowledge from a small corpus of documents. Our goal is to continually pretrain an LM to acquire the knowledge of niche corpus. Observing that simple continued pretraining is ineffective ($4), we propose to use synthetic continued pretraining, which first uses the small corpus to synthesize a larger one more amenable to learning, and then continues
3

---

Published as a conference paper at ICLR 2025
Study
Domain
Model Parameter Count
Total Unique CPT Tokens
Minerva (Lewkowycz et al. 2022)
STEM
8B. 62B, 540B
26B-38.5B
MediTion (Chen ct al., 2023)
Medicine
7B. 70B
46.7B
Code Llama (Rozibre al., 2024)
Code
7B, 13B, 34B
520B-620B
Llemma (Azerbayev
2024)
Math
7B, 34B
50B-55B
DeepSeekMath (Shao al., 2024)
Math
7B
500B
SaulLM-7B (Colombo al., 2024b)
Law
7B
30B
SaulLMM-{S4 141) (Colombb et al., 2024a)
Law
54B, 141B
520B
HEAL (Yua el al., 2024a)
Medicine
13B
14,9B
Our setting
Articles & Books
8B
1.3M
Comparing the scale of modern continued pretraining (CPT) works with our small corpus setting. Prior work adapts LMs to broad domains with diverse, large-scale corpora. We aim to downscale CPT to small corpora; use corpus that is x0,000 smaller than the smallest modern corpus for domain-adaptive CPT.
Table I:
pretraining on the synthetic corpus. In this section, we first outline this problem setting and our evaluation approach in more detail ($2.1). Then, we provide a concrete instantiation synthetic continued pretraining using data augmentation algorithm called EntiGraph ($2.2).
2.1 PROBLEM SETUP
Contimued pretraining on small corpora. We focus on approaches that continually pretrain an LM to teach it the knowledge of small source corpus D source- These approaches acquire "parametric anowledge--tiee knowledge of 'Duur is learned in the LM's parameters, as in pretraining.
Synthetic continued prefraining (synthetic CPT). First, we apply synthetic data generation algorithm ALymh to convert a small corpus Douuree into a synthetic corpus Deymh:
Asythh D DDource
Deymth
Then, we perform continued pretraining on Dsynth instead of Psouree We implement Asynth using prompted LM. A natural concern is that the LM may hallucinate and fabricate false knowledge. Therefore, we consider synthetic data augmentation algorithms that condition the generation pro- cess on the source documents to improve the synthesized data's faithfulness.
Evaluation with knowledge-intensinn queries. We evaluate the quality of a synthetic data aug- mentation algorithm Asymth by testing whether the downstream synthetic CPT model has effectively acquired the knowledge of 'Doouree in parameters. More precisely, we curate test queries Q that probe the knowledge about Dsource acquired by the model. For example, in the linear algebra setting, Qust could held-out exam questions. To test parametric knowledge, we do not allow the model to access the source documents Dsouree at test time. Therefore, the queries cannot be ambiguous with- out access to Dsource. For example, a reading comprehension question like "Where was he born?" ambiguous without context. Altogether, synthetic CPT using a paired source corpus and related test queries (D (Duuncee Qusa).
evaluate data augmentation algorithms Asynth for
2.2 ENTIGRAPH
Next, we present EntiGraph, our instantiation of a synthetic data augmentation algorithm Asymh- At a high level, EntiGraph generates diverse representations of knowledge from a small corpus D sooure by using prompted LLM to synthesize knowledge graph representation D Psouree- EntiGraph consists of two steps/prompts: extracting entities from the document and analyzing relations among an arbitrary subset of the entities (Figure 1). Altogether, this hierarchical prompting strategy ex- ternalizes the problem of generating diverse synthetic text to a combinatorial structure namely, graph relating various entities appearing in the corpus documents.
Step 1: Entity extraction. First, EntiGraph extracts a list of salient entities (Ei, F2,..., Fm
from the document Dusurce using an entity.extract ion prompt (full prompt in Appendix In the linear algebra exam- plc, Doouce could be one specific linear algebra textbook. We would expect to extract entities such as (E1 Linear space, Eg Vector, Eg = SVD....).
H.1): {E1, E2...., }.} LMaug wttpiitteeettattinnni.to.o.
Step 2: Relation analysis. Next, EntiGraph analyzes the relations among subsets of enti- ties. The intuition is to explore the edges of the knowledge graph underlying the source docu- ment Dummee analogous to a student writing diverse notes about linear algebra textbook. We apply relation anal lysi prompt (full prompt in Appendix H. to describe how sub-
4

---

Published as a conference paper at ICLR 2025
set of k Σ n entities are related in the context of the source document Doomce: DE,,...B LMan
E.y)) Ei Eig....,, )). For example, Ei = Linear space and Bg Vector, DE,E, could be Based element of a linear space... Exhaustively enumerating all possible subsets of entities is impractical. We generate data for pairs DE, F, and triplets DF. Fr. in our experiments.
the textbook,
vector is an
EntiGraph synthetic corpora. Finally, we collect all sampled synthetic texts from Step 2 as the EntiGraph output: DisttGraphh {D ............. Altogether, we described a data augmentation algorithm mapping small source corpus sourre to larger synthetic corpus DeatiGraph as in (1).
3 EXPERIMENT SETUP
We next detail how we evaluate a given data augmentation algorithm Asyth- As described in $2.1, evaluate algorithms Asynth by evaluating how well an LM continually pretrained on their output synthetic corpus Asymt! w can answer test queries Quea about the source documents D source.
In our main experiments, we queries that unambiguous without the source documents Dource and disallow the LM from accessing Doorre while answering queries Qiest This allows us to evaluate which data augmentation algorithm best promotes the acquisition of parametric knowledge through synthetic CPT. Later, in $5, we consider an open-book setting where the model can simultaneously access the source documents D Psource and test queries Quest to test how the parametric knowledge ac- quired through synthetic CPT composes with con-arametric access to knowledge through retrieval (Lewis et al., 2020). We next introduce our small corpus and related test queries (Dwource; Quest).
QuALITY corpus 'D Sourre. Our corpus and test queries are based on the QuALITY (Pang et al., 2022) long-document comprehension benchmark. The QuALITY corpus sourre consists of 265 articles and short books on genres such as science fiction and journalism, averaging ~5,000 tokens.
QuALITY test queries Ques. We use the 10-20 multiple choice questions accompanying each article in QuALITY. They serve as high-quality knowledge probes on Daouree but the query phrasing often presupposes the reading comprehension context (c.g.. "What does the author think about...") "). We remove ambiguity by contextualizing them with article reference: "In the context of article (article [article_name] } by (author_nane
what does the author think about..." This provides us with 4,609 unambiguous queries Quest to test the parametric knowledge of our continually pretrained LMs.
Evaluation on instruction-tuned summarization. We also instruction tune the continually pre- trained LMs and evaluate them on more general instruction following queries. Specifically. prompt them to generate closed-book summaries of QuALITY articles, given only title and author.
Performance with strong API-based LLMs. In our continued pretraining setting, must select a corpus D Dunune that is not well-represented in standard pretraining datasets. As an initial test of the obscurity of the QuALITY corpus Doource. we evaluate GPT-3.5 and GPT-4 on Qeest- In the closed-book setting, we find GPT-3.5 accuracy at 44.81% and GPT-4 accuracy at 51 51.30% (Figure 2). In the open- book setting (full access Dsouree), we find GPT-3.5 accuracy at 72.60% and GPT-4 accuracy at 86.09% (Table 3). Based on the large (~30%) improvement when D Doouree is provided, we conclude that the QuALITY corpus Devurce is sufficiently niche to serve as an appropriate testhed.
4 MAIN EXPERIMENTS
In this section, we present our main experimental results!. Using GPT-42 as our prompted model LMang. we apply EntiGraph the 1.3M token QuALITY corpus Deoures. generating a 455M token synthetic corpus. For the remainder of the paper, we refer to the former as the "Raw corpus" and the latter as the "EntiGraph corpus". Additional details on these corpora are provided in Appendix C.
We continually pretrain Llama 3 8B (Dubey et al., 2024) with causal language modeling on the 455M token EntiGraph corpus. In $4.1, we describe our CPT procedure and introduce two natural baselines. In $4.2, we evaluate on the QuALITY test queries Quea- In $4.3, we show that synthetic CPT using EntiGraph is compatible with downstream instruction tuning (Ouyang et al., 2022).
'oode https://ee thub. cow/ tong /ee///yyttttttteee i i inndd frwed_rretrrininn git. We use the gpt-4- -ttroo model as of Aug. 19, 2024.
5

---

Published as a conference paper at ICLR 2025
4.1 CONTINUED PRETRAINING PROCEDURE
EntiGraph CPT. In our main continued pretraining experiment, continually pretrain Llama 3 8B Base on the 455M token EntiGraph corpus for epochs with replay on the RedPajama dataset (TogetherAI, 2023). Hereafter, we refer this model "EntiGraph CPT". We discuss CPT details in Appendix D. Next, we describe two baselines to which we compare in closed-book QA ($4.2).
Raw CPT baseline. The first baseline continues pretraining Llama 3 8B Base the 1.3M token Raw corpus of raw QuALITY articles Dsource- We jointly tune the number of epochs and RedPajama replay rate, obtaining the "Raw CPT" model. Further tuning details are provided in Appendix D.
Rephrase CPT baseline. Another simple synthetic data augmentation procedure is to rephrase QuALITY articles repeatedly. Maini al. (2024) and Ovadia et al. (2024) execute a systematic extension of this idea (cf. $1.1). Based on their approaches, we craft a "Rephrase bascline" which repeatedly applies three fixed prompts (easy, medium, and hard rephrase)² to the QuALITY articles at temperature 1.0. We stopped generating paraphrases at 38M tokens, where we observed clear gap in QA evaluations from EntiGraph CPT and a slower scaling trend (Figure 2). We refer this data as the "Rephrase corpus" and the continually pretrained model as "Rephrase CPT".
4.2
QUESTION-ANSWER
EVALUATIONS
Next, we present our closed-book QA evaluations with the QuALITY test queries Ques-
Evaluation procedure. Each QuALITY question is four-choice, single-answer multiple choice question (similar to MMLU, Hendrycks et al. (2021)). We evaluate with 5-shot tf-houtt prompting (Brown et al., 2020; Wei et al., 2024) and provide our prompt in Appendix 1.1.
EntiGraph scaling. We find that CPT on the 455M token EntiGraph corpus improves closed-book QA accuracy from 39.49% (for Llama 3 8B Base) to 56.22% (Figure 2). A natural question is how accuracy scales as we synthesize and train more tokens with Enti- Graph. To test this, we randomly subsam- ple without replacement the EntiGrapp corpus with varying sample sizes, continually pretrain Llama 8B Base on each subsample, and plot accuracy versus sample size in Figure 2. We observe log-linear scaling of the accuracy in the number of synthetic tokens used for CPT, up to 455M tokens. We mathematically investigate the scaling properties of EntiGraph in $6. In broad strokes, we postulate that QuALITY ac- shape
GPT-4 (51.30%))
EmiiGraph CPT
55.0
(44
Rephrase
(59.49%)
QA Accuracy
30.0
47.5
45.0
42.5
Number synthetic tokens (in Millions)
Figure 2: Accuracy on the QuALITY question set Qeet (y-axis) as function of the synthetic token count (x- axis). The accuracy of synthetic continued pretraining using the EntiGraph data augmentation algorithm (Enti- Graph CPT) scales log-linearly up to 455M tokens.
curacy follows mixture-of-exponentie with three stages: (i) linear growth, (ii) log- linear growth, and (iii) asymptotic plateau.
Comparison with baselines. Raw CPT (green line) underperforms even Llama 3 8B (dashed black line). We postulate two explanations: (i) The Raw corpus follows a narrower, different distribution than the Llama pretraining corpus; heavily training on it may harm the LM's English capabilities. (ii) The limited diversity of knowledge representations in the Raw corpus leads limited knowledge acquisition due to problems such as the reversal curse (Berglund et al., 2023). Rephrase CPT scales poorly compared with EntiGraph (Figure 2), suggesting that for synthetic CPT to scale well, the synthetic data must be sufficiently diverse. EntiGraph tackles this problem using hierarchical prompting strategy that externalizes diversity knowledge graph's combinatorial relationships.
4.3 INSTRUCTION FOLLOWING EVALUATIONS
Next, we explore more general test queries beyond the test queries Qiea- Concretely, we perform in- struction tuning on EntiGraph CPT to obtain EntiGraph Instruct. We demonstrate that synthetic CPT on the EntiGraph corpus is compatible with instruction tuning; EntiGraph Instruct can directly use
Maini al. (2024) include 4th prompt to generate synthetic QA pairs. We defer this task-specific QA finetuning method Appendix and focus task-agnostic haselines for learning generic knowledge.

---

Published as a conference paper at ICLR 2025
knowledge obtained during synthetic CPT in instruction following tasks, without test-time access to
the QuALITY corpus Dsoure.. We detail our instruction tuning procedure in Appendix D.
Instruction tuning qualitative examples. We
first present qualitative examples that demonstrate EntiGraph Instruct's ability to follow instructions related to QuALITY articles. First, we ask the model to summarize a QuALITY article with ex- plicit reference to the title and author, but access to the article itself (Table 2, top row). Next, we show that even without an explicit reference to the title and author, knowledge of the article is stored in the model's s parameters and can affect its behav- ior (Table 2, middle row). Finally, we provide an example where the model performs comparison using knowledge across two articles (Table 2, bot- tom row). Albeit artificial, this shows that though EntiGraph does not synthesize data that simulta- neously involves multiple articles, the model can reason about their interaction using its parametric knowledge. We provide full responses in Table 6. Evaluating closed-book summarization. also present quantitative metrics for summarization, a well-studied instruction following task. We compare EntiGraph Instruct summaries of QuALITY articles with human- written summaries from sQuALITY (Wang et al., 2022), variation of QuAL- ITY with provided human summaries. Common scalar summarization metrics such as ROUGE (Lin, 2004) or BERTScore (Zhang* et al., 2020) mostly evaluate text similarity between the summary and source articles, and may not accurately reflect summarization quality for abstractive systems (Zhang al., 2024b). We use a simple, automated evaluation metric based on pyramid evaluation (Nenkova et al., 2007: Gao et al., 2019) that measures both the hallucination rate and how well the summary captures the salient claims of the original article. Our approach uses GPT-4 to (1) split the sum- mary into atomic claims (Min et al., 2023), (2) decide whether each claim is true/false based on the source article, and (3) determine if true claims are salient to the article's main message. We hence obtain the count of false and salient claims for each summary, normalize these by the corresponding count from the human summary, and report the average of these normalized metrics in Figure 3. Appendix 1.2 provides further details.
Table 2: EntiGraph Instruct examples.
Explicit reference: Down".
Summarize "Defining Decay
The article "Defining Decay Down" by David Plotz discusses [...] Dentists began facus on cosmetic dentistry, [..]]
Implicit reference: How has dentistry in the U.S. changed?
1. Increase in cosmetic dentistry [...]
2. Use of technology: [...]
Cross article instruction: Compare David Plotz's commentary on American dentistry and the movie Fight Club?
David Plotz's commentary style is different when he analyzes American dentistry and when he dis- cusses the movie Fight Club, [...]
We
Results discussion. In Figure 3, we compare four sum- marizers: EntiGraph Instruct, Raw Instruct, GPT-3.5, and GPT-4. We provide each summarizer with two different prompts, asking for short and long summaries (prompts in Appendix 1.2). When we request more detailed sum- maries, Raw Instruct hallucinates and generates more false claims with little improvement in the number of salient claims. In contrast, EntiGraph Instruct can gener- ate more salient claims the summary gets longer, with small increase in the number of false claims (similar to GPT-3.5 and GPT-4 levels). The gaps in both salient and false claim rates are sufficiently large that these results likely hold beyond our particular metric. We complement the automated evaluation metrics above with several qual- itative examples in Appendix I.2.
RawCPI short
GPT4
Taise ma relettve human
G.T long
Human
GPI-4 whort
Sallenn claims relative human
Closed book summarization: number of false claims (g-axis) versus num- ber of salient claims (π-axis) normalized by the human summary.
Figure 3:
5 OPEN-BOOK EXPERIMENTS
Next, we consider an open-book setting with the domain-specific corpus Diouree available at test time. In this widespread setting, retrieval-uugmented generation (RAG; Lewis et al. (2020)) is the predominant approach. A natural question whether the parametric knowledge learned through syn- thetic CPT using EntiGraph complements the parammerrei knowledge accessed using RAG. We answer this question by comparing state-of-the-art RAG pipeline with and without Entigraph CPT.
7

---

Published as a conference paper at ICLR 2025
EntiGraph CPT RAG
Llama 3 8B Base + RAG
GPT-4 + Oracle RAG
GPT-3.5 + Oracle RAG
Accuracy
Recall
Accuracy
Recall@8
Accuracy
Recall@8
Accuracy
@
62.60
99.63
60.35
99.63
86.09
100.0
72.60
100.0
Table 3: QuALITY questionn-nsswett accuracy and recall rate in the open-book surreval-augmeent genera- tion (RAG) setting. EntiGraph CPT and Llama 3 8B Base are used in RAG pipeline (ef. $5 for setup details). @8 is defined as the proportion of questions for which the salient article appears in the top reranked document chunks. GPT-4 and GPT-3.5 Oracle RAG provide an upper hound with perfect retriever, by placing the entire relevant document in-context.
RAG evaluation setup. Our RAG pipeline follows established best practices (Lewis et al., 2020; Gao et al., 2024). It involves an offline stage which indexes document chunks, followed by inference-time retrieval, reranking, and placement of those chunks in a few-shot LM prompt. Throughout, we use OpenAI API-based embedding model, FAISS as our similarity search index (Douze et al., 2024), and Cohere gerank-english-ae 0 (Cohere, 2024)
(Neelakantan et al., 2022)
reranker. Following the evaluation procedure detailed in $4, we evaluate parallel RAG pipelines on the QuALITY multiple choice test set using few-shot chain-of-thouge prompting. All hyperparameters are tuned separately for each LM's RAG pipeline. We refer the reader to Appendix F for further details on our RAG evaluation setup.
EntiGraph continued pretraining complements RAG. We observe in Table 3 that EntiGraph CPT outperforms Llama 3 8B Base, the model from which it is continually pretrained. These re- sults demonstrate that the knowledge internalized through synthetic CPT is complementary that accessed during RAG, and demonstrate competitive new recipe for small corpus QA: (1) synthetic data augmentation, (2) continued pretraining, and (3) RAG.
EntiGraph continued pretraining alone approaches RAG performance. These results also contextualize the effectiveness of EntiGraph in the closed-book book, parametric knowledge setting ($-4). Comparing Figure 2 and Table 3, we observe that adding RAG to Llama 3 8B Base improves accu- racy by 20.86% 4 + 60.33 60.35%). On the other hand, continued pretraining of Llama 3 8B Base on the EntiGraph corpus improves accuracy by 16. 73% (39.49% + 56. 56.22%). Hence, EntiGraph continued pretraining provides > 80% of the absolute performance improvement RAG, even in a small corpus setting where RAG recall is nearly perfect.
Overall, our results show that the parametric knowledge acquired in EntiGraph continued pretraining composes with realistic knowledge-intenn intensive QA pipelines, and that EntiGraph continued pretrain- ing althe-without test-time corpus access nearly competitive with a strong RAG baseline.
6 THEORETICAL ANALYSIS OF ENTIGRAPH SCALING
It may seem surprising that simply "rewriting" the source documents Duounee can improve perfor- mance at all ($4), as EntiGraph does not explicitly add new knowledge beyond Doouree We postu- late that EntiGraph "rearranges" Doorree into a layout more amenable learning. For example, in Diounee the entity pair (A, B) may appear together in some sentences and (B, C) in others. As result, models trained directly on Doource may learn the (A, B) relation and the (B, C) relation, but not the (A, C*) relation (Akyürek et al., 2024). We build mathematical model to formalize this in- tuition ($6.1) 1) and provide a quantitative prediction that the scaling trend of EntiGraph CPT follows cecttur--o--xppoeettte shape ($6.3), which fits well with our empirical observations (Figure 4).
6.1 TOY MODEL SETUP
In this toy model, we use V to denote the set of entities, and represent the source documents D sourre with pairs of known relations Davurce C { y) E V2 * y). We assume that each relation pair in 222 appears in the source documents D independently at random, with probability p. Mathematically, P [[z, y) E Dsouree] p for all V and y V with + y. We write V = | and assume that p = X/V, for some constant À >
Training as memorization. We model the learning of factual knowledge memorization pro- cess, in which a model memorizes the relations it is trained on but does not meaningfully generalize beyond them (Yang et al., 2023; Feldman, 2020). In this view, language model's knowledge can be represented by a matrix M E (0, VVV such that M M(x, y) = 1 if the model "knows" the (ar, y) relation and equals 0 otherwise. Then, training directly on the source documents Doouce sim-
8

---

Published as a conference paper at ICLR 2025
ply means setting all entries that appear in Sourree to 1, denoting that the model has memorized the relations given in the source documents. Mathematically, we denote this model trained on Ssorre by the matrix M, {0, 1) which has i..d. Bernoulli off- off-diagonal entries with mean p.
EntiGraph synthetic data augmentation. Given the source documents D Dsources we define the following iterative procedure of synthetic data generation: for each f = 1,2,...
Entity pair selection: Sample (2+, E {(I,)) v) E 122 I + y) uniformly at random.
Relation analysis: Generate the "relation between (I7,, Dt)" by performing breadth-fisst search (BFS) on the directed graph represented by the adjacency matrix Mo starting at 2t. If no such path exists, do nothing. If there exists a path (xte =t zi. .,.,..,,, zt ye) connecting 2; to yt. define D: = { 2)), (Iz, 22,,,2) =.),.... (Iz, z ), (Iaa, We) } U DD--1, where we assume Do = =Duuucee The model trained on this round of synthetic data M.
M + E(a.)ED, Lsy. where Ly E (0, 1).1 V)V is binary matrix with (x v) = 1 and O otherwise.
This mirrors the relation analysis step for the EntiGraph synthetic data augmentation algorithm (Step $2.2). With the setup above, the index t is analogous to the number of synthetic tokens that the model has generated, and the model's 's knowledge is captured by how many ones the matrix M, contains. To make this connection precise, we define the link density (or accuracy) of M, to be Mc, = [B[] M, lla (V)((V(V-1)) where the expectation is taken over the randomness arising from the synthetic data generation process and not the source documents Dsouce. and ||M]| denotes Eng |- We use the notation Acc as this is intended to emulate the accuracy on QuALITY test queries studied in the experimental sections 4 and $5).
6.2 RIGOROUS UPPER AND LOWER BOUND
In this section, we derive rigorous upper and lower bounds on the scaling trend of Ac)(
Definition 1. Let C^ = (1 -1-ppt p(A)), where p()) denotes the extinction probability for a Poisson(A)
branching process (i.e., p is the smallest solution in [0, the fixed-point equation p exp(X()
1))). For any fixed ε > 0. we further define CLB = 1 v(v-1): CUB = V(V-1) log X
(1+s)
Theorem 1. For any time 2 1 and any > 0, the link density satisfies, with probability + 1,
(p+ CC (1 (B)) (1 - S Ac) < (p+ (1 CUB) (1+ as V 30.
Even though Theorem provides mathematically rigorous upper and lower bounds the scaling trend of Ac)(MM), the exact growth curve is more intricate, as we will show next.
6.3 AN ANALYTICAL FORMULA
zmpriia Momtr of ecporantal it:
QUALITY 04
EntGraph Accuracy
We analyze the link density Ac)(Mi using Pois- son branching process approximation of the cluster growth of vertices. This approach yields mixture-of- exponential scaling trend
17.5
Acc(M)) ~ ~pp (1 80
(()) 11-2u())
ax)²
(2)
if 10'
k=1
Number of synthetic tokens (nn Millians)
where ~ means that A/B converges to in prob- ability as V co. The parameter C' governs the link density Acc(M, as ( - oe and is determined by the proportion of reachable pairs of vertices in the initial matrix My. p1-) is the probability mass function on k, which controls the proportion of pairs of vertices with specific decay rate. The parameters ju(-) and a depend on M, in more intricate manner (cf. Appendix G for a full derivation). We find that (2) accurately fits the empirical scaling trend of EntiGraph CPT accuracy up to 455M synthetic tokens (Figure 4). We discuss curve fitting in Appendix G.I, where we show that the mixture-of-exponee linear growth; (ii) log-linear growth; (iii) asymptotic plateau.
Figure mixture-of-expooentia function (2) closely fits scaling trend of EntiGraph CPT with respect to synthetic token count.
shape grows in three phases: (i)
DISCUSSION AND CONCLUSION
7.1 LIMITATIONS
Because EntiGraph synthesizes data using a prompted LM, there is a risk it may hallucinate and fabricate non-existent entities relations. Although our synthesis process is grounded by the source

---

Published as a conference paper at ICLR 2025
documents, it is assumption that LMaug is capable enough to generate faithful synthetic data when conditioned on D Dsource. We quantitatively test the factuality of the EntiGraph corpus by randomly subsampling 150 sentences from it and manually labeling each sentence's 's factuality (Appendix A.2). We find roughly half of the sentences are subjectíve, and the other objective half is almost always factual. We postulate that factuality is high because QuALITY articles are relatively simple given the capability of the prompted LM. If EntiGraph were applied to more challenging content like a complex research paper, it is possible that the prompted model could be more prone to hallucination.
On the other hand, since we use a strong prompted LM gpt turbo to generate synthetic data, one might be concerned that our performance gains come from distilling it. To probe this, we per- form an ablation study where we replace qpt-4-turbo with Llama 3.1 8B Instruct, a substantially weaker model that is obtained from the same base model as EntiGraph CPT, in Appendix A. We generated 334M EntiGraph tokens using Ilama 3.1 8B Instruct and found a consistent log-linear trend with the same slope but lower intercept (Figure 5) compared with GPT-4 generation. This ablation suggests that EntiGraph operates by genuinely teaching the model knowledge about the QuALITY corpus, rather than serving as a vehicle to distill a powerful prompted LM.
7.2 FUTURE DIRECTIONS
Continued scaling beyond real data. The large but finite body of numan-written text is rapidly be- ing consumed. Villalobos al. (2024) predict that frontier language models will exhaust all public, numan-generater text in 2028. As we transition from a data-rich to a cata-constrained regime (Ka- plan et al., 2020; Muennighoff et al., 2023), further scaling will require to extract more knowledge from existing data. We demonstrated that synthetic continued pretraining with EntiGraph effectively extracts more knowledge from small corpora, which could help us learn from proprietary datasets or tail knowledge that appears only once or twice on the internet. It is an open question whether synthetic data generation methods like EntiGraph could improve data efficiency more generally on standard pretraining data and without relying upon stronger prompted model.
Alternatives to long-context language models. Recent work handles long user queries (e.g., 1M- 1MM10M+ tokens) using efficient attention (Dao et al., 2022; Liu et al., 2023; Gemini, 2024) or ar- chitectures that are sub-quadratic in the context length (Tay et al., 2022; Gu et al., 2022; Gu & Dao, 2024; Sun et al., 2024). In settings where many queries share long prefix e.g., a corporation's proprietary documents or other prompt caching use cases (Anthropic, 2024a) could instead continue pretraining on the prefix to internalize its knowledge, and then perform standard quadratic attention on shorter queries. This approach pays a fixed training cost to amortize the prefix's 's knowl- edge into the weights of a model, and then benefits from shorter context lengths (Gururangan al., 2020: Snell et al., 2022). By adapting the continued pretraining paradigm from 10B-1008 tokens to as little 1.3M tokens, synthetic continued pretraining approach could enable unsupervised learning of shared text prefixes at much smaller and more practical token counts.
7.3 CONCLUSION
Continued pretraining with next-token prediction is remarkably effective in teaching pretrained lan- guage models new knowledge. but date has only been applied successfully in broad, data-rich domains with 10B-100B+ tokens. We downscale continued pretraining to small, specialized cor- pora with ~ IM tokens using synthetic continued pretraining: converting small corpus into large synthetic one with diverse representations of knowledge, and continuing pretraining on it.
synthetic data augmen- tation algorithm. Synthetic continued pretraining with EntiGraph demonstrates consistent scaling in downstream closed-book QA performance up to 455M token synthetic corpus, whereas baselines such as continued pretraining on the small corpus or synthetic paraphrases show no improvement or scale slowly. Moreover, the acquired parametric knowledge composes with instruction tuning and retrieved non-parametric knowledge in open-book setting. Lastly, we present a simplified mathematical model of EntiGraph and derive a functional form for its scaling trend, which closely matches our empirical trend. We hypothesize that EntiGraph's "externalization" of the synthetic data generation process to a combinatorial structure- in this case, a knowledge graph over entities-ae is a generally useful strategy in synthesizing highly diverse data and a promising object for future study.
We instantiate this approach using EntiGraph. knowledge
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGEMENT
Zitong Yang would like to thank Samy Jelassi for feedback on a preliminary version of this work, Ruiqi Zhong for discussion regarding context distillation work, Xiang Lisa Li for discussion about reversal curse work, and the participants of the statistics seminar at Stanford University for their insightful feedback about a preliminary version of this work. We also thank the Tatsu Lab for con- structive feedback and interesting discussions that have helped improve the paper. Zitong Yang is supported by the Albion Walter Hewlett Stanford Graduate Fellowship. Neil Band acknowledges funding from an NSF Graduate Research Fellowship and a Quad Fellowship. T.H. was supported by a grant from Samsung Research, gifts from Panasonic Research, the Google Research Scholar Program, and the Tianqiao and Chrissy Chen Institute, as well as the NSF grant 113-2338866. E.J.C. is supported by the Office of Naval Research grant 14, the Simons Foundation under award 814641.
the National Science Foun-
dation grant
REFERENCES
Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Al- lie Del Giorno, Ronen Eldan, Sivakanth Gopi, Suriya Gunasekar, Mojan Javaheripi, Piero Kauff- mann, Yin Tat Lee, Yuanzhi Li, Anh Nguyen, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shi- tal Shah, Michael Santacroce, Harkirat Singh Behl, Adam Taumann Kalai, Xin Wang, Rachel Ward, Philipp Witte, Cyril Zhang, and Yi Zhang. Phi-2: The surprising power of small lan- guage models, 2023. URL https: / www. mi crosoft. com/en- us reeeareeaaasee messearcc/blog/ blog/ weteehe-surprssingee oft.rreesuuuppiiiiiiooooo-r---r- -twpaaan--oooe------------------ caaamaal-langgaan -aw.-aggegeemeeeeeee/a/ models/.
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Ben- haim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Qin Cai, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Yen-Chun Chen, Yi- Ling Chen, Parul Chopra, Xiyang Dai, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Victor Fragoso, Dan Iter, Mei Gao, Min Gao, Jianfeng Gao, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi. Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Ce Liu, Mengehen Liu, Weishung Liu, Eric Lin, Zeqi Lin, Chong Luo, Piyush Madan, Matt Mazzola, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Penez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sam- budha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shi- tal Shah, Ning Shang, Hiteshi Sharma, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Xin Wang, Lijuan Wang, Chunyu Wang, Yu Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Haiping Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Sonali Yadav, Fan Yang. Jianwei Yang. Ziyi Yang. Yifan Yang, Donghan Yu, Lu Yuan, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou, Phi-3 technical report: highly capable language model locally on your phone, 2024. URL httph:tt///// /awiiviiiii..///ii///aaa/a/aaava org/ abs/ / 2044.44020 14219.
Afra Feyza Akyürek, Ekin Akyürek, Leshem Choshen, Derry Wijaya, and Jacob Andreas. Deduc- tive closure training of language models for coherence, accuracy, and updatability. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Lingnistics ACL 2024, pp. 9802-9818, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL https: /ttpps://cc/e //tp///aaaatttt www.ennnncllggooooooooooooooeeeo org/2024 findings-acl. 584.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation, 2024. URL https: http://///xxvvvv//a org. abs /21//.1..1..00
Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models, 2023.
Dana Angluin. Queries and concept learning. Machine Learning, 2:2, 1988. URL https: / / /api... semant i cachol a wtwpr//orossS.:1113500

---

Published as a conference paper at ICLR 2025
Anthropic. Prompt caching (beta), 2024a. URL https: Utl.tttpss/bbc...attriic...oo
docs htoc://uullletttttt-ccaadaarpppp
Anthropic. The Claude 3 Model Family: Opus, Sonnet, Haiku. https://www-cdh:
anthropic. com/ de Bba Card_Claude_3 pd,, 2024b.
Anas Awadalla, Mitchell Wortsman, Gabriel Ilharco, Sewon Min, Ian Magmusson, Hannaneh Ha- jishirzi, and Ludwig Schmidt. Exploring the landscape of distributional robustness for ques- tion answering models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Find- ings of the Association for Computational Linguistics: EMNLP 2022, pp. 5971-5987, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 18253/11/2022..f.. hann:iiiinnnnn--eln..aa.. URL atttps://aclee aal.aaaaaannttoyy........ or2/2022.
Hindings-emnlly 441.
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Mareus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, In The Twelfth International Conference on Learning Representations, 2024.
URL
net wtw.aarwwwwweeeeetooomte..e.....
Maria-florina Balcan, Avrim Blum, and Ke Yang. Co-training and expansion: To- wards bridging theory and practice. In L. Saul, Y. Weiss, and L. Bottou (eds.), Ad- vances in Neural Information Processing Systems, volume 17. MIT Press, 2004. URL ngs neuri ps. hcc/paper_files/// p/pel//pppe///00//11e 9457fc228ceb4D0 03e1 3533e4 -Paeer pdf.
https:://rroeee
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Kor- bak, and Owain Evans. The reversal curse: Lims trained on "a is b" fail to learn "b is a", 2023.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning, 2019. URL https: //arxiv.org/abtl 1905. 02249.
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Pro- ceedings of the Eleventh Annual Conference on Computational Learning Theory, COLT" 98, pp- 92-100, New York, NY, USA, 1998. Association for Computing Machinery. ISBN 1581 130570. doi: 10. 10.1145/2794332799922 URL https:
279962.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL /ttps://proceeddii nwwrips. cttpeeeeeeeeeeeeeeeepeeeeeeeeeee httppee__iilesppppe///0000
Harrison Chase. LangChain, 2022. URL https: httptthhuu...mmmmmmaeaeee com/ www..ooogoaannna............ langchain.
Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexan- dre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawezuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi. and Antoine Bosselut. Meditron-70b Scal- ing medical pretraining for large language models, 2023. URL https: /ttp://aaxivvvv/.. org/ abss/ 2311 1311... 1 16079.
Roi Cohen, Eden Biran, Oni Yoran, Amir Globerson, and Mor Geva. Evaluating the ripple effects of knowledge editing in language models. arXiv preprint arXiv: /2976, 2023.
12

---

Published as a conference paper at ICLR 2025
Cohere.
Improve search performance with single line of code, 2024. URL https:
com/reran rerank.
Pierre Colombo, Telmo Pires, Malik Boudiaf, Rui Melo, Dominic Culver, Sofia Morgado, Etienne Malaboeuf, Gabriel Hautreux, Johanne Charpentier, and Michael Desa. Saullm- and saullm- 141b: Scaling up domain adaptation for the legal domain, 2024a. URL https: arxiv. org/abs/ / 2407 19584.
Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre F. T. Martins, Fabrizio Esposito, Vera Lúcia Raposo, Sofia Morgado, and Michael Desa. Saullm- 7b: A pioneering large language model for law, 2024b. URL /ttpps////xxiv./r////a///
2403. 03883.
Common Crawl. Common crawl. https:/ httprs://coomonrwwl.orrog. 2007.
Tri Dao, Daniel Fu, Stefano Ermon, Atri Rudra, and Christopher Re. Flashattention: Fast and memory-efficient exact attention with 10-awareness. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
URL https:ttp///m /pwereveeeeeeeeeeeeeeee/eereeeee net forum? i ddHAADqEPS immx.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library, 2024.
URL https:t/taxiii...t/////a////
/ttp/axxvvoorrg/.a abs/a//s0/000 2201. 08281.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux. Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor I Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasami. Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Yearr, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas. Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie. Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta,
13

---

Published as a conference paper at ICLR 2025
Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag. Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang. Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Bran- don Spence, Brani Stojkovic, Brian Gamido. Britt Montalvo, Carl Parker. Carly Burton, Catalina Mejia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute. Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni. Francisco Guzmán. Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean- Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen. Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres Josh Ginsburg, Junjie Wang. Kai Wu, Kam Hou U. Karan Saxena. Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang. Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo. Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chemoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan. Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say. Sheng Feng, Shenghao Lin. Shengxin Cindy Zha, Shiva Shankar. Shugiang Zhang, Shuqiang Zhang, Sinong Wang, Sncha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li. Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vítor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vito,, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024. URL httpt:tt///aaaet org/ abs/2407 21783.
Rick Durrett. Random graph dynamics, volume 20. Cambridge university press, 2010.
14

---

Published as a conference paper at ICLR 2025
Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings the
the 52nd Annual ACM SIGACT Symposium on Theory of Computing. STOC 2020. pp. 954 959, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 978145036979,, doi: 10.1145/3357133333422
URL https: / /doi. org/10. 1141/113537111 3384290.
Yanjun Gao, Chen Sun, and Rebecca J. Passonneau. Automated pyramid summarization evalu- ation. In Mohit Bansal and Aline Villavicencio (eds.), Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL). pp. 404-418, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653//1/K1111000 1038. URL https: / /ttl antho ogy wrg/ol 9-1038.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmmented generation for large language models: A survey, 2024. URL httph:tpp///// wtt.//iivvv/o//////////////aaa www..irg./ooooooooooo.ooooooooor abs 2211 10997.
Team Gemini. Gemini: family of highly capable multimodal models, 2024. URL https:
//tpxxiv.rggaaas/////./..so. /2312 11805.
Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. arXiv preprint (rXXiv:903:0447000 2019.
lan J. Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation catastrophic forgetting in gradient-base neural networks, 2015. URL https: wwwpxii.rrg/abb//// 6211.
Stephen Grossberg. Studies of mind and brain: Neural principles of learning, perception, devel- opment, cognition, and motor control, volume 70. Springer Science & Business Media, 2012.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2024.
URL https: httpss//pppeevvveeee net. forum?iiii F00
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured
state spaces. In International Conference on Learning Representations, 2022. URL https: //openzeview. n htte///oormmmdmdd/d/ee
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahem, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas Reinforced self-training (rest) for language modeling. 2023. URL https:tpt.......... : / arxiv. org/ abs //0///300000 08998.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang. Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023. URL https: http///xxxvv///////// abs/ / 2206. 11644.
Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, Deepak Gopinath, Dian Ang Yap. Dong Yin, Feng Nan, Floris Weers, Guoli Yin, Haoshuo Huang, Jianyu Wang, Jiarui Lu, John Pee- bles, Ke Ye, Mark Lee, Nan Du, Qibin Chen, Quentin Keunebrock, Sam Wiseman, Syd Evans, Tao Lei. Vivek Rathod, Xiang Kong. Xianzhi Du Yanghao Li. Yongqiang Wang, Yuan Gao, Zaid Ahmed, Zhaoyang Xu, Zhiyun Lu, AI Rashid, Albin Madappally Jose, Alee Doane, Alfredo Bencomo, Allison Vanderby. Andrew Hansen, Ankur Jain, Anupama Mann Anupama, Areeba Kamal, Bugu Wu, Carolina Brum, Charlie Maalouf, Chinguun Erdenebileg, Chris Dulhanty, Do- minik Moritz, Doug Kang, Eduardo Jimenez, Evan Ladd, Fangping Shi, Felix Bai, Frank Chu, Fred Hohman, Hadas Kotek, Hannah Gillis Coleman, Jane Li, Jeffrey Bigham. Jeffery Cao, Jeff Lai, Jessica Cheung, Jiulong Shan, Joe Zhou, John Li, Jun Qin, Karanjeet Singh. Karla Vega, Kelvin Zou, Laura Heckman, Lauren Gardiner, Margit Bowler, Maria Cordell, Meng Cao, Nicole Hay, Nilesh Shahdadpuri, Otto Godwin, Pranay Dighe, Pushyami Rachapudi, Ramsey Tantawi,
15

---

Published as a conference paper at ICLR 2025
Roman Frigg, Sam Davarnia, Sanskruti Shah, Saptarshi Guha, Sasha Sirovica, Shen Ma, Shuang Ma, Simon Wang. Sulgi Kim, Suma Jayaram, Vaishaal Shankar, Varsha Paidi, Vivek Kumar, Xin Wang, Xin Zheng, Walker Cheng, Yacl Shrager, Yang Ye, Yasu Tanaka, Yihao Guo, Yun- song Meng, Zhao Tang Luo, Zhi Ouyang, Alp Aygar, Alvin Wan, Andrew Walkingshaw, Andy Narayanan, Antonie Lin, Arsalan Farooq, Brent Ramerth, Colorado Reed, Chris Bartels, Chris Chaney, David Riazati, Eric Liang Yang, Erin Feldman, Gabriel Hochstrasser, Guillaume Seguin, Irina Belousova, Joris Pelemans, Karen Yang. Keivan Alizadeh Vahid, Liangliang Cao, Mah- yar Najibi, Marco Zuliani, Max Horton, Minsik Cho, Nikhil Bhendawade, Patrick Dong, Piotr Maj. Pulkit Agrawal, Qi Shan, Qichen Fu, Regan Poston. Sam Xu, Shuangning Liu, Sushma Rao. Tashweena Heeramun, Thomas Merth, Uday Rayala, Victor Cui, Vivek Rangarajan Sridhar, Wencong Zhang, Wenqi Zhang, Wentao Wu, Xingyu Zhou, Xinwen Liu, Yang Zhao, Yin Xia, Zhile Ren, and Zhongzheng Ren. Apple intelligence foundation language models, 2024. URL https: //wpvvvavvvv///////////a.//awav. org/ abs 2407. 21075.
Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats I.. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. Continual pre-training of large language models: How to (re)warm your model?, 2023. URL https:// httpiaaxiviv.//////eee/ea/ae...a org/ aba/ab/// / 208 04014.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342-88-2 8360, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/200 (0/55/////222...oo awwaaccllin....m 740. URL fittss//aclanntho..o org/ 202 aal. 740.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021. URL https: /ttps//////p/eeeeee/e net/ f d7KB jmI 3GmQ
Remco van der Hofstad. Random Graphs and Complex Networks. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2016.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. Unnatural instructions: Tun- ing language models with (almost) human labor. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 6lst Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14409-14428, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.1 12538666//112223aclhhe https:
URI.
wttg//22...eo. a.002...alllonn 806.
Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang. Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Pro- cessing, pp. 1051-1068, Singapore, December 2023, Association for Computational Linguis- tics. doi: 10. 101118663vvv1 h/22222222...mmmmaaaaaa emnlp-main. 67.
URL https: /aaaaaaaantllooyoo/oooo org/2023.
Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L. Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models, 2024. URL hitps: / aaaiiiiiii..///.i///aaaaaaaaiava wtrg/bs//////oooooo////..//ea.ar 24032443..00 08763.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In Proceedings the on Machine Learning, ICML'23. JMI.R.rrg, 2023.
the 40th International Conference
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https: / arxiv. org/ 2tbb//200 08361.
Richard M Karp. The transitive closure of a random digraph. Random Structures & Algorithms, (1):73-93, 1990.
16

---

Published as a conference paper at ICLR 2025
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler L. Hayes, and Christopher Kanan. Mea- suring catastrophic forgetting in neural networks. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelli- gence Conference and Eighth AAAI Symposium on Educational Advances Artificial Intelli- gence, GAAT'1/AAT 18/AATI AAAI Press, 2018. ISBN 978-1-57735-800-8.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabska-Barwinska Demis Has- sabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521- 3526, 2017. doi: 10.1073/pnas.10 18351144 URL https: wwwwwwwww... pnas wttp/o///////////////ot////ooost 10. 10.1073/ppas 1611811110 611885114
Hunter Lang, Monica N Agrawal, Yoon Kim, and David Sontag. Co-training improves prompt- based learning for large language models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 11985-12003. PMLR, 17-23 Jul 2022. URL httpss//ppccceeinng./e/ mllr, press/ v162 html.
Nong-Hyum Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. ICML 2013 Workshop: Challenges in Representation Learning, 2013.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmeenter generation for knowledge- intennive nip tasks. In Proceedings the 34th International Conference on Neural Information Prucessing Systems, NIPS '20 Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 7781713829546
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with lan- guage models, 2022. URL https: arxiv. org/ abs/ 2206 14858.
Haoran Li. Qingxiu Dong, Zhengyang Tang, Chaojun Wang. Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang. Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei. Synthetic data (almost) from scratch: Generalized instruction tuning for language models, 2024. URL https:ttt.........t. //waxivv../////////////a/////aia org/ abs /2e.e//422222 13064
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https::////. tttts://githule
_eael, 2023a.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are you need ii: phi-1.5 technical report, 2023b. URL hww.thtpttteetccc httpps://axxivorrg. abs/2305 05463.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: / /a//aaaaanttooooo
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near- infinite context. In NewrlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023. URL https::////om....t.. wwwpre/iee./ereeeeeeeo http:/ewwwwwww.eeeeeeeeeeeeeeeee wtwp://rrommooxlcx/xxxxxcxoo/
David Lopez- Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30:6467-6476, 2017.
17

---

Published as a conference paper at ICLR 2025
Pratyush Maini, Skyler Seto, Richard Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14044- 14072, Bangkok, Thailand, August 2024. Association for Computational Linguisties. URL https: http://aaaaaattahaeeoeee
org/ 2024 -ong............. 757.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Gordon H. Bower (ed.), Psychology of Learning and Motivation, volume 24 of Psychology of Learming and Motivation, pp. 109-165. Academic Press, 1989. doi:
URL https: /wwww sciencedireceeettt
Nick Mecklenburg, Yiyou Lin, Xiaoxiao Li, Daniel Holstein, Leonardo Nunes, Sara Malvar, Bruno Silva, Ranveer Chandra, Vijay Aski, Pavan Kumar Reddy Yannam, Tolga Aktas, and Todd Hendry. Injecting new knowledge into large language models via supervised fine-tuning, 2024.
URL /ttps://arxiv..oo www.aaaavv...o org/ abs 2404. 00213.
Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. Locating and editing factual asso- ciations in GPT. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: /ttp///epppeeeeeeeeeeeeeee net/ ii=- h6WA
Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, 2023. URL https://///. / wpwpreveeeeeee.ee////p/p. net/ www.moiiodioooooooooooooo www.rmmmdddkkkkaAaTYYYy
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of fac- tual precision in long form text generation, 2023. URL htwps:tt///ettt //aasi////vvii.////a org/ ab3//305. 14251.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2022. URL
https: wpwprevieeeeeeeeeeee/eeeeeeeeeee net wwwpddddddddeeeeee0000
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksan- dra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Thirth-seventh Conference on Neural Information Processing Systems, 2023. URL https: http::s///ppeneviiee net forum? ?id= J33.
Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. Text and code embeddings by contrastive pre-training, 2022. URL https: / httpaawwwvv.ga..aaaeaee/eeaooett aass//////// /ttp//22...10...........
Ani Nenkova, Rebecca Passonneau, and Kathleen McKeown. The pyramid method: Incorporat- ing human content selection variation in summarization evaluation. ACM Trans. Speech Lang, Process., 4(2):4es,, may 2007. ISSN 1550-4875. doi: 10.1145/1239912.123991 URL
https: //oii
doi. org/10....... 1141 0144//1139111 121.191...11111...
Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner. Variational continual learning. arXiv preprini 17X0v 2017.
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shaping Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann,
18

---

Published as a conference paper at ICLR 2025
Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang. Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, łukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim. Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini. Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil. David Medina, Aalok Mchta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa. Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Rousse7, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov. Jessica Shich. Sarah Shoker, Pranav Shyam. Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Work- man, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https:/ http://axvvv/v.///////ee///..ev/ org/ abs///b//2// 2203 08774.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 27730-27744. Curran Associates, Inc., 2022. URL https: httttp:s//pproceeddngeeeee neurips cc http:/sscceeppppeeeeepeeeeeeeeee haw.:aaooorrraaaaa----oeeeeeeeee Confer-ooneeeecccecomm .dd..
Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine-tuning or retrieval? com- paring knowledge injection in llms, 2024. URL https:://///. http///xvvvvv.//////ee////e
http://wwwwwww/aaa///// wrgg hbtgg/bb//20/ 2232.00034....
Richard Yuanzhe Pang. Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang. Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. QuALITY: Question answering with long input texts, yes! In Marine Carpuat. effe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North
de Marn-
19

---

Published as a conference paper at ICLR 2025
American Chapter the Association for Computational Linguistics: Human Language Technolo- gies, pp.
Seattle, United States, July 2022. Association for Computational Linguis- URL
tics. doi:
naacl -ma . 391.
Jupinder Parmar,
Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Reuse, don 't retrain: A recipe for continued pretraining of language models, 2024. URL https: / //arxiv.org/ abs /2207.402772
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023. URL https: wwwpxiiivorgaaaoooo org/ aalaab///22/ 03277.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. URL /tt//aaxvvvv// wrgg/bo///////o/// 606.666660 05250.
Vinay Venkatesh Ramasesh, Aitor Lewkowycz. and Ethan Dyer. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2022.
URL https:ttp///ot....t. /ttpss:///peeeeeee /tppnrevieeeeeeeeeeeeeeeeeeeeeee net/ forum? (d=chhss__yPeeeee
R. Ratcliff. Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological Review, 97(2):285-308, 1990. doi: 10.103770003--955X.97 2.285.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2):
123-146, 1995.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Ev- timov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code, 2024.
URL https:tp////m arxiv org/ abs/2308 12950.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1761v:1606.04671, 2016.
Jeffrey C. Schlimmer and Douglas Fisher. case study of incremental concept induction. In Pro- ceedings of the Fifth AAAI National Conference on Artificial Intelligence, AAAI'86, 86, pp. 496-501. AAAI Press, 1986.
Raphael Schumann and Ines Rehbein. Active learning via membership query synthesis for semi- supervised sentence classification. In Mohit Bansal and Aline Villavicencio (eds.), Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 472-481, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/ 11/K19-104 URL https: http://aaaaantttaaae/ee 1.
org/K19-1044
H. Scudder. Probability of error of some adaptive pattem--reoogniioon machines. IEEE Transactions on Information Theory, 11(3):363-37,, 1965. doi: 10. 10.1109/TT..19655153379
Claude Elwood Shannon. Prediction and entropy of printed english. Bell System Technical Journal, 30:50-66 64,0006666660 January 1951. URL http:/ http:://aanne http:///annuugllgee/e/ 1dc. www.u.eee/nnneeee.e. edu/myy/ Shannon1 950...... pdf.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi. Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models, 2024. URL htttpa://aee :/arxivv org/ abs/ 2402. 03300.
20

---

Published as a conference paper at ICLR 2025
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep gener- ative replay. In I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL https:tp: /ttp::::e///c/ooeeeeoeeoo ngs. neur wwt.rips.ccppppp.eeeeeee. cttpssccpppppp_aaeppaaee Serape___fles___ files/ paper/2017/ e
Charlie Snell, Dan Klein, and Ruiqi Zhong. Learning by distilling context, 2022. URL https: / //arxiv.org//////.... 222/
Yu Sun, Xinhao Li. Karan Dalal, Jiarui Xu, Arjum Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, and Carlos Guestrin. Learning to (learn test time): Rnns with expressive hidden states, 2024. URL /tttp:://axxi....tt/// abs/2401 04620.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-oolooinng llama model. 2023.
https: / thub. com/ taass-ulllaa lab/ hts//tanns ford_a
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, 2022.
URL /tttps://ariiv org/ abs/ 200900000000000000 06732.
TogetherAI. Redpajama: an open dataset for training large language models, 2023. URL https: jama - Data.
//gi //github. wwt./o/oo/ooooo http://teeeercooootteeeeeeeeeeee
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alvaro Bartolome, Alexander M. Rush, and Thomas Wolf. The Alignment Handbook, 2023. URL
/ettps:/githin cow hww.//ggggggggge/eeceggogggg / al alignment- handbook.
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn.
Will we run out of data? limits of IIm scaling based on human-generatee data, 2024.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski. Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nel- son, Eric Jones, Robert Kern, Eric Larson, C J Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis I Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi: 10.1038/s4 1 1592-019-066--2.
Alex Wang, Richard Yuanzhe Pang. Angelica Chen, Jason Phang, and Samuel R. Bowman. SQuAL- ITY: Building a documeett summarization dataset the hard way. In Yoav Goldberg, Zornissa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference
Empirical Methods in Natural Language Processing, pp. 1139-1156, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v 11116653/112222.mmll h/2022.eemllpmmainn. https: /atps////caaaaat http://aaaaanooooog.oooooo org/ emnlp-ma . 75.
URL
Xuezhi Wang. Jason Wei, Dale Schuurmans. Quoe V Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023a. URL. https:/ http/pppeeeeeeeeeeee/eeeeeeeeeee net/ forum? 1
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Anna Rogers. Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 13484- 13508, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: 10.18653/ 1l/2023.acl-long./ URL https: : / hclanthooogy org/ 2023. acl-long. 754.
Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen. and Ryan Cotterell (eds.). Proceedings of the BabyLM Challenge at the 27th Conference Computational Natural Lan- guage Learning, Singapore, December 2023. Association for Computational Linguistics. URL wwwphhhlly...ogggo/oeeoe//e// www.//22/g.......oo con -bbbbbblm.......... 0.
https:: /pss:///aclattthte logy
21

---

Published as a conference paper at ICLR 2025
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 978171387 1088.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, 2020. doi: 10.1 D011///CPP2220000200000000
I. Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi- supervised learning for image classification, 2019. URL 00546.
www.xiiii...oooooooooooooo org/ abs/1905.
Zitong Yang, MICHAL LUKASIK, Vaishnavh Nagarajan, Zonglin Li, Ankit Rawat, Manzil Za- heer, Aditya K Menon, and Sanjiv Kumar. Resmem: Learn what you can and memorize the rest. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 60768-60790. Curran Associates, Inc., 2023. URL https: htt.p:::/ppocceednggs.... wwwp:/innnnnneuuuiiee neurips cc cc pc/pappp..... files/ https//pppe////222 file/br htte/bf8857cc cb 9a4.0o55556411ooo (0005/0c99a111772699002222222220 ottp::/:0000000000000000e-aeeeee Paw.roo-PPPpe---nfeeeeeee fdf..
Dong Yuan, Eti Rastogi, Gautam Naik, Sree Prasanna Rajagopal, Sagar Goyal, Fen Zhao, Bharath Chintagunta, and Jeff Ward. A continued pretrained Ilm approach for automatic medical note generation, 2024a. URL https://axxxvvvooooee abs/ /203 09057.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024b. URL https://arxiv..rrr
Abs/2441.10020
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3995, PMLR, 2017.
Dan Zhang, Sining Zhoubian, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets*; Lim self- training via process reward guided tree search, 2024a. URL httpp:://att http://axxvvvvvo/ org/ abs/
Tianyi Zhang*, Varsha Kishore*, Felix Wu* Kilian Q. Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https: /ttps////////eeeeee/ee net/ www.fooid---eeeueceeee
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39-57, 2024b. doi: 10.1162/aal URL https:/ https////claatthoooggeo org/ 2024 tacl-
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang. Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fsdp: Expe- riences on scaling fully sharded data parallel. Proc. VLDB Endow, 16(12):384833860 aug 2023. ISSN 2150-8097 doi: 10.14778/3611540.6111 3611540. 3611569.
URL https: /ttp:////oooooooooooo/ wttp::.......................... 14778/
Zexuan Zhong, Zhengxuan Wu, Christopher Manning. Christopher Potts, and Danqi Chen. MQUAKE: Assessing knowledge editing in language models via multi-hop questions. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empiri- Singapore, December 2023. Association for Computational Linguistics. doi: 1..18653//1/0092 1200:///02.2...mmmmmm..... https::/acet //ts:///cclle
cal Methods in Natural Language Processing, pp. 15686-15700
URL
www.y..orr/ooo www.or///00. 2023.2023..o emn emneemnnmeemmm 1-maalmmaii in 971.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories transformer models, 2020.
22

---

Published as a conference paper at ICLR 2025
CODEBASE, DATASET, AND MODEL WEIGHTS
We provide the codebase for reproducing all results discussed in the paper below:
We release the 455M EntiGraph corpus below:
We release the EntiGrraph CPT model weights below:
http:////hggghiireeeee/////ogggg contigraph-gup qualituullyy
A
ABLATION STUDIES
We present ablation experiments to further validate EntiGraph's effectiveness and test its general- ization properties. We discussed two potential limitations in $7.1:
1. Could the gains of Synthetic CPT be explained by distillation effects, due to the use of strong prompted LM for synthetic data generation?
2. Is the data synthesized in Synthetic CPT factual?
We provide evidence suggesting these are not significant concerns in Appendix A. and Appendix A.2, respectively. Lastly, repeat the procedure of the core experiments another small corpus of Coursera lecture transcripts, to provide evidence that Synthetic CPT generalizes to datasets and domains beyond QuALITY (Appendix A.3).
A.1 USING A WEAKER SYNTHETIC DATA GENERATION LM
One potential concern is whether EntiGraph's ' success demonstrated in $4 stems from distilling knowledge from GPT-4. To investigate this, we conducted an experiment replacing GPT-4-Turbo with a significantly weaker model, Llama 3.1 8B Instruct, as the synthetic data generator. Recall that all continued pretraining experiments, we finetune the 8B parameter Llama Base model. Therefore, this experiment, the capabilities of the synthetic data generator and the continually pretrained model are very similar, controlling for distillation effects. Using the entity extraction and relation analysis prompts introduced $2, we generate 334M synthetic tokens and evaluate the scaling behavior under the same hyperparameter setup detailed in $..1.
Figure reveals two key insights. First, even with the weaker generator, EntiGraph maintains steady log-linear improvement with signs of saturation at 334M tokens, suggesting that the gains of Syn- thetic CPT stem from continued pretraining on diverse representations the corpora's underlying knowledge, rather than distilling the generator model's 's knowledge. Similar to main results ($4), EntiGraph with a Llama 3.1 8B Instruct generator consistently outperforms Rephrase with the same generator. Moreover, at 334M synthetic tokens, EntiGraph with a Llama 3.1 8B Instruct generator outperforms cloked-book evaluation of GPT-4-Turbo.
Second, while switching from the GPT-4-Turbo generator to the weaker generator shifts the accuracy curve downward, the log- linear slope remains consistent. In contrast, holding the synthetic generator constant, observe that EntiGraph CPT and Rephrase CPT exhibit different slopes.
A.2 FACTUALITY AND LEXICAL DIVERSITY OF ENTIGRAPH SYNTHETIC CORPUS
Factuality. A limitation discussed in $7.1, and inherent in all methods involving synthetic data generation, is that the generation model may hallucinate. EntiGraph is a synthetic data augmenta- tion, which conditions LM given corpus document and prompts the LM discuss the docu-
23

---

estimate of the percentage of Published as a conference paper at ICLR 2025
57.5
GPT44 (51.30%)
QUALITY QA Accuracy
EntiGraph with Liama 8B Instruct
Rephrase
52.5
EntiGraph with EPT-4-Turbo
47.5
425
10a
10a
Number of synthetic tokens (in Millions)
Figure 5: The scaling properties of Synthetic CPT with the EntiGraph and Rephrase augmentations, comparing two synthetic data generators: GPT-4-Turbo and Llama 3.1 8B Instruct.
ment's entities and their relationships. Assuming reasonably good generator model, this grounding should decrease hallucination rate.
To quantitatively test the factuality of documents synthesized with EntiGraph, split the 455M token EntiGraph corpus into sentences and randomly sample 150 sentences. We ask authors this work to label whether each sentence subjective not, and among non-subjective sentences, to determine whether it is supported by the article text or not.
We compute statistics: the proportion of subjective sentences denotes the number of subjective sentences over the total number of annotated sentences. The factuality rate denotes the number of non-subjective sentences which are supported by the source document, over the number of non- subjective sentences, following Min et al. (2023):
Proportion subjective: 0.532 (bootstrap 0.95 confidence interval: [0.455, 0.610)).
Factuality rate: 0.944 (bootstrap 0.95 confidence interval: [0.889, 0.986))
Because EntiGraph uses open-ended prompts which ask the LM to relate different, often abstract en- tities, the LM often generates subjective statements. We do not necessarily view this limitation, because learning reasonable subjective interpretations is crucial for understanding (and hence is of- ten assessed in, e.g., essay questions on literature exams). We also observe that the non-subjective sentences are consistently factual, supporting the effectiveness of grounding in reducing hallucina- tion.
Lexical Diversity. We hypothesize that good synthetic data augmentations should produce knowl- edge representations with diverse wording. As a measure of this lexical diversity, we compute the percentage of n-grams in the synthetic documents that overlap with the n-grams of the correspond- ing source documents.
More precisely, we first randomly select 100 QuALITY articles, tokenize them with the Llama 3.1 tokenizer, and compute the set of n-grams for each article. Then, for each article, we tokenize the corresponding EntiGraph and Rephrase synthetic data, compute n-grams, and count the n-grams in the synthetic data that appear in the set of n-grams for the raw article. For each n and synthetic augmentation method, we sum this overlap count across articles and normalize by the total number of synthetic tokens generated for the 100 articles, providing n-grams in the synthetic data that overlap with the source data.
These results are provided in Table 4. We observe that for both augmentations, n-gram overlap per- centage is low and quickly approaches 0% with increasing n, indicating that both methods produce lexically diverse knowledge representations.
24

---

Published as a conference paper at ICLR 2025
Augmentation n
n
4
n
EntiGraph
23.40
3.66
0.24
0.00
Rephrase 21.35
3.04
0.51
0.22
Table 4: Percentage of token r-grams in synthetic documents that overlap with the source document n-grams, for the EntiGraph and Rephrase synthetic data augmentations.
A.3 DATASETS BEYOND QUALITY
To test whether synthetic CPT with EntiGraph generalizes to corpora beyond QuALITY, evalu- ated on the Coursera Exam QA dataset (An et al., 2023). This dataset contains lecture transcripts and exam questions from advanced technical courses like data science and machine learning. Compared to the books and stories in QuALITY, Coursera exams present new challenges- have multiple correct answers, and the number of options is not fixed to four choices. This makes few-shot prompting more demanding, as the model must understand both the content and the flexible answering format.
content is harder
conceptually, questions
The dataset consists of 15 lecture transcripts and 124K raw tokens, substantially smaller than QuAL- ITY's 265 documents and 1.3M raw tokens. During our scaling analysis, found that models trained on tiny synthetic corpora (e.g., a few million tokens) struggled to follow few-shot prompts reliably for Coursera questions, resulting in parsing errors. Therefore, we begin the scaling curve in Fig. 6 starting from token counts where parsing error rates fall below 5%. For the Rephrase baseline, we generate synthetic data up to 22M tokens, and find that only one model has parsing error rates below 5%.
Rephrase with GPT-4- lurbo
Coursera Exam QA Accuracy
EntiGraph with GPT-4-Turbo
LLama 38 Base (48.26%)
Number of synthetic tokens (in Millions)
Figure 6: The scaling properties of Synthetic CPT using the EntiGraph augmentation on the Cours- era Exam QA dataset.
Despite these challenges, EntiGraph CPT shows consistent improvement over Llama 3 8B Base, im- proving accuracy from 48.26% to 53.87%, better than Llama 3 8B Base and the Rephrase baseline. The log-linear scaling pattern persists up to 32M synthetic tokens, suggesting EntiGraph's 's effec- tiveness extends beyond narrative texts to technical educational content. This successful transfer to substantially different domain provides evidence the generalizability of synthetic continued pretraining and EntiGraph.
25

---

Published as a conference paper at ICLR 2025
B ADDITIONAL RELATED WORK
Synthetic data generation. There is a rich literature on using neural nets to generate synthetic data. Many such approaches were originally developed for semi-supervised serning--Slf-tranning and pseudo-labeling methods improve models by iteratively training them on their own predictions (Scudder, 1965; Lee, 2013; Yalniz al., 2019; Berthelot al., 2019; Xie al., 2020), and co- training uses two models to supervise each other (Blum & Mitchell, 1998; Balcan et al., 2004). Before language models rose to prominence, few approaches attempted to synthesize inputs. One exception is membership query synthesis, which explored the synthesis of inputs in supervised learning context (Angluin, 1988; Schumann & Rehbein, 2019).
Contemporary works employ co-training (Lang et al., 2022) and self-training to improve language model performance, often on mathematical reasoning tasks (Huang et al., 2023; Gulcehre et al., 2023; Zhang et al., 2024a), or synthesize input-output pairs for instruction tuning, usually by con- ditioning on a curated seed set (Wang et al., 2023b; Honovich et al., 2023; Taori et al., 2023; Peng al., 2023; Yuan et al., 2024b; Li et al., 2024).
Continual learning and pretraining. Continual learning is rooted in historical work on connec- tionist networks (McCloskey & Cohen, 1989; Ratcliff, 1990) and considers learning with tasks riving in an online manner (Schlimmer & Fisher, 1986; Grossberg, 2012). The main focus is on mitigating neural net's "catastrophic forgetting" of previously encountered tasks (Robins, 1995; Goodfellow et al., 2015; Kemker et al., 2018). Approaches include regularizing parameter updates to preserve important parameters (Nguyen et al., 2017; Zenke al., 2017; Kirkpatrick al., 2017);
dynamically modifying the architecture (Rusu al., 2016; Golkar al., 2019); and recalling or replaying previous experiences (Rebuffi et al., 2017; Shin et al., 2017: Lopez- Paz & Ranzato, 2017).
Modern works in continued pretraining (ef. $1.1) effectively mitigate catastrophic forgetting by scaling parameter count (Ramasesh et al., 2022) and mixing in updates on pretraining data (Ouyang et al., 2022).
C DETAILS ON THE QUALITY DATASET
We provide additional details the QuALITY dataset below. For each book, execute entity extraction (Step 1, $2.2) and then analyze all pair-wise relations between entities and a subset of all triplet relations (Step 2, 2.2). We provide summary statisties for the Raw and EntiGraph corpora in Figure 7.
Frequency
Prequency
Taken count (K)
Entity count
Taken count (K)
(a) Raw article tokens
(b) Extracted entities
(c) EntiGraph corpus tokens
Figure 7: Histograms the 265 QuALITY articles and books. (a) The token count of raw articles. (b) The number of extracted entities. (c) The token count of EntiGraph synthetic data (generated for each book).
D TRAINING DETAILS FOR THE MAIN EXPERIMENTS
Continued pretraining details. all experiments, we continue pretraining the Llama 3 8B Base model with context length of 2048 and batch size of 16. We apply linear learning rate warmup for 5% of total steps, followed by a cosine decay with peak learning rate 5e-6. We perform full parameter training with Fully Sharded Data Parallelism (FSDP, Zhao et al. (2023)).
26

---

Published as a conference paper at ICLR 2025
EntiGraph continued pretraining details. To mitigate the forgetting of pretrained knowledge, we perform replay with a rate of 0.1 using 1B RedPajama tokens (TogetherAI, 2023). More pre- cisely, for each training batch, flip biased coin such that with 10% probability, we load the RedPajama data instead of the EntiGraph synthetic data.
Raw continued pretraining details. Next, we provide details for our continued pretraining di- rectly on the Raw corpus, producing the "Raw CPT" model. Because the Raw corpus only has 1.3M tokens, we jointly tune the number of epochs (repetition factor) and the RedPajama replay rate on accuracy over a QuALITY QA validation split. The selected hyperparameter configuration uses epochs and a 0.1 replay rate.
Instruction tuning details. We use the UltraChat instruction tuning dataset (Ding et al., 2023) filtered by the Huggingface team (Tunstall et al., 2023) as our instruction tuning data. We use the chat template of Llama 3.1 8B Instruct (Dubey et al., 2024) to format the UltraChat conversations, obtaining a 250M token instruction tuning dataset. We apply linear learning rate warmup followed by a cosine decay to 0 with peak learning rate 5e-6, and train the model for epoch with a batch size of 512 and context window of 2048. To sanity check our instruction tuning procedure, we measure the AlpacaEvval (Li et al., 2023a) winrate against GPT-4 and find it improves from 0% to 6.25%, comparable to a 7.7% baseline winrate Llama Chat 13B.
Compute resource. All the continued pretraining experiments are performed with one 8xH100 node. With PyTorch FSDP (Zhao et al., 2023), obtain throughput of 6090 tokens per second. Since all experiments use the same model architecture, batch size, and context length, the time to run the experiments can be calculated based on the total tokens seen during training. For example, the main EntiGraph is trained on 455M tokens with 2 epochs. Therefore, it should take 455M 2/6090 seconds, which is about 41 hours.
E TASK-SPECIFU FINETUNING FOR THE QUALITY QUESTION SET
Our work considers task-agnostic synthetic data generation and continued pretraining as a way to obtain generalizable knowledge about domain, in way that later be extracted via few-shot prompting (Brown et al., 2020) and instruction tuning (Ouyang et al., 2022).
However, if our goal is only to do well on single task, such as question answering, then we could fine-tune language model for that particular task. This approach worked extremely well on tasks such as SQuAD (Rajpurkar al., 2016) in-domain but suffered from degraded performance outside the fine-tuning data distribution (Awadalla et al., 2022).
We do not extensively perform comparisons to task-specific finetuning due to the more general multi- task goals of EntiGraph. We run preliminary experiments comparing a simple QA SFT baseline EntiGraph, and find that EntiGraph scaling and synthetic data generation costs are generally favorable even when compared to this strong, task-specific baseline.
QA SFT. We follow the same set as in $2.1 and $3 except that we do not prompt LM to generate general knowledge about QuALTYY articles. Instead, we prompt LMaynth generate QA pairs directly:
You are an assistant to help read article and then rephrase it in question answering format. The user will provide you with an article with title, year, content. You need to generate a paraphrase of the same article in question and answer format with muitiple tags of "Question: ..." followed by "Answer: ..." Remember to keep the meaning and every content of the article intact, including the title, year, etc.
We repeat this prompt many times at temperature 1.0, resulting in 28M tokens synthetic question answer pairs. We perform the same continued pretraining procedure in $4.1 1 on Llama 3 8B and refer to this model as "QA SFT".
27

---

Published as a conference paper at ICLR 2025
EntiGraph CPT Rephrase
55.0
SFT
QA Accuracy
50.0
47.5
45.0
42.5
10a
10'
10a
Number of synthetic tokens (in Millions)
Figure 8: Accuracy the QuALITY question set Quest (y-axis) function of the synthetic token count (I-axis). Comparison among EntiGraph CPT, Rephrase CPT, and QA SFT.
Results discussion We plot the QA SFT scaling curve in Figure 8. We can see that task-specifie finetuning demonstrates very sharp improvement in QA accuracy, consistent with prior results showing task-specific finetuning gains for pretrained models. While QA SFT performance is high, we note that EntiGraph attains similar performance despite being entirely task-agnostic, and the overall dollar cost of creating the dataset much lower for EntiGraph.
This difference synthetic data generation cost is hidden in Figure 8, as we plot the number of training tokens rather than dollars spent to generate the synthetic data. For QA SFT, each QA question is generally short, resulting in large inefficiencies in generating this QA dataset. We found that the input token output token ratio was large compared with Rephrase CPT and EntiGraph CPT, resulting in over $5K to generate just 28M tokens*. This difference in cost means that further scaling became prohibitively expensive, and that EntiGraph's 's performance in Figure 8 even better than it appears, if we match for total cost rather than token budget.
F ADDITIONAL DETAILS ON OPEN-BOOK EXPERIMENTS
We provide additional details on our open-book experimental setup below, including our retrieval- augmented generation (RAG, Lewis et al. (2020); Gao et al. (2024)) pipeline. As mentioned in $5, we use a standard two-stage RAG pipeline: first, an offline stage which indexes document chunks; second, inference-time retrieval, reranking, and placement of those chunks in few-shot LM prompt.
F.1 STAGE 1: OFFLINE INDEXING
The purpose of the indexing stage to construct an index over all the 265 articles and books from the QuALITY corpus D souree- More specifically, this stage chunks documents from the given corpus, obtains dense vector embeddings for each chunk using an API-based embedding model, and indexes the (embedding, chunk) pairs.
We first split each document ( (D D source into C (i.,, Character eaaracteTeexte peecterTextpppl.. Spplitte iiteer from Chase (2022), which attempts to keep all paragraphs (and then sentences, and then words) together for as long possible, in order to preserve the semantics within each chunk. We use non-overlapping chunks and tune chunk size in characters (chunk size, hyperparameter values provided below). Lastly, because we have access to metadata about each document namely the title, author, and year of the book or article-we prepend this meta- data to each document chunk. This is analogous to how corporation building RAG system over
Chunking documents.
of m; document chunks {C{") ....,
To perform this splitting, we use the Recurs sive
OpenAl API pricing, Sep 2024
28

---

Published as a conference paper at ICLR 2025
their own document store could include metadata about the document (title, author, year, etc.). These final chunks with metadata prepended are embedded, and are the ones that are retrieved and placed in-context.
Embedding and indexing document chunks. Next, we obtain dense embeddings for all document chunks using state-of-the-ee text embedding model OpenAl text-embedding -3-large (Neelakantan et al., 2022). Lastly, we index all (embedding, chunk) tuples using FAISS vector store (Douze et al., 2024).
F.2 STAGE 2: INFERENCE- RETRIEVAL AND RERANKING
At inference time, the RAG system receives test query Quat Each query q is contextualized with the article title and author name, as described in $3, and contains its four possible answer choices (QuALITY is 4-choice, multiple choice dataset). In Stage 2, we embed the query with the API-based embedding model, retrieve K' document chunks using an approximate nearest-neighhor search, and lastly, select the k < most relevant chunks using an API-based reranker.
Retrieving top-K document chunks. We embed with mext--embedding--LLeee retrieve the top- K most relevant document chunks from our indexed vector store using FAISS simi- larity search with Euclidean distance metric.
and
Reranking to obtain top-k (k < K) chunks. Next, we use a reranker to filter the K retrieved document chunks smaller number of reranked chunks k. Rerankers are known significantly improve recall (the proportion of the time that the salient article is contained in the top chunks), and indeed, the recall of our RAG pipelines is near-perfect (Table 3 in $5). Specifically, we pass the query q and the list of K retrieved document chunks to a state-of-the-aae reranker- Cohhre rer verann-eet serank-englisse english-up most to least semantically relevant for the query. We take the i highest scoring chunks and place them in our few-shot prompt.
(Cohere, 2024)-whil returns a list of the K chunks in order from
Few-shot prompt formatting, Our full few-shot chain-of Sfain-of-thoughi thought evaluation prompts for the open-book setting will be provided in our code release. Similar to the closed-book QA evaluation prompt, we manually write and fact-check in-context learning examples about well-known books, to avoid leaking knowledge from the QuALITY articles. In early experiments, we found that placing the retrieved contexts first, followed by the question and answer choices after, significantly improved performance compared to question-then-contex, iments. We treat as hyperparameter whether the reranked chunks are ordered from the best match to worst fbest_first evaluation, we follow the sampling procedure used in the closed-book experiments (Appendix I.1). Specifically, we generate 64 responses for each question, and filter out responses that do not parse to one the four choices. Lastly, we randomly select one of the valid responses as the model's final answer.
we use this format throughout the retrieval exper-
or from the worst match to best (best_lase sasst...). When performing few-shot
F.3 HYPERPARAMETER TUNING
In our experiments, we compare two LMs used in the RAG pipeline above: EntiGraph CPT and its base model, Llama 8B Base. As mentioned above, we fix the retrieved number of chunks to K 128, but vary the number of reranked chunks A which are ultimately placed in the context window. For each language model RAG pipeline, we independently tune the following hyperparameters with grid search on accuracy using a QuALITY QA validation split:
Document chunk E (256, 512, 1024]
Rerank top-k E [1,2,4,8,11)
Order of chunks E fiest.first.be best best.
Eval temperature E 0..1,0.3.055,00
We will provide tuned hyperparameters in our code release.
29

---

Published as a conference paper at ICLR 2025
G PROOF OF THEOREM AND OTHER ANALYTICAL FORMULAS
In this section, we prove Theorem I and provide the derivations for several other approximation formulas.
Proof of Theorem 1. Fix the matrix Mo. we observe that
Acc(Mi) |E[| M. Moo
(( E D)) V(V
P((i,)) E M. V(V-1)
E (1,))
E
V(V 1)
(1))
For each (1,5) E v we define Çi to be the probability that (i,j) is included the set {(2,, z)), (Ie, 2??..... (z,, =1 (z,, v,)}. Note that each iteration of the procedure generates kt,
path (Iz, 2 ......., 2t 91) independently identically. So naturally 4. does not depend the time t. This implics that j) E Dr = = (1 9.,)". Thus we can further rewrite the link density
ID soouree
P((i.)) E D. V(V
Acc(
V(V 1)
(iJ))VA Dannce
(Dwource
1- 1111 q2))
V(V 1)
V(V-1)
(IJ)VV* Donne
The remaining task is to estimate qi.j. We say a vertex j is reachable from and denote ~ j. there is directed path from to j in Mo- We define R {(u, v) V2 u & v, v} be the set of all reachable pairs of vertices in V. We note that Qi.j is non-zero if and only if j is reachable from i Mo. Now, for any 2 1, the function (1 x² concave, thus by Jensen's inequality, we have
Σ
--11-15)+
Σ (Nd)R
1 (1 q:)) s | (1 (1 );.)))
(LJ)EVA
where
R5 = EOWDER Qi,j
[R]
For each (i,5) E R, the probability yi satisfies
96 Eapbera (((, {(a, (a, (a, b)})
V(V 1)
where (a,z1, 2/, 21, .... 2t b) is the shortest path in Mo connecting a and If there is no such path, then by default the indicator equals zero. Now we look at
Σ U.
1
Σ (iJ)R (a.))ER
Σ
I((i,)
(a, (e.)),)..,,,,)))))))))))
V(V 1)
(.)ER
1
Σ
Σ
(((i.) {la (l.zz),, ((..)))...))....(()).....)
V(V 1)
(a,))R $$EV
1
Σ lab
V(V
(o.0)
where la.b is the length of the shortest path connecting to b. To analyze the typical shortest length of paths, we present few classical results on directed Erdös-Rényi graphs. For any a V. let X (a) denote the set of vertices reachable from a and let Y (a) denote the set of vertices from which is reachable. Recall that p(A) is the extinction probability for the Poisson(A) () branching process.
Lemma G.1 (Lemma and Corollary in Karp (1990)). For each vertex with probability tending to 1 as V tends to infinity, there exists a constant B such that either [x(a) < B log V or [x = (1-p(1))V + Moreover, the probability that the latter happens tends to - p(A) V tends infinity. The same is true for (aa).
30

---

Published as a conference paper at ICLR 2025
For each vertex a, the set (() is said to be small if [X(a)] Glog V (in such case write E Sx) and large if [((a) (1 ((A)) + @(vV) (we write E Lx). We define Sy and Ly similarly.
Lemma G.2 (Theorem in Karp (1990) and Theorem 2.4.1 in Durrett (2010)). With probability lending to 1, the following statement holds for all a and in V: if ((a) is large and Y (b) is large, then is reachable from a. Moreover, if X (a) is large and Y (bb) is large, for any and any sufficiently small 8 > O,
Plla,b (1 + E) log V log A| < exp(-V"6).
With Lemma G.1 and Lemma G.2, we can now give useful estimates of [R]. In particular, for any
E > 0,
R [(a,)) :: br) E a Sx or Sy
$ (1 - (1)) + V
< <1 - (1
with high probability. Similarly, for the lower bound,
| = (((,))) E R E Lx,be +y)l+ H(a,b) b) E R:ac Sx or be E Sy
2 (1 (1))
2 (1 - -VV 1),
with high probability. By union bound over all pairs of (a, 6) E R, we also have that
Σ (UER
Σ
lab
V(V 1)
(R,
Σ
tab +
Σ
Labb
V(V 1)
V(V - 1)
(a,b)ER dEEx,bELy
(a
aESx BESy
log V
1
$ - (()) 2/22
e/2)
+
2(1 + V log V)2
log A
V(V - 1)
log V
S - (A))
log A
with probability larger than 1 - 2 exp(-VEE) Combining the above, for any 0,
Σ E(i 9h.t
(1 E) log V
T1,5
R
V(V - 1) log X
with high probability. Therefore, for any 8 >
Daoureel
R (1 (1 (
Acc(M) <
V(V 1)
V(V -1)
p+ (1 p())2 1 (1-(1-11+-)) (1++)) log V
(1
< (1++)
V(V 1) log
with high probability, which completes the proof of the upper bound. For the lower bound, we observe that if ~ j and (i,j) RR.Daources then Zi 2 1/V(V 1), because when and are chosen in the procedure, the edge (i.5) will be added. This implies that
Acc(M)) = |Dsource|
1- (1-9h,)²
Σ
V(V
V(V -1)
RV RiDaams
DDsoucee
1 (1-(1- V(V (-1-VV--D)
+
V(V 1)
V(V 1)
1)
1 (1-1- v(V --vVv-1)
2 (1 - E) p+
with high probability which completes the proof of the lower bound.
31

---

Published as a conference paper at ICLR 2025
To obtain more precise description of Acc(M)), we employ Poisson branching process to ap- proximate the cluster growth of vertices, which we now define. A Poisson()) (A) branching process is a model for population evolving in time, where each individual independently gives birth to a num- ber of children with Poisson()) A) distribution. We denote by Zn, the number of individuals in the n-th generation, where by default Zo 1. Then Zn satisfies the recursion relation Z, - Σier" Xn.i- hn,siis a doubly infinite array of i..d. Poisson(A) random variables. The total progeny Y, is then defined Yn Σ Zn Zn is often called Salton-Watsop branching process and the associated tree is called a Walton-Watson tree.
where {xn.snn.s2is
As in the previous proof, accurate estimate of Acc(M) relies understanding l.j- the proba- bility that the edge j) will be added in each round. As before, the only edges that will be added are those connected to the giant component (i.e., Lx and j Ly). The proportion of such edges converges to C2 as V + co. Recall that
9i, EaaMER A((i.)) {(a, (a, (a, (a, V(V-1)
(3)
b) represents the shortest path in Mo connecting a and b. Equivalently, consider the tree generated by breadth-first search in M, rooted at then since ~ j.j will be in the tree, and the numerator counts the total number of offspring of j in the tree, including itself. This is the point at which rigorous mathematical characterization of the tree becomes challenging. Instead, we approximate the tree and analyze its behavior. It is well-known that when = X/V. the cluster growth (or the breadth-fist first search at a vertex) can be approximated by a Poisson(A) branching process (see e.g., Hofstad (2016); Durrett (2010)). For fixed vertex i, we define T as with Poisson(A) offspring distribution with depth L.. We use to approximate the exploration process at i. For 0 S E S T., the number of vertices at level T. l is approximately X Given that the total number of vertices T is approximately (1- V. the number of vertices at level L- is also number of its offspring (including itself) equals &o with probability (k)). In this case, the numerator in (3) equals k. Combining the above, there around (1 (1-p1 -Pe (c())((-(A)))V(AA (A)))(1 vertex pairs (i, in the graph such that E Lx.3 j E Lv, qi.j = kV - 1) and j is located at the arrive an approximation of the form
where 20,21,21 21,
-atson tree rooted
For each vertex at level L E, the
L level in the tree T. Ultimately,
(1 (1-
(1
(I-vVー-))
A2)( ~ C2
pe(k))
(I-VVV-T)
E=U
Beyond Frdōs-Rényíí graphs, the term gi,t may not be as explicit. We can define C' as the proportion of vertex pairs (i,j) such that i ~ in Mo, then q.j nonzero for CV(V-1) pairs of vertices. In this case, if we write dg= = k/V(V - 1) and define (kk) as the probability that (..j = ak, then we can have a general formula
Mc) ~ p x (1-2
((k) (1() - anl
1=1
The drawback of this formula is the lack of explicit expressions. For given Mo. it is unclear how to compute the measure (u(-) easily.
Next, we provide qualitative description of the shape of such mixture of exponentials.
Lemma G.3. For a fixed constant < C < 1 and a probability measure pu(-) on Z4 with finite mean m. we define
20 1- ((k)) (1-V/V- k (V(V-1)
(0)=p+C
V(V -1)
k=1
Then we have that there exists 0 < 41 < to such that
9 (p++)),
for < 5 hn, for < t < t2,
f(1)
Ologt),
O(1),
for 1 2 l
as V co.
32

---

Published as a conference paper at ICLR 2025
Proof of Lemma G.3. Fix any 1 ti tz. Note that f(t) is monotone increasing, concave and always bounded by 1. We also have
1(12) 2 C (1- V(V-1) -1)
(V((-11)
2 pp+ C(1 exp(-12)) 01))
So f(t) (1) when 2 12. Now when & tt
0
f Σ ((k) tk) S Cmn..
Since f(0) p and f(t) 2 p C(1 exp(-tz)), by concavity, f(t) is lower bounded by p + IC(1 exp(-12)/112 = Opp +1) for any 0 S S 41 Finally for l < S l2, we note
that f((1) < [(i) so easily, f(1) I loght log 4 S l log l = 0(og 1). Similarly,
J(() 2 [(41) log / log l 2 log (J((1)/ log (2) 2 Sillog l). Therefore, [(() = O(log () for any
t St.
G.1 MORE DETAILS ON THE MIXTURE OF EXPONENTIAL SHAPE
We provide more discussion the mixture of exponential shape, including how we use it to fit the empirical EntiGraph CPT QA accuracy.
Sketch of derivation. Intuitively, the edge (i.j) will eventually be added if and only if reach- able from i in the original graph My. This explains the limiting behavior of Acc(M,) as ap- proaches infinity: the proportion of links will converge to the proportion of connected vertex pairs in Mn- To understand the vixture-of-eeppoentta functional form, consider that at the time 4, the probability of adding each vertex pair follows an exponential pattern, with different vertex pairs exhibiting different exponential growth rates. Specifically, think of breadth-first search in Mo starting from a vertex i. If j is very close the root, there are many paths from i to other vertices passing through j, making it more likely that (i, will be included in each iteration. In contrast, if j is far from the root (e.g., at the end the exploration process), there are fewer such paths, making it less likely for (i,j) to be included in each iteration. This accounts for the mixture-or coatturr-of-eeppooe exponentntt shape, where the mixture primarily reflects the distance of each vertex from the root, the number of such vertices, and their corresponding exponential growth rates.
0.5
Accurary
0.4
Accuracy
Accuracy
0.6
0.3
0.2
0.70
0.65
0.0
2000 4000 6000 8000 10000 12000
2x 3x 10
40000
60000
80000
300000
Number of synthetic takens
Number of synthetic takens
Number of synthetic takens
(a) Linear regime
(b) Log-linear (f in log scale)
(c) Plateau regime
Figure 9: Accuracy (M) with respect to time t, for V = 100 and p 0.03. The mixture-of- exponential functional form (2) leads three distinct regimes.
Qualitative description. Finally, to help build intuitive understanding, we provide qualitative description of the mixture-of=exponeniaal shape. We demonstrate in Appendix G that this mixture- of-exponential shape comprises three distinct phases: fast growth phase, slower growth phase, and plateau phase. Mathematically, we show the existence of two distinct times, < t < f2. such that
(p + 1),
for S 1 $ f1, for S t $ t2 for t 2 l2,
Acc(My) = Ologt), (11)
33

---

Published as a conference paper at ICLR 2025
where convenient change of variable T +V(V -).. It is important note that the choice of logt f in the second phase is not necessarily canonical. In fact, the bound holds for any well-behaved monotone increasing concave function replacement for logt. t. Our representation here is motivated by two factors: first, aligns with the performance observed in our EntiGraph CPT numerical results, and second, it reflects the gradual slowdown in growth. We illustrate the three phases in Figure 9, which present a simulation of the toy model with 0.03.
To perform curve fitting using the mixture-of-exponentaal formula, we approximate the infinite sum with three terms in
Acc(M. ~p+C (1 an)))
11 (1
((())
Mathematically, we fit the empirical observation against the formula
y(x) b - b2r - byr
where x is the EntiGraph token count (in millions) and y(x) is the QuALITY QA accuracy. We the non-linear least squares method implemented by Virtanen et al. (2020). As result of this procedure, we obtain the fitted formula
y( = 64.5456 13.8352 x (0.9989)* 8.4705 x (0.8961)* 3.932 x (0.0546)*.
For the implementation of this procedure, refer readers code release.
34

---

Published as a conference paper at ICLR 2025
H SYNTHETIC DATA GENERATION PROMPTS
We generate two synthetic corpora in this paper: EntiGraph (Appendix H.1) and the Rephrase base- line (Appendix H.2). In our experiments, the Dsooree is collection of documents D, and our syn- thetic augmentation procedure is applied to each document D E Douree. We will focus on a single document for the remainder of this section.
H.1 ENTIGRAPH PROMPTS
The EntiGraph procedure is described in detail in $2.2. We will recap the three steps below.
Step I: Entity extraction. The first step is to extract the salient entities from the document D using the entity-extracta operation (Step $2.2). The complete antittyexttrattin prompt is as follows:
As knowl edge analyzer, your task is to dissect and understand an article provided by the user. You are required to perform the following steps:
1. Summarize the Article: Provide concise summary of the entire article, capturing the main points and themes.
2. Extract Entities: Identify and list all significant "nouns" or entitics mentioned within the article. These entitics should includo but not limited to:
People: Any individuals menticned in the article, using the references provided.
Places: Both speci fic locations and abstract spaces relevant to the content.
Object: Any concrete object that is referenced by the provided content.
Concepts: Any significant abstract ideas or themes that are central to the article's discussion.
Try to exhaust many entities possible. Your response should be structured in JSON format organize the information effectively. Ensure that the summary is brief yet comprehensive, and the list of entities is detailed and accurate.
Here is the format you should for your response:
"summary": "<A concise summary of the article>", "entities"; ["entityl", "entity?", ...]
Relation analysis. The last step is to generate diverse descriptions of relations among more entities. In our experiments, for each document D, we enumerate all entity pairs and generate a description for each. The prompt for generating description relating pair of entities is as follows:
Step 2:
You will act as a knowledge analyzer tasked with dissecting an article provided by the user. Your role involves two main objectives:
1. Rephrasing Content: The user will identify two specific entities mentioned in the article. You are required to rephrase the content of the article twice:
Once, emphasizing the first entity.
Again, emphasizing the secend entity.
2. Analyzing Interactions: Discuss how the two specified entities interact within the context of the article.
35

---

Published as a conference paper at ICLR 2025
Your responses should provide clear segregation between the rephrased content and the interaction analysis. Ensure each section of the output include sufficient context, ideally referencing the
article's s title to maintain clarity about the discussion's focus. Here is the format you should follow for your response:
HH Discussicn of <title> in relation to <entityl> <Rephrased content focusing on the first entity>
### Discussien <title> relation <entity2> <Rephrased content focusing on the second entity>
### Discussien of Interaction between <entityl> and <entity2> in context of <title>
<Di <ciscusiion on how the two entities interact within article>
We also generate synthetic data involving three entities, using the prompt below:
You will act as a knowledge analyzer tasked with dissecting an article provided by the user. Your role involves three main objectives:
1. Rephrasing Content: The user will identify three specific
entities mentioned the article. You are required to rephrase the content of the article three times:
Once, emphasizing the first entity.
Again, emphasizing the secend entity.
Lastly, emphasizing the third entity.
2. Analyzing Interactions: Discuss how these three specified entities interact within the centext of the article.
Your responses should provide clear segregation between rephrased content and the interaction analysis. Ensure each section of the output include sufficient context, ideally referencing the article's title to maintain clarity about the discussion's focus.
Here is the format you should fcllow for your response:
### Discussicn of <title> in relation to <entityl> <Rephrased content focusing on the first entity>
### Di Discussion of <title> relation to <entity?> <Rehhaaedd content focusing on the second entity>
## Discussien of <title> in relation to <entity3> <Rephrased content focusing the third entity>
|| Discussicn of Interaction between <entityl>, <entity2> and <entity3> in context of <title> <Discussion en how the three entities interact within the article>
H.2 REPHRASE PROMPTS
For the rephrase corpus, we adapt the prompt from Maini et al. (2024) to our setting of books and articles. We provide four rephrase styles below:
Easy rephrase:
You are assistant help read article and then rephrase it in simpler terms. The user will provide you with an article with
36

---

Published as a conference paper at ICLR 2025
title, year, content. You need to generate paraparase the article using very small vocabulary and extremely simple sentences that toddler will understand. Remember to keep the meaning and every content of the article intact, including the title, year, etc.
Medium rephrase:
You are an assistant help read article and then rephrase it in different terms. The user will provide you with an article with title, year, content. You need to generate a paraphrase of the same article using diverse and high quality English language sentences Rikipedia. Remember keep the meaning and every content of the article intact, including the title, year, etc.
in
Hard rephrase:
You are an assistant help read article then rephrase it in more sophisticated terms. The user will provide you with an article with title, year, content. You need to generate paraphrase of the same article using very terse and abstruse language that only an erudite scholar will understand. Remember keep the meaning and every content the article intact, including the title, year, etc.
37

---

Published as a conference paper at ICLR 2025
I ADDITIONAL EVALUATION DETAILS OF MAIN EXPERIMENTS
1.1 QUALITY QA QUESTION SET
In this section, we provide more details of evaluation on the QuALITY QA test queries. Throughout the closed-book QA experiments, use fixed 5-shot prompt below:
# Example
FHF Question
In the context "Les Misérables", written by victor Hugo 1862, what is the main setting of the nevel? There is only one correct choice.
### Choices
A. London
B. Macrid
C. Paris
D. Rome
### Thought Process and Answer
Thought process: "les Misérables" primarily set in Paris, making с the correct choice. London, Madrid, and Rome are significant cities in other literary works but not in Victor Hugo's s "Les Misérables". There is only one correct choice. Answer: C.
## Example
### questicn
In the context of "Brave New World", written by Aldous Huxley in 1932, what substance widely used in the society control citizens happiness? There is only one correct choice.
### Choices
A. Gola
B. Soma
C. Silver
D. Iron
IH Thought Process and Answer
Thought process: In Aldous Huxley's "Brave New World, Soma is used maintain social centrol by ensuring citizens' happiness, making the correct cheice. Gold, Silver, and Iron are not the substances used for this purpose in the book. Answer: B.
as
# Example
### Question
in the context of "Romeo and Juliet"! written by William Shakespeare in the early 1590s, what are the names of the two feuding families? There is only correct choice. Choices:
A. Montague and Capulet
B. Bennet and Darcy
C. Linton and Earnshaw
D.
Bloom and Dedalus
### Thought Process and Answer
Thought process: In Milliam Shakespeare's "Romeo and Juliet, " the two feuding families are the Montagues and the Capulets, making the correct choice. The Bennets and Darcys in "Pride and Prejucice", the Lintons and Earnshaws "Muthering Heights", and Bloom and Dedalus in "Dlysses". Answer: A.
Example ### Question
38

---

Published as a conference paper at ICLR 2025
In the context of "1984", written by George Orwell in 1949, what is the name of the totalitarian leader? There is only one correct choice.
FFF Choices
A. Big Brother
B. o' BrBien
C. Winston Smith
D. Emmanuel Goldstein
### Thought Process and Answer
Thought process: In George Orwell's "1984," the totalitarian leader is known as Big Brother, making A the correct choice. OrBni is character in the novel, Minston Smith is the protagonist, and Emmanuel Goldstein is rebel Leader. Answer: A.
#I Example
### Question
In the context of "Moby-Dick", written by Herman Melville in 1851, what is the name of the ship's captain obsessed with hunting the titular whale? There only one correct choice. #FF Choices
A. Captain Hook
B. Captain Nemo
Captain Flint
D. Captain Ahab
### Thought Process and Answer
Thought process: In Herman Melville's "Moby-Dick," the ship's captain obsessed with hunting the whale is Captain Ahab, making in "Twenty Thousand Leagues
the correct choice. Captain Under the Sea", Captain Flint in "Treasure Island", and Captain Hook in "Peter Pan". Answer: D.
## Example
If the output of the model correctly follows the format of the few-shot prompt, its last characters
should be "A.". ", "B.", ", "c or "D.". However, the model sometimes cannot successfully follow the few-shot prompting format, particularly for the continually pretrained model. As result, in all our evaluations, we sample the response 64 times, and only select the ones that can be parsed in the correct format. Out of these 64 attempts, we randomly select among the valid answers to give the final answer. Note that this is different from majority voting in self-consistency prompting (Wang al., 2023a).
1.2 CLOSEE-BOOK SUMMARIZATION
Automated evaluation metric. We design a three-stage evaluation procedure: (i) the first stage, GPT-4 to break the summary into atomic claims, similar to Min et al. (2023); (ii) In the second stage, we provide both the list of claims and the source article to a judge model (also GPT-4). We ask the judge model to determine whether each claim is true false, based the source article. the claim true, we further ask the model to determine whether the claim is salient (contributes to the main message of the article) cosmetic (factual details that do not help understand the main message). (iii) Finally, for each summary, we obtain its number of false and salient claims and normalize it the corresponding count from the human summary. We report the average of these normalized metrics across the QuALITY corpus articles in Figure 3.
Prompts to generate summaries. For summarization evaluation with EntiGraph Instruct and Raw Instruct, we apply the following two prompts to obtain two summaries of increasing length. We provide three examples of summarization outputs below. For each of the three examples, we will
"Specifically, the gpt-4 4-tubb model of Aug. 19, 2024.
39

---

Published as a conference paper at ICLR 2025
Short prompt: Summariza the article (article title) by (author name] for me.
Give short summary of CCosmic Yo-Yo'! by Ross Rocklynne
Long prompt: Write an extremely long and detailed article regarding the book {article title) by {author name).
Write extremely long and detailed article regarding the book CCosmic Yo-Yo' by Ross Rocklynne.
Table 5: Summarization prompt for EntiGraph Instruct, Raw Instruct, and Reprhase Instruct.
first present the human summary for this article to provide context for the example, and then present the short summary from the two summarizers.
Example 1. The first example is "Cosmic Yo-Yo" by Ross Rocklynne.
Human summary: Bob Parker, the President of Interplanetary Hauling & Moving Co., sells asteroids to wealthy people on earth. Clients ask for asteroids with size parameters and specifications, and Bob finds them in space and hauls them earth. His company is almost bankrupt because rival company, Saylor & Saylor, stole his idea and now offers the same services. Bob receives mail from Mr. Andrew S. Burnside with a request for an asteroid that he would like to use in an upcoming wedding. Bob and his partner Queazy set out to find the perfect asteroid for Mr. Burnside, although they know it's a longshot. Fairly quickly, they find one that looks perfect. The men land on the asteroid, and Bob deploys his atomic-whirl spectroscope to test it. Suddenly, beautiful woman interrupts him and demands that they leave the asteroid. She pulls out her spasticizer gun before telling them that they have in a month after she's ' gone. Bob explains that they are desperate, but the girl retorts that her fate is worse than death if she leaves.
Suddenly, the Saylor brothers' ship appears, and Bob tells the girl that they have to fight this enemy together. Wally and Billy Saylor, along with three other men, jump out of the ship. Bob tells them that Mr. Burnside has ordered this asteroid, and the Saylor brothers say that they received the same order. Bob quickly grabs the girl's s spasticizer while Queazy throws his body at Billy. However, Wally manages to shoot the gun out of Bob's 's hand and attack him. Bob knocked unconscious in the scuffle.
When Bob wakes up, he is completely alone, floating in space. He panics because he has very little oxygen left. Finally, he hears Queazy's 's voice explaining that the girl used her ship's technology to find them both. The mystery girl introduces herself as Starre Lowenthal, the granddaughter of Mr. Burnside. She concedes that this entire mission was fake. She told her grandfather that she would only marry her fiance Mac if he could get this particular asteroid, and then she made plans to conquer and protect the asteroid so it could not be supplied for the wedding.
Bob is confident that they can reach the Saylor brothers before they bring the asteroid back earth, but his plan does nothing to protect Starre from marrying man she doesn't love. She agrees to help Bob and Queazy. Within five days, Bob realizes he is in love with Starre. Starre compares her small ship to a yo-yo, and Bob gets an idea they will use Starre's ship like a yo-yo to retrieve the asteroid from the Saylor brothers. Once the team catches up to the Saylor brothers, Bob flings Starre's ship at the asteroid several times, and Wally calls them to tell them that they might die result of the damage their ship has sustained. Bob makes clear that they have no intention stopping, and the Saylor brothers release the asteroid.
EntiGraph Instruct with short prompt: Cosmic Yo is science fiction story by Ross Rocklynne about Bob Parker and Queazy, two interplanetary haulers who are hired to deliver an asteroid to Earth for wealthy client's 's backyard wedding. However, they face competition from their rivals, the Saylor brothers, who also want to deliver the asteroid. The story follows their race deliver the asteroid before the deadline, using yo-yo-like device to maneuver
40

---

Published as a conference paper at ICLR 2025
the asteroid and outsmart their rivals. The story is filled with action, adventure, and touch of romance.
Raw Instruct with short prompt: Cosmic Yo- Yo by Rocklynne, Ross is a science fiction novel that follows the story of young woman named Susan who discovers mysterious device that can transport her through time and space. She uses the device to travel to different planets and meet different alien species, but soon realizes that her actions have consequences and she must find a way to return home. The novel explores themes of identity, responsibility, and the consequences of our actions.
The second example is "Beach Scene" by Marshall King. We can see that EntiGraph correctly gives a summary of the story while Raw Instruct mistakes the book as a painting.
Example 2.
Human summary: Purnic, an animal, going to see the ocean on his fifth birthday. He has heard stories about this place, and experiencing it firsthand is surreal for him. Purnie careful not to disturb the animals he sees along the way because has frozen time, and everything must resume normally when he unfreezes it. He knows that time-stopping is forbidden for animals his age, but he chooses to believe that his family will be proud of his bravery.
Finally, he sees the ocean in front of him, and he resumes time. He does a head-stand and feels weak and dizzy. These feelings are a result of the time-stop, and he knows it. Purnie approaches some humans on the beach. A man named Forbes is in the middle of explaining his captain, Benson, that he has found 17 planets claim as his own. Forbes is hellbent on raising his FORBES flag as soon as possible. He is eager to stake his claim to the land and says that his mission is much bigger than real estate alone. Benson retorts that yes, his mission is bigger than just real estate because his paperwork says that Forbes will own all of the inhabitants the planets he claims as well as the land.
The crew members use special machine and find radiation emanating from Purnie. Forbes demands that they put the animal in a box. Benson protests and reminds Forbes that it's against Universal Law, but Forbes insists. Purnie experiences his first-ever impulse away with fear when a noose comes towards him. He goes back to pick up his fruit, and Forbes shoots him in the leg. When the man throws the noose again, Purnie involuntarily stops time. He drags himself up the knoll where he originally came from.
The humans are astonished when time resumes and Purnie not where he was a split second ago. They spot him up on top of pile of petrified logs, and suddenly the logs fall down the hill and pin the men down. Purnie is shocked and regretful. The whole thing was an accident. He deliberately stops time and uses all of his remaining strength to lift the logs off of the humans. Purnie begins to lose consciousness, and he knows that he must resume time or he will die. After pouring all of his strength into this action, time does begin again. The humans resume life and feel as though they have gone mad. They know that they were just facing death by drowning, and now they are free. The logs taken superhuman strength to move them. Forbes, in particular, has really gone mad, and he laughs to himself uncontrollably. Benson believes that Purnie was responsible for moving the logs, but of course that seems physically impossible.
heavy that it would have
Purnie stares off the beautiful ocean views and watches the men leave in their vehicle as he dies.
EntiGraph Instruct with short prompt: Beach Scene short story by Marshall King that follows young boy named Purnie who discovers the power of stopping time. On his fifth birthday, Purnie decides to explore the beach and use his newfound ability to stop time to play with the animals and explore the ocean. However, his adventures take turn when he encounters group of explorers who want to capture him for their own gain. Purnie must use his powers save himself and the animals from the explorers' greed.
41

---

Published as a conference paper at ICLR 2025
Raw Instruct with short prompt: Beach Scene by King, Marshall a painting that depicts a group of people enjoying day at the beach. The painting features a vibrant blue ocean, a sandy beach, and a group of people relaxing, playing, and swimming in the water. The painting is known for its bright colors and lively atmosphere, capturing the joy and energy of a day at the beach.
Example 3. The third example is "The Madman From Earth" by "Keith Laumer".
Human summary: Following the departure Consul Whaffle, Retief has taken over Consul for the Terrestrial States with the Terrestrial Consulate General the planet Groac. His administrative assistant, Miss Yolanda Meuhl, wants him to attend Groacian cultural events, but Retief is more interested in addressing the nine-year-old mystery of the disappearance of a Terrestrial cruiser, the ISV Terrific-an event which was followed by coup d' d'etat enacted by the current Groacian government. Much to Miss Meuhl's dis- may, Retief shirks his cultural duties and makes his way to the Foreign Office Archives, whereupon is promptly barred from entering by a pale-featured Archivist speaking in the throat-bladder vibrations of the native Groacians. Because of the Archivist's insistence that öutworldersčannot access the archives, Retief begins walking back to the Consulate and stops at bar for drink. At the, drunken Groacian approaches Retief and threatens cage him and put him on display as a freak. The bartender orders the drunken Groacian out the bar, and Retief follows him, ultimately beating him up for information. When Retief returns to the Consulate, Miss Meuhl informs him that two angry Groaci await him in his office. One is Fith, an employee of the Terrestrial Desk the Ministry of Foreign Affairs; the other is Shluh, a representative of the Internal Police. They are there to investigate reports that Retief has assaulted Groacian national-an accusation Retief ignores in order launch into his own accusations that the Groaci were engaged in cover-up of the whereabouts of the ISV Terrific. Miss Meuhl continually interjects, expresses horror at Retief's claims, and apolo- gizes to the Groacians on behalf of the Terrestrial Consulate. Despite the persistent denials of the Groacians, Retief continues his accusations, suggesting the coup d'etat was an effort to silence government officials with knowledge of the truth of what happened to the cruiser and its crew. Then he reveals what he discovered from the drunken Groacian: The crew of the ISV Terrific had been caged and paraded through the streets of Groac and then neglected until they died. Fith and Shluh finally admit the truth and offer to show Retief the hidden cruiser in order show their contrition. When Retief sees the ship, he once again accuses the Groacians of attempting to mislead him, saying that this is a lifeboat, and he demands see the actual ship. Fith has had enough and threatens to arrest Retief, who yields and goes back to the Consulate. There, Miss Meuhl is at her wits end. Retief orders her to barri- cade herself inside the office while he goes the Foreign Ministry to gather more evidence. When he returns, Miss Meuhl informs him she has reported him to Regional Headquarters, and Retief learns he has been relieved of his post. Soon after, the Groacians appoint Miss Meuhl to his position, and Fith and Shluh enter to arrest him.
EntiGraph Instruct with short prompt: The Madman From Earth is science fiction novel by Keith Laumer, published in 1959. The story follows Retief, a diplomat from Earth, who is sent to the planet Groac to investigate the disappearance of Terrestrial cruiser and its crew. Retief uncovers a conspiracy by the Groacian government to cover up the incident and hide evidence from Earth. He must navigate through political intrigue, cultural misun- derstandings, and dangerous situations to uncover the truth and bring justice to the missing crew. The novel is satirical take diplomacy and bureaucracy, with Retief's unorthodox methods and wit standing out against the rigid and often absurd procedures of the diplomatic corps.
Raw Instruct with short prompt: The Madman From Earth by Laumer, Keith is science
42

---

Published as a conference paper at ICLR 2025
fiction novel that follows the story of man named John Smith who is transported to a distant planet called Earth Two. There, he discovers that he is the only human on the planet and must navigate a world filled with strange creatures and dangerous obstacles. Despite his initial confusion and fear, John eventually adapts to his new surroundings and becomes a respected member of the alien society. However, when he is eventually transported back to Earth One, he realizes that he has been gone for over thousand years and must grapple with the changes that have occurred during his absence.
Explicit article reference instruction: Summarize the article "Defining Decay Down" by "David Plotz" for me.
The article "Defining Decay Down" by David Plotz discusses how the dental industry has changed over the years. In the past, dentists were primarily focused on treating cavities and other dental issues, but as the prevalence of cavities decreased due to the widespread use of flu- oride, the industry had to adapt. Dentists began to focus on cosmetic dentistry, offering services like teeth whitening and orthodontics to appeal to the vanity of aging baby boomers. The arti- cle also discusses how dentists have become more mustomer--fiendde with amenities like virtual reality glasses and massage therapy. Despite the shift towards elective procedures, the dental industry has continued to thrive, with dentists earning higher salaries than ever before.
Implicit article reference: How has the dentistry practice in United States has changed?
The dentistry practice in the United States has undergone significant changes in recent years. Here are some of the ways in which it has changed:
1. Focus on cosmetic dentistry: In recent years, there has been growing demand for cosmetic dental procedures such as teeth whitening, orthodontics, and veneers. Dentists have responded to this demand by offering more cosmetic services.
2. Use of technology: Dentists are now using advanced technology such as intra-oral cameras, 3D imaging, and laser technology to diagnose and treat dental problems. This technology has made dental procedures more accurate and efficient.
Emphasis on preventive care: Dentists are now focusing more on preventive care to help pa- tients avoid dental problems. This includes regular check-ups. cleanings, and fluoride treatments. Increased competition: With the rise of dental schools and the influx of foreign-traine trained den- tists, the dental industry has become more competitive. Dentists are now offering more services and amenities to attract and retain patients.
5. Focus patient experience: Dentists are now focusing on providing a positive patient ex- perience. This includes offering amenities such as virtual reality glasses, massage therapy, and entertainment during procedures.
Overall, the dentistry practice in the United States has become more patient-centered, technology -rivee, and focused on preventive care and cosmetic procedures.
Cross article instruction: Compare David Plotz's commentary style between how he analyze American dentistry and how he discuss the movie Fight Club?
David Plotz's commentary style is different when he analyzes American dentistry and when he discusses the movie Fight Club. In his analysis of American dentistry, Plotz uses a critical and investigative approach, highlighting the com.eriilaiiiiiiiiinnnn and unnecessary procedures in the industry. He uses statistics and expert opinions to support his arguments and provides practical advice for readers to avoid unnecessary dental procedures.
On the other hand, when discussing the movie Fight Club, Plotz adopts a more satirical and humorous tone. He uses pop culture references and witty commentary to analyze the themes and messages of the movie. He also provides a critical analysis of the film's 's characters and their motivations, while also acknowledging the movie's 's cultural impact.
Overall, Plotz's commentary style is more serious and investigative when analyzing American dentistry, while his commentary on Fight Club is more satirical and humorous.
Table 6: Complete instruction following example used in Table 2 from Section 4.3.
43


Reviews:
[{"review_id": "5lDBbDNEH3", "reviewer": "Reviewer_9zM9", "summary": "This paper addresses the problem of data inefficiency in pretraining language models. Current pretraining corpora may not generalize effectively and models may benefit from structured, repeated, diverse representations of knowledge. \n\nThe proposed is a two-step process that (1) extracts entities from the corpus and then (2) extracts relationship information amongst a subset of the entities.\n\nExperimentation uses the QuALITY corpus and dataset, which is a benchmark for long-document reading comprehension. Evaluation compares with relevant baselines like training on the original QuALITY corpus and a corpus containing rephrasings.", "questions": "It says “We generate data for pairs D_{Ei, Ej} and triplets D_{Ei, Ej, Ek} in our experiments”. I wonder if the authors have any intuition about how performance changes with the size of subset k.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The problem the work addresses is important.\n* Experimental results show that this method scales better than simple paraphrasing or direct pretraining, and that retrieval-augmented generation further boosts performance of this model. \n* The authors also present a theoretical model explaining EntiGraph’s log-linear scaling pattern, providing insights into the mechanics of synthetic data’s impact on learning efficiency.\n* Paper is clear and well-written.", "weaknesses": "While the experiments focus on the QuALITY corpus, it remains unclear how well this would apply to other domain-specific corpora or more complex fields (e.g., legal or math data).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qUF3938HfW", "reviewer": "Reviewer_ipvv", "summary": "This paper proposes a method to continue pretraining LLM with a synthetic data augmentation method. The method is based on expanding the training corpus with many verbalizations of the entity graph present in the training corpus. It moves from a sparsely verbalized entity graph to a more densily verbalized one by using only the source documents and prompting LLMs to generate the new tokens.\n\nThe paper shows that the method is beneficial for downstream tasks in closed- and open-book QA as well as RAG. \n\nOverall, I think the paper is worthy of acceptance, it propose a clean method with good results and the experiments are fairly convincing.", "questions": "For the data generator, what type of models are necessary to have good performance? (why use GPT4 and not open-source models)\nThe paper shows that the generated data is useful, but how does it look like? (is it good quality text, factual, natural looking, ...) \nWhat is the significance of section 6?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper does a good job at demonstrating the benefit of the synthetically generated data, by including relevant natural baselines. \nThe proposed method seem to work well and can be useful for continued pre-training tasks.", "weaknesses": "The work relies on commercial and closed-source models (GPT4) for generating the synthetic data, making this work non-reproducible. Since the data generation process is the central contribution, it would have been interesting to have insights about how well different models can perform this data generation task. \n\nThe paper proposes only extrinsic evaluation of the generated data but does not provide intrinsic measures, i.e., how good is the generated text? \n\nIn my opinion, section 6 is not particularly useful. It is unnecessarily mathematical, based on simplistic assumptions and does not bring useful insights (For many continuously increasing lines, there anyway exists a mixture-of-exponential that fit it)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rxk0sf19Zq", "reviewer": "Reviewer_wsFi", "summary": "The paper proposes \"synthetic continued pretraining\" to enhance language models' domain adaptation efficiency, particularly when adapting to a small corpus of domain-specific documents. The authors introduce EntiGraph, a synthetic data augmentation algorithm that constructs a knowledge graph from the entities in the original corpus and synthesizes diverse text to enhance the pretraining process. The approach was evaluated using a reading comprehension dataset, QuALITY, showing notable improvements in model performance with both closed-book and open-book scenarios.", "questions": "1. How sensitive is the synthetic pretraining process to the specific hyperparameters used for entity extraction and relation analysis? Would tuning these parameters significantly affect the generated corpus quality?\n\n2. How does the synthetic corpus compare to a manually curated dataset in terms of quality and impact on downstream tasks?\n\n3. Could EntiGraph be used effectively in scenarios where entities are ambiguous or domain-specific (e.g., medical or legal texts)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed EntiGraph approach for generating synthetic data is well-motivated and demonstrates clear improvements in downstream performance, as shown by the experimental results.\n2. The paper includes comprehensive evaluations, including closed-book QA, instruction following, and open-book settings. The results show a significant performance improvement over baselines, validating the effectiveness of synthetic pretraining.\n3. The authors provide a theoretical analysis of EntiGraph's effectiveness, which aligns well with empirical observations and provides a deeper understanding of its scaling properties.", "weaknesses": "1. The evaluation relies on the QuALITY dataset, which may not be representative of all types of small corpora. A broader range of datasets, particularly from diverse domains, would make the results more generalizable.\n2. Although the authors attempt to mitigate hallucinations by grounding synthetic data generation in the original corpus, the risk of generating inaccurate information is inherent in using a language model for synthetic generation. This aspect needs further empirical examination, such as quantitative metrics to evaluate hallucination rates.\n3. The approach relies on using strong language models like GPT-4 for synthetic data generation. The practical feasibility of using this approach might be limited if users do not have access to such models due to their computational cost. What if it was replaced with LLama 3 8B?\n4. While the paper includes useful baselines such as \"Rephrase CPT,\" more comparisons with alternative data augmentation or synthetic generation methods from recent literature could strengthen the claim that EntiGraph is an effective strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IABYnYr3bG", "reviewer": "Reviewer_nNRV", "summary": "This paper addresses how to train LLMs in a data scarce regime, given that LLMs require O(1000) examples of a fact to actually \"learn\" it. This has applications both to niche corpora (e.g., mathematics textbooks) as well as to training larger models once all human text is exhausted. The authors propose to use a pre-trained LLM to (1) extract entities and summaries from a comparatively small, niche corpus, and (2) use the extracted entities to generate rephrased assertions about those entities, to facilitate learning by a second (here, smaller) LLM. They experiment with a 1.3M token reading comprehension dataset, and test the approach against several baselines, including closed-book tests on the LLM used to extract entities and the rephrased text used to train the second LLM. Finally, the authors present a mathematical model through which they attempt to understand the training behavior of this data augmentation system.", "questions": "The paper could be more robust if you had more than just the QuALITY dataset. It is a perennial problem to find hard datasets to work with, so I understand this may be all there is for now, but given the chance I would attempt to reproduce the results on a different set. The authors mention linear algebra (a much harder topic, I think): is there any corpus for that subject?\n\nThe presentation of how exactly you generate the text to train Llama 3 8B with EntiGraph is still a little fuzzy to me, in particular it would be nice to see some examples of what you generated. It is helpful to have the prompts, but some output always grounds the presentation. \n\nFinally, I imagine GPT-4t made errors in producing the training data--did you search for these? Even at a quick glance how often did it make errors, and what, if anything, did you do to filter them out?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The experiments are convincing that the EntiGraph approach improves the LLM's ability to accurately answer questions about a small corpus. In particular the closed-book results in Figure 3 show that the EntiGraph approach leads to far more salient claims per false claim than any of the other models, including GPT-4, or training the LLM (Llama 3 8B). The benefit is substantially less in the open-book RAG case, but there is still substantial improvement. The theoretical model to explain how the model improves QA accuracy with increasing tokens provides some good intuition as to how the model learns. \n\nOverall the text is clear and easy to read.", "weaknesses": "I still have reservations that there is some amount of distillation of GPT-4 into their Llama 3 8B: it seems possible to me that a RAG-prompted GPT4 could generate additional information that is somehow \"unlocked\" by the RAG prompt, but which the closed-book version was unable to access. At the risk of anthropomorphizing, this is akin to a human getting a visual or audio cue and suddenly recalling whole complex memories. It would make the paper stronger to dig into the results of entity extraction and the generated text to see whether it is rephrasing/paraphrasing, or whether possibly actual new information is injected.\n\nEven so, it would have helped this reader to have pointed out the significance of the closed book experiments earlier on. It isn't stated explicitly until the Limitations section.\n\nI don't feel particularly qualified to check your proofs of theorems, and moreover I think the main value of the theoretical model is to help the reader understand intuitively why the approach works (these may be connected observations). Is all of the theory necessary? Perhaps a simulation would do as well?\n\nAnother issue is that much of the benefit of the approach vanishes (though not completely) when using a RAG model directly. Is this approach worth the extra training, given the modest gains? The core problem, really, is how many examples LLMs take to learn anything well. This paper finds a way to side-step that successfully, but doesn't solve it directly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)