Synthetic continued pretraining

## Abstract

Abstract Pretraining on large-scale, unstructured internet text enables language models to acquire a significant amount of world knowledge.
However, this knowledge acquisition is data-inefficient —to learn a given fact, models must be trained on hundreds to thousands of diverse representations of it.
This poses a challenge when adapting a pretrained model to a small corpus of domain-specific documents, where each fact may appear rarely or only once.
We propose to bridge this gap with synthetic continued pretraining : using the small domain-specific corpus to synthesize a large corpus more amenable to learning, and then performing continued pretraining on the synthesized corpus.
We instantiate this proposal with EntiGraph, a synthetic data augmentation algorithm that extracts salient entities from the source documents and then generates diverse text by drawing connections between the sampled entities.
Synthetic continued pretraining with EntiGraph enables a language model to answer questions and follow generic instructions related to the source documents without access to them.
If, instead, the source documents are available at inference time, we show that the knowledge acquired through our approach compounds with retrieval-augmented generation.
To better understand these results, we build a simple mathematical model of EntiGraph, and show how synthetic data augmentation can “rearrange” knowledge to enable more data-efficient learning.

## 1 Introduction

Language models have demonstrated a remarkable ability to acquire knowledge from unstructured text, enabling them to perform challenging knowledge-intensive tasks (Brown et al., ; OpenAI et al., ; Gemini, ; Anthropic, ; Dubey et al., ; Gunter et al., ).
These successes are enabled by the combination of the next-token prediction objective (Shannon, ) and large-scale internet data (Common Crawl, ).
However, it is becoming increasingly apparent that this approach is *data-inefficient;* for example, a 13-year-old human acquires knowledge from fewer than 100M tokens, while state-of-art open-source language models are trained on 15T tokens (Warstadt et al., ; Dubey et al., ).
Recent works have highlighted a range of related problematic phenomena, including the “reversal curse”, where models struggle to learn the relation “B=A” when trained on “A=B” (Berglund et al., ), and the requirement that models be exposed to thousands of examples per fact for knowledge acquisition (Allen-Zhu & Li, ).

These drawbacks pose a challenge when adapting the next-token prediction paradigm to learn from small-scale corpora.
Because large-scale pretrained models already capture much of public common knowledge, further advancements will necessitate learning from the tails of the distribution (Kandpal et al., ): niche data that is either contained in small, private domains or appears only once or twice on the internet.
This challenge of data-efficient, parametric knowledge acquisition is becoming increasingly important as the growing compute capacity enables language model providers to exhaust publicly available data (Muennighoff et al., ; Villalobos et al., ).

We propose to address this problem of acquiring knowledge from small corpora with *synthetic continued pretraining*.
To illustrate, consider the problem of teaching a language model a new area of mathematics, succinctly documented by a small set of authoritative textbooks.
Directly training the model on those textbooks is unlikely to be effective due to the limited volume of text (typically only tens of thousands of words), and the model will struggle to generalize from this compressed representation of knowledge.
In contrast, learning well-established areas of mathematics like linear algebra is more straightforward because a large-scale corpus with diverse knowledge representations is accessible: for example, online lecture notes, Stack Exchange discussions, or Python implementations of the singular value decomposition.
Synthetic continued pretraining bridges this gap by first converting a small and data-constrained domain into a synthetic corpus with diverse knowledge representations, and then continuing pretraining on it.

One basic approach is to simply paraphrase or rewrite the source documents in multiple ways.
However, we demonstrate that this generic rephrasing does not cover the gap in the diversity of knowledge representations.
We repeatedly rephrase a small corpus and find that the value of incremental synthetic data quickly decreases, with downstream model performance scaling poorly.
We attribute this failure to the lack of diversity in paraphrasing alone.
In the linear algebra example, online lecture notes and Stack Exchange discussions go beyond a simple rewrite of any textbook—they provide deeper analysis and application of the underlying concepts and techniques.

Figure: Figure 1: Synthetic continued pretraining (synthetic CPT) converts a small source corpus into a large synthetic corpus that is amenable to learning via standard continued pretraining. We instantiate synthetic CPT using a synthetic data augmentation algorithm called EntiGraph, which forms a knowledge graph over entities extracted from documents, and then prompts an LM to synthesize a text-based representation of the graph.
Refer to caption: x1.png

To address this shortcoming, we propose EntiGraph, an entity-centric augmentation algorithm.
EntiGraph first breaks down a text corpus into a list of entities and then uses a language model to generate text descriptions about relations among the extracted entities, iteratively “filling in” the knowledge graph underlying the corpus (Figure 1).

To concretely measure progress towards effective knowledge acquisition from small corpora, we propose an experimental setting based on a standard reading comprehension dataset (QuALITY, Pang et al. ()).
This setup enables the evaluation of synthetic data generation methods for data-efficient learning without incurring the high compute costs of pretraining from scratch.
Specifically, we evaluate methods in a scenario where we are given access to a collection of 265 books, totaling 1.3M tokens.
Our task is to synthesize a corpus such that continued pretraining on it enables a model to answer queries (e.g., multiple-choice QA or user instructions related to the book content) *without* access to the source texts.

In our main experiments (§5), we use EntiGraph to generate 455M synthetic tokens from 1.3M real tokens using gpt-4-turbo (OpenAI et al., ).
Then, we continually pretrain Llama 3 8B (Dubey et al., ) on the synthetic tokens and evaluate its QA accuracy on the QuALITY question set.
We observe a log-linear scaling trend in the accuracy as the number of tokens increases, up to 455M synthetic tokens (§4.2).
At the endpoint, we find that synthetic continued pretraining with 455M EntiGraph tokens provides 80% of the accuracy improvement of having those source documents available at inference time (§5).
Beyond QA accuracy, we also perform instruction tuning on the continually pretrained model and find that it is capable of following open-ended instructions (e.g., summarization) related to the QuALITY books (§4.3).

To summarize, our key contributions are as follows:

- •
We propose to learn from small corpora with synthetic continued pretraining—converting the small corpus into a large, diverse, synthetic corpus and continuing pretraining on it—and instantiate this approach using the EntiGraph synthetic data augmentation algorithm (§2.2).
- •
We demonstrate that continued pretraining on the EntiGraph-synthesized corpus yields a QA accuracy scaling trend that is log-linear in the synthetic token count, significantly outperforming continued pretraining on the original documents or paraphrases (§4.2).
Furthermore, we show that instruction tuning the EntiGraph continually pretrained model enables it to follow more diverse queries related to the source documents (§4.3).
- •
We complement the main experiments with an open-book setup (§5), providing the model with access to the source documents when answering queries.
We demonstrate that the knowledge acquired through synthetic continued pretraining with EntiGraph is *complementary* to the knowledge accessed through retrieval-augmented generation (RAG, Lewis et al. ())—RAG with the EntiGraph continually pretrained model outperforms RAG with the base model.
- •
Lastly, we build a mathematical model that captures the intuition behind synthetic data augmentation with EntiGraph.
Analysis of this model provides a parametric formula for the scaling trend of a continually pretrained model’s accuracy with respect to EntiGraph synthetic tokens, which closely matches our empirical observations (§6).

Practically, synthetic continued pretraining using EntiGraph enables pretrained language models to adapt to specialized domains by acquiring *parametric* knowledge, rather than the non-parametric knowledge accessed through retrieval methods.
At a higher level, our approach points toward a family of synthetic data generation algorithms that allow us to convert compute into data efficiency for (continued) pretraining (Kaplan et al., ).

### 1.1 Related work

We next discuss recent work most related to our setting of synthetic data generation for continued pretraining.
In Appendix A, we provide an extended survey of classical work on synthetic data generation and continual learning.

#### Synthetic generation of pretraining data.

Recent approaches synthesize *pretraining* data using hierarchical prompting methods to promote dataset diversity.
Eldan & Li () prompt API-based LLMs to generate children’s stories containing sampled keywords, and demonstrate that even small language models trained on their dataset can generate fluent text.
Gunasekar et al. () synthesize a diverse dataset of textbooks and code exercises by conditioning on topic, target audience, and function names, and later release strong LLMs pretrained on synthetic data in follow-up work (Li et al., ; Abdin et al., ; ).
However, their datasets and prompts are not publicly available.
Maini et al. () prompt an LM to rephrase documents for pretraining, improving training efficiency.
Different from all above works, our focus is teaching a pretrained LLM the knowledge of a small corpus.
Mecklenburg et al. () consider task-specific finetuning and propose a fact-based synthetic QA generation procedure, but do not show improvement on generic instruction following tasks beyond simple QA.
We instead focus on teaching a model generally useful knowledge about a small corpus, untied to a particular downstream task.
Ovadia et al. () continually pretrain Llama 2–based language models on synthetic paraphrases of Wikipedia articles, but do not observe consistent performance improvements.
We adapt the approach of Maini et al. () and Mecklenburg et al. () to our small corpus setting as the “Rephrase baseline” in §4.
We find that our graph-based augmentation algorithm outperforms it, likely because our approach enforces diversity through entity-based generation.

#### Continued pretraining.

Continual or continued *pretraining* works (Gururangan et al., ) successfully adapt pretrained large language models to broad target domains such as code (Rozière et al., ), medicine (Chen et al., ), or mathematics (Lewkowycz et al., ; Shao et al., ; Azerbayev et al., ) by collecting massive datasets (often $>$100B tokens, shown in Table 1) and developing efficient training recipes using causal language modeling (Gupta et al., ; Ibrahim et al., ; Parmar et al., ).
This work aims to extend the success of continued pretraining to small, specialized domains such as proprietary document stores.
Observing that standard continued pretraining is ineffective on small corpora, we propose a knowledge graph–inspired approach to synthesize a diverse related corpus and find it more amenable to learning.

#### Knowledge editing.

A related line of literature updates language models with small units of factual knowledge, such as $(\text{subject, relation, object})$ tuples.
Zhu et al. () studies a constrained fine-tuning approach, limiting the model’s complexity to better suit the learning of simple factual relations.
Later approaches attempt to localize where factual knowledge is stored in Transformers and update only those weights (Mitchell et al., ; Meng et al., ; ), or maintain an external memory of edits and prepend them as context during generation (Zhong et al., ; Cohen et al., ).
Most relevant to our work is deductive closure training (Akyürek et al., ), which first deduces implications of a factual edit and then finetunes the language model on those implications.
The line of knowledge editing differs from our setting in that we aim to learn from a small corpus of documents, rather than atomic, sentence-length facts.

## 2 Our method

We focus on learning parametric knowledge from a small text corpus.
Our goal is to continually pretrain a language model to acquire the knowledge of a niche corpus of documents.
Observing that simple continued pretraining is ineffective (§4), we propose to use synthetic continued pretraining, which first uses the small corpus to synthesize a larger one more amenable to learning, and then continues pretraining on the synthetic corpus.
In this section, we first outline this problem setting and our evaluation approach in more detail (§2.1).
Then, we provide a concrete instantiation of synthetic continued pretraining using a data augmentation algorithm called EntiGraph (§2.2).

### 2.1 Problem Setup

**Table 1: Comparing the scale of modern continued pretraining (CPT) works with our small corpus setting. Prior work adapts language models to broad domains with diverse, large-scale corpora. We aim to downscale continued pretraining to small corpora; we use a corpus that is 10,000$\times$ smaller than the smallest modern corpus for domain-adaptive CPT.**
| Study | Domain | Model Parameter Count | Total Unique CPT Tokens |
| --- | --- | --- | --- |
| Minerva (Lewkowycz et al., ) | STEM | 8B, 62B, 540B | 26B-38.5B |
| MediTron (Chen et al., ) | Medicine | 7B, 70B | 46.7B |
| Code Llama (Rozière et al., ) | Code | 7B, 13B, 34B | 520B-620B |
| Llemma (Azerbayev et al., ) | Math | 7B, 34B | 50B-55B |
| DeepSeekMath (Shao et al., ) | Math | 7B | 500B |
| SaulLM-7B (Colombo et al., ) | Law | 7B | 30B |
| SaulLM-{54, 141}B (Colombo et al., ) | Law | 54B, 141B | 520B |
| HEAL (Yuan et al., ) | Medicine | 13B | 14.9B |
| Our setting | Articles & Books | 8B | 1.3M |

#### Continued pretraining on small corpora.

We focus on approaches that use continued pretraining to teach a pretrained language model the knowledge of a small set of source documents ${\mathcal{D}}_{\text{source}}$.
These approaches acquire “parametric knowledge”, i.e., the knowledge of ${\mathcal{D}}_{\text{source}}$ is learned in the model’s parameters much like during the pretraining process.

#### Synthetic c ontinued p re t raining (synthetic CPT).

First, we apply a synthetic data generation algorithm $\mathcal{A}_{\text{synth}}$ to convert a small corpus ${\mathcal{D}}_{\text{source}}$ into a synthetic corpus $\mathcal{D}_{\text{synth}}$:

$$ $\mathcal{A}_{\text{synth}}:{\mathcal{D}}_{\text{source}}\longmapsto\mathcal{D} _{\text{synth}}.$ (1) $$

Then, we perform continued pretraining on $\mathcal{D}_{\text{synth}}$ instead of on ${\mathcal{D}}_{\text{source}}$.
We implement $\mathcal{A}_{\text{synth}}$ using a language model.
A natural concern is that the language model may hallucinate and fabricate false knowledge.
Therefore, we consider synthetic data augmentation algorithms that condition the generation process on the source documents to improve the synthesized data’s faithfulness.

#### Evaluation with knowledge-intensive queries.

We evaluate the quality of a synthetic data augmentation algorithm $\mathcal{A}_{\text{synth}}$ by testing whether the downstream synthetic CPT model has effectively acquired the knowledge of ${\mathcal{D}}_{\text{source}}$ in its parameters.
More precisely, we curate some test queries ${\mathcal{Q}}_{\text{test}}$ that probe the knowledge about ${\mathcal{D}}_{\text{source}}$ acquired by the model.
For example, in the linear algebra setting, ${\mathcal{Q}}_{\text{test}}$ could be held-out exam questions.
To test parametric knowledge, we do not allow the model to access the source documents ${\mathcal{D}}_{\text{source}}$ at test time.
Therefore, the queries cannot be ambiguous without access to ${\mathcal{D}}_{\text{source}}$.
For example, a reading comprehension question like “Where was he born?” is ambiguous without context.
Altogether, we can evaluate data augmentation algorithms $\mathcal{A}_{\text{synth}}$ for synthetic CPT using a paired source corpus and related test queries $({\mathcal{D}}_{\text{source}},{\mathcal{Q}}_{\text{test}})$.

### 2.2 EntiGraph

Next, we present EntiGraph, our instantiation of a synthetic data augmentation algorithm $\mathcal{A}_{\text{synth}}$.
At a high level, EntiGraph generates diverse representations of knowledge from a small corpus ${\mathcal{D}}_{\text{source}}$ by using a prompted LLM to synthesize a knowledge graph representation of ${\mathcal{D}}_{\text{source}}$.
EntiGraph consists of two steps/prompts: extracting entities from the document and analyzing relations among an arbitrary subset of the entities (Figure 1).
Altogether, this hierarchical prompting strategy *externalizes* the problem of generating diverse synthetic text to a combinatorial structure—namely, a graph relating various entities appearing in the corpus documents.
In what follows, we provide abbreviated prompts to illustrate the algorithm, and defer full prompts to Appendix G.1.

#### Step 1: Entity extraction.

First, EntiGraph extracts a list of salient entities $\{E_{1},E_{2},\dots,E_{n}\}$ from the document ${\mathcal{D}}_{\text{source}}$ using an entity_extraction prompt:

$$ $\{E_{1},E_{2},\dots,E_{n}\}\sim\mathsf{LM}_{\text{aug}}\big{(}\texttt{entity_ extraction}({\mathcal{D}}_{\text{source}})\big{)}.$ $$

We show the abbreviated entity_extraction prompt below:

In the linear algebra example, ${\mathcal{D}}_{\text{source}}$ could be one specific linear algebra textbook.
We would expect to extract entities such as $\{E_{1}=\texttt{Linear space},~{}E_{2}=\texttt{Vector},~{}E_{3}=\texttt{SVD},\dots\}$.

#### Step 2: Relation analysis.

Next, EntiGraph analyzes the relations among subsets of entities.
The intuition is to thoroughly explore the edges of the knowledge graph underlying the source document ${\mathcal{D}}_{\text{source}}$, analogous to a student writing diverse notes about a linear algebra textbook.
We apply a relation_analysis prompt to describe how a subset of $k\leq n$ entities are related in the context of the source document ${\mathcal{D}}_{\text{source}}$, obtaining a synthetic document

$$ $\widetilde{D}_{E_{i_{1}}\dots E_{i_{k}}}\sim\mathsf{LM}_{\text{aug}}\big{(} \texttt{relation_analysis}(D,E_{i_{1}},E_{i_{2}},\dots,E_{i_{k}})\big{)}.$ $$

Specifically, we use the prompt below (abbreviated):

For example, if $E_{1}=\texttt{Linear space}$ and $E_{2}=\texttt{Vector}$, $\widetilde{D}_{E_{1}E_{2}}$ could include the text
Based on the textbook, a vector is an element of a linear space...
Exhaustively enumerating all possible subsets of the $n$ extracted entities is impractical.
We choose to generate data for all pairs $\widetilde{D}_{E_{i}E_{j}}$ and triplets $\widetilde{D}_{E_{i}E_{j}E_{k}}$ in our experiments.

#### EntiGraph synthetic corpora.

Finally, we collect all sampled synthetic texts from Step 2 as the EntiGraph output: ${\mathcal{D}}_{\text{EntiGraph}}=\{\widetilde{D}_{E_{i_{1}}\dots E_{i_{k}}},\dots\}$.
Altogether, we described a data augmentation algorithm mapping a small source corpus ${\mathcal{D}}_{\text{source}}$ to a larger synthetic corpus ${\mathcal{D}}_{\text{EntiGraph}}$, as in (1).

## 3 Experiment setup

In this section, we describe in detail how we evaluate a given data augmentation algorithm $\mathcal{A}_{\text{synth}}$.
As described in the problem setup (§2.1), we evaluate such algorithms $\mathcal{A}_{\text{synth}}$ by evaluating whether a language model continually pretrained on their output synthetic corpus $\mathcal{A}_{\text{synth}}({\mathcal{D}}_{\text{source}})$ can accurately answer test queries ${\mathcal{Q}}_{\text{test}}$ about the source documents ${\mathcal{D}}_{\text{source}}$.

In our main experiments, we use queries that are unambiguous even without the source documents ${\mathcal{D}}_{\text{source}}$, and disallow the model from accessing ${\mathcal{D}}_{\text{source}}$ while answering the queries ${\mathcal{Q}}_{\text{test}}$ (§2.1).
This allows us to evaluate which data augmentation algorithm best promotes the acquisition of parametric knowledge through synthetic CPT.
Later, in §5, we consider an open-book setting where the model can access both the source documents ${\mathcal{D}}_{\text{source}}$ and test queries ${\mathcal{Q}}_{\text{test}}$ at the same time, in order to test how the parametric knowledge acquired through synthetic CPT composes with non-parametric access to knowledge through retrieval (Lewis et al., ).

We next introduce the small corpus and related test queries $({\mathcal{D}}_{\text{source}},{\mathcal{Q}}_{\text{test}})$ used in our experiments.

#### QuALITY corpus 𝒟 source subscript 𝒟 source {\mathcal{D}}_{\text{source}} caligraphic_D start_POSTSUBSCRIPT source end_POSTSUBSCRIPT .

Our corpus and test queries are based on the QuALITY dataset (Pang et al., ), a long-document comprehension benchmark.
The QuALITY corpus ${\mathcal{D}}_{\text{source}}$ is composed of 265 articles and short books on genres ranging from science fiction to journalism, with an average length of $\sim$5,000 tokens.

#### QuALITY test queries 𝒬 test subscript 𝒬 test {\mathcal{Q}}_{\text{test}} caligraphic_Q start_POSTSUBSCRIPT test end_POSTSUBSCRIPT .

To curate the test queries ${\mathcal{Q}}_{\text{test}}$, we use the 10-20 multiple choice questions accompanying each article in QuALITY.
These questions serve as high-quality knowledge probes on ${\mathcal{D}}_{\text{source}}$, but the query phrasing often presupposes the reading comprehension context (e.g., “What does the author think about…”).
We remove ambiguity by contextualizing them with the corresponding article reference: “In the context of article {article_name} by {author_name}, what does the author think about…”.
Altogether, this provides us with 4,609 unambiguous queries ${\mathcal{Q}}_{\text{test}}$ to test the parametric knowledge of our continually pretrained language models.

#### Evaluation on instruction-tuned summarization.

In addition to evaluation using the above test queries ${\mathcal{Q}}_{\text{test}}$, we also instruction tune the continually pretrained LMs and evaluate them on more general instruction following queries.
Specifically, we evaluate their closed-book summarization abilities by prompting them to generate summaries of QuALITY articles given only title and author.

#### Performance with strong API-based LLMs.

In our continued pretraining setting, we must select a corpus ${\mathcal{D}}_{\text{source}}$ that is not already well-represented in standard pretraining datasets.
As an initial test of the obscurity of the QuALITY corpus ${\mathcal{D}}_{\text{source}}$, we evaluate GPT-3.5 (Brown et al., ) and GPT-4 (OpenAI et al., ) on ${\mathcal{Q}}_{\text{test}}$.
In the closed-book setting, we find GPT-3.5 accuracy at 44.81% and GPT-4 accuracy at 51.30% (Figure 2).
In the open-book setting (full access to ${\mathcal{D}}_{\text{source}}$), we find GPT-3.5 accuracy at 72.60% and GPT-4 accuracy at 86.09% (Table 3).
Based on the large ($\sim$30%) improvement when ${\mathcal{D}}_{\text{source}}$ is provided, we conclude that the QuALITY corpus ${\mathcal{D}}_{\text{source}}$ is sufficiently niche to serve as an appropriate testbed.

## 4 Main experiments

In this section, we present our main experimental results(^1^11Code [https://github.com/ZitongYang/Synthetic_Continued_Pretraining.git](https://github.com/ZitongYang/Synthetic_Continued_Pretraining.git).).
Using GPT-4 (the gpt-4-turbo model as of Aug. 19, 2024) as our prompted model $\mathsf{LM}_{\text{aug}}$, we apply EntiGraph to the 1.3M token QuALITY corpus ${\mathcal{D}}_{\text{source}}$, generating a 455M token synthetic corpus(^2^22Data [https://huggingface.co/datasets/zitongyang/entigraph-quality-corpus](https://huggingface.co/datasets/zitongyang/entigraph-quality-corpus).).
For the remainder of the paper, we refer to the former as the “Raw corpus” and the latter as the “EntiGraph corpus”.
Additional details on these corpora are provided in Appendix B.

We continually pretrain Llama 3 8B (Dubey et al., ) with standard causal language modeling on the 455M token EntiGraph corpus(^3^33Model [https://huggingface.co/zitongyang/llama-3-8b-entigraph-quality](https://huggingface.co/zitongyang/llama-3-8b-entigraph-quality).).
In §4.1, we describe our continued pretraining procedure and introduce two natural baselines.
In §4.2, we evaluate all methods on the QuALITY test queries ${\mathcal{Q}}_{\text{test}}$.
In §4.3, we show that synthetic CPT using EntiGraph is compatible with downstream instruction tuning (Ouyang et al., ), an important feature of real pretraining data.

### 4.1 Continued pretraining procedure

Figure: Figure 2: Accuracy on the QuALITY question set ${\mathcal{Q}}_{\text{test}}$ ($y$-axis) as a function of the synthetic token count ($x$-axis). The accuracy of synthetic continued pretraining using the EntiGraph data augmentation algorithm (EntiGraph CPT) scales log-linearly up to 455M tokens.
Refer to caption: x2.png

#### EntiGraph CPT.

In our main continued pretraining experiment, we continually pretrain Llama 3 8B Base on the 455M token EntiGraph corpus for 2 epochs with replay on RedPajama dataset (TogetherAI, ).
For the remainder of the work, we will refer to this continually pretrained model as “EntiGraph CPT”.
We provide details on continued pretraining setup in Appendix C.
Next, we describe two baselines which we compare to EntiGraph CPT in closed-book QA (§4.2).

#### Raw CPT baseline.

The first natural baseline is to continually pretrain Llama 3 8B Base on the 1.3M token Raw corpus (the raw QuALITY articles ${\mathcal{D}}_{\text{source}}$, defined in §3).
We jointly tune the number of epochs and RedPajama replay rate, and refer to this continually pretrained model as “Raw CPT”.
Further tuning details are provided in Appendix C.

#### Rephrase CPT baseline.

Another simple synthetic data augmentation procedure is to rephrase QuALITY articles many times.
As discussed in §1.1, Maini et al. () and Ovadia et al. () execute a systematic extension of this idea.
Based on their approaches, we craft three fixed prompts (easy, medium, and hard rephrase) and repeatedly apply them to the QuALITY articles at temperature 1.0(^4^44Note that Maini et al. () also includes a fourth prompt that generates synthetic QA pairs. We defer this task-specific QA finetuning approach to Appendix D and focus on task-agnostic baselines that teach generic knowledge about QuALITY articles.).
We refer to this data augmentation algorithm as the “Rephrase baseline”.
We stopped generating paraphrases at 38M tokens, where we observed a clear gap in QA evaluations from EntiGraph CPT and a slower scaling trend (Figure 2).
We will refer to this data as the “Rephrase corpus” and the continually pretrained Llama 3 8B Base models as the “Rephrase CPT”.

### 4.2 Question-answering evaluations

Next, we provide the detailed setup of our closed-book QA evaluations with QuALITY test queries ${\mathcal{Q}}_{\text{test}}$, and present results.

#### Evaluation procedure.

Each QuALITY question is a four-choice, single-answer multiple choice question (similar to MMLU, Hendrycks et al. ()).
We evaluate with 5-shot chain-of-thought prompting (Brown et al., ; Wei et al., ) and provide our prompt in Appendix H.1.

#### EntiGraph scaling.

We find that continued pretraining on the 455M token EntiGraph corpus improves closed-book QA accuracy from 39.49% (for Llama 3 8B Base) to 56.22% (Figure 2).
A natural question is how performance scales as we synthesize and train on more tokens with EntiGraph.
To test this, we randomly subsample without replacement the EntiGraph corpus with varying sample sizes, continually pretrain Llama 3 8B Base on each subsample, and plot QuALITY accuracy with respect to sample size in Figure 2.
We observe log-linear scaling of the accuracy in the number of synthetic tokens used for continued pretraining, up to 455M tokens.
We will mathematically investigate the scaling properties of EntiGraph in detail in §6.
In broad strokes, we postulate that QuALITY accuracy follows a mixture-of-exponential shape and follows three stages: (i) linear growth, (ii) log-linear growth, and (iii) asymptotic plateau.

#### Comparison with baselines.

Raw CPT performs even worse than Llama 3 8B Base (dashed black line in Figure 2).
We postulate two reasons for this:
(i) The Raw corpus follows a narrower, different distribution than the Llama 3 8B pretraining corpus, and heavily training on these tokens may harm the overall English capabilities of the model.
(ii) The limited diversity of knowledge representations in the Raw corpus leads to limited knowledge acquisition due to problems such as the reversal curse (Berglund et al., ).
Rephrase CPT scales poorly compared with EntiGraph (Figure 2), suggesting that for synthetic CPT to scale, the synthetic data must be sufficiently diverse.
EntiGraph tackles this problem using a hierarchical prompting strategy, which externalizes diversity to the combinatorial relationships encoded in entity knowledge graphs.

### 4.3 Instruction following evaluations

**Table 2: EntiGraph Instruct examples.**
| Explicit reference: Summarize “Defining Decay Down”. |
| --- |
| The article “Defining Decay Down” by David Plotz discusses […] Dentists began to focus on cosmetic dentistry, […] |
| Implicit reference: How has dentistry in the U.S. changed? |
| 1. Increase in cosmetic dentistry […]<br>2. Use of technology: […] |
| Cross article instruction: Compare David Plotz’s commentary on American dentistry and the movie Fight Club? |
| David Plotz’s commentary style is different when he analyzes American dentistry and when he discusses the movie Fight Club. […] |

In this section, we explore more general test queries beyond the QuALITY test queries ${\mathcal{Q}}_{\text{test}}$.
Concretely, we perform instruction tuning on EntiGraph CPT to obtain EntiGraph Instruct.
We demonstrate that synthetic CPT on the EntiGraph corpus is compatible with instruction tuning: EntiGraph Instruct can directly use knowledge obtained during synthetic CPT in instruction following tasks (Wei et al., ), without any test-time access to the QuALITY books and articles ${\mathcal{D}}_{\text{source}}$.
We provide details about our instruction tuning procedure in Appendix C.

#### Instruction tuning qualitative examples.

We first present a few qualitative examples to demonstrate EntiGraph Instruct’s ability to follow instructions related to QuALITY articles.
As a first test, we ask the model to summarize a QuALITY article given an explicit reference to the title and author, but no access to the article itself (Table 2, top row).
This article provides context for the coming examples.
Next, we show that even without an explicit reference to the title and author, knowledge of the article is stored in the model’s parameters and can affect its behavior (Table 2, middle row).
Finally, we provide an example where the model performs a comparison using knowledge across two articles (Table 2, bottom row).
Albeit artificial, this shows that even though EntiGraph does not synthesize data that simultaneously involves multiple articles, the model can reason about their interaction using its parametric knowledge.
We provide the full responses in Table 5.

#### Evaluation metric for closed-book summarization.

We also present quantitative metrics for summarization, a well-studied instruction following task.
We compare EntiGraph Instruct summaries of QuALITY articles with human-written summaries from sQuALITY (Wang et al., ), a variation of QuALITY with provided human summaries.
Common scalar summarization metrics such as ROUGE (Lin, ) or BERTScore (Zhang* et al., ) mostly evaluate text similarity between the summary and source articles, and may not accurately reflect summarization quality for abstractive systems (Zhang et al., ).

We use a simple, automated evaluation metric based on pyramid evaluation (Nenkova et al., ; Gao et al., ) that measures both the hallucination rate and how well the summary captures the salient claims of the original article.
Our approach uses GPT-4 to (1) split the summary into atomic claims (Min et al., ), (2) decide whether each claim is true/false based on the source article, and (3) determine if true claims are salient to the article’s main message.
We hence obtain the count of false and salient claims for each summary, normalize these by the corresponding count from the human summary, and report the average of these normalized metrics in Figure 3.
Appendix H.2 provides further details.

#### Results discussion.

Figure: Figure 3: Closed-book summarization: number of false claims ($y$-axis) versus number of salient claims ($x$-axis) normalized by the human summary.
Refer to caption: x3.png

In Figure 3, we compare four summarizers: EntiGraph Instruct, Raw Instruct, GPT-3.5, and GPT-4.
We provide each summarizer with two different prompts—asking for progressively more detailed summaries.
We provide exact prompts in Appendix H.2, as well as a smaller-scale token-matched comparison to Rephrase CPT in Appendix H.3, where we find EntiGraph CPT has consistently lower false claims relative to Rephrase CPT.
As we request more detailed summaries, Raw Instruct consistently hallucinates and generates more false claims with little improvement in the number of salient claims.
In contrast, EntiGraph Instruct can generate more salient claims as the summary gets longer, with a small increase in the number of false claims (similar to GPT-3.5 and GPT-4 levels).
The gaps in both salient and false claim rates are sufficiently large that these results likely hold beyond our particular metric.
We complement the automated evaluation metrics above with several qualitative examples in Appendix H.2.

## 5 Open-book experiments

Next, we consider an open-book setting with the domain-specific corpus ${\mathcal{D}}_{\text{source}}$ available at test time.
In this widespread setting, retrieval-augmented generation (RAG; Lewis et al. (); Gao et al. ()) is the predominant approach.
It has strong tooling (Chase, ; Han et al., ; Pinecone, ), avoids finetuning, supports continual learning as the corpus is updated (Wu et al., ), and has high recall (proportion of queries for which the correct documents are retrieved).

Therefore, it is a natural question whether the parametric knowledge learned through synthetic CPT using EntiGraph complements the non-parametric knowledge accessed using RAG.
We answer this question by comparing a state-of-the-art RAG pipeline with and without Entigraph CPT.

#### RAG evaluation setup.

Our RAG pipeline follows established best practices (Lewis et al., ; Gao et al., ).
It involves an offline stage which indexes document chunks, followed by inference-time retrieval, reranking, and placement of those chunks in a few-shot LM prompt.
Throughout, we use OpenAI text-embedding-3-large (Neelakantan et al., ) as our API-based embedding model, FAISS as our similarity search index (Douze et al., ), and Cohere rerank-english-v3.0 (Cohere, ) as our reranker.
Following the evaluation procedure detailed in §4, we evaluate parallel RAG pipelines on the QuALITY multiple choice test set using few-shot chain-of-thought prompting.
All hyperparameters are tuned separately for each LM’s RAG pipeline.
We refer the reader to Appendix E for further details on our RAG evaluation setup.

**Table 3: QuALITY question-answering accuracy and recall rate in the open-book retrieval-augmented generation (RAG) setting. EntiGraph CPT and Llama 3 8B Base are used in a RAG pipeline (cf. §5 for setup details). Recall@$8$ is defined as the proportion of questions for which the salient article appears in the top $8$ reranked document chunks. GPT-4 and GPT-3.5 Oracle RAG provide an upper bound with a perfect retriever, by placing the entire relevant document in-context.**
| EntiGraph CPT + RAG | Llama 3 8B Base + RAG | GPT-4 + Oracle RAG | GPT-3.5 + Oracle RAG |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Accuracy | Recall@$8$ | Accuracy | Recall@$8$ | Accuracy | Recall@$8$ | Accuracy | Recall@$8$ |
| 62.60 | 99.63 | 60.35 | 99.63 | 86.09 | 100.0 | 72.60 | 100.0 |

#### EntiGraph continued pretraining complements RAG.

We observe in Table 3 that EntiGraph CPT outperforms Llama 3 8B Base, the model from which it is continually pretrained.
These results demonstrate that the knowledge internalized through synthetic CPT is complementary to that accessed during RAG, and demonstrate a competitive new recipe for small corpus QA: (1) synthetic data augmentation, (2) continued pretraining, and (3) RAG.

#### EntiGraph continued pretraining alone approaches RAG performance.

These results also contextualize the effectiveness of EntiGraph in the closed-book, parametric knowledge setting (§4).
Comparing Figure 2 and Table 3, we observe that adding RAG to Llama 3 8B Base improves accuracy by $20.86\%$ ($39.49\%\rightarrow 60.35\%$).
On the other hand, continued pretraining of Llama 3 8B Base on the EntiGraph corpus improves accuracy by $16.73\%$ ($39.49\%\rightarrow 56.22\%$).
Hence, EntiGraph continued pretraining provides $>\!80\%$ of the absolute performance improvement of RAG, even in a small corpus setting where RAG recall is nearly perfect.

Overall, our results indicate that the parametric knowledge acquired in EntiGraph continued pretraining composes with realistic knowledge-intensive QA pipelines, and that EntiGraph continued pretraining alone—without test-time corpus access—is nearly competitive with a strong RAG baseline.

## 6 Theoretical analysis of EntiGraph scaling

It may seem surprising that simply “rewriting” the factual content of the source documents ${\mathcal{D}}_{\text{source}}$ can improve performance at all (§4), as the EntiGraph data augmentation algorithm does not explicitly add new factual information beyond ${\mathcal{D}}_{\text{source}}$.
In this section, we build a mathematical model based on a stochastic process on graphs to offer an explanation for this phenomenon.
We postulate that EntiGraph does not create knowledge *de novo*; rather, it simply “rearranges” the knowledge of ${\mathcal{D}}_{\text{source}}$ into a layout more amenable to learning.
For example, in ${\mathcal{D}}_{\text{source}}$, the entity pair $(A,B)$ may appear together in some sentences and $(B,C)$ in others.
As a result, models trained directly on ${\mathcal{D}}_{\text{source}}$ with a next-token prediction objective may learn the $(A,B)$ relation and the $(B,C)$ relation, but not the relation between $A$ and $C$ (Akyürek et al., ).
We will build a mathematical model that formalizes this intuition (§6.1).
Based on this model, we provide a quantitative prediction that the scaling trend of EntiGraph CPT follows a mixture-of-exponential shape (§6.3), which fits well with our empirically observed scaling trend (Figure 4).

### 6.1 Toy model setup

In this toy model, we use ${\mathcal{V}}$ to denote the set of entities, and represent the source documents ${\mathcal{D}}_{\text{source}}$ with pairs of known relations ${\mathcal{D}}_{\text{source}}\subset\{(x,y)\in{\mathcal{V}}^{2}:x\neq y\}$.
We assume that each relation pair in ${\mathcal{V}}^{2}$ appears in the source documents ${\mathcal{D}}_{\text{source}}$ independently at random, with probability $p$.
Mathematically, $\mathbb{P}\left[(x,y)\in{\mathcal{D}}_{\text{source}}\right]=p$ for all $x\in{\mathcal{V}}$ and $y\in{\mathcal{V}}$ with $x\neq y$.
We write $V=|{\mathcal{V}}|$ and assume that $p=\lambda/V$, for some constant $\lambda>1$.

#### Training as memorization.

We model the learning of factual knowledge as a memorization process, in which a model memorizes the relations it is explicitly trained on but does not meaningfully generalize beyond them (Yang et al., ; Feldman, ).
In our knowledge graph setting, a language model’s knowledge can be represented by a matrix ${\bm{M}}\in\{0,1\}^{V\times V}$ such that ${\bm{M}}(x,y)=1$ if the model “knows” the $(x,y)$ relation and equals 0 0 otherwise.
Then, training directly on the source documents ${\mathcal{D}}_{\text{source}}$ simply means setting all entries that appear in ${\mathcal{D}}_{\text{source}}$ to $1$.
This denotes that the model has memorized the relations given in the source documents.
Mathematically, we denote this model trained on ${\mathcal{D}}_{\text{source}}$ by the matrix ${\bm{M}}_{0}\in\{0,1\}^{V\times V}$, which has i.i.d. Bernoulli off-diagonal entries with mean $p$.

#### EntiGraph synthetic data augmentation.

Given the source documents ${\mathcal{D}}_{\text{source}}$, we define the following iterative procedure of synthetic data generation: for each $t=1,2,\dots$

- 1.
Entity pair selection: Sample $(x_{t},y_{t})\in\{(x,y)\in{\mathcal{V}}^{2}:x\neq y\}$ uniformly at random.
- 2.
Relation analysis: Generate the “relation between $(x_{t},y_{t})$” by performing a breadth-first search (BFS) on the directed graph represented by the adjacency matrix ${\bm{M}}_{0}$ starting at $x_{t}$:
•
If there exists a path $(x_{t},z_{t}^{1},z_{t}^{2},\dots,z_{t}^{k_{t}},y_{t})$ connecting $x_{t}$ to $y_{t}$, define
${\mathcal{D}}_{t}=\{(x_{t},z_{t}^{1}),(x_{t},z_{t}^{2}),\dots,(x_{t},z_{t}^{k_
{t}}),(x_{t},y_{t})\}\cup{\mathcal{D}}_{t-1},$
where we assume ${\mathcal{D}}_{0}={\mathcal{D}}_{\text{source}}$. The model trained on this round of synthetic data would be
${\bm{M}}_{t}={\bm{M}}_{t-1}+\sum_{(x,y)\in{\mathcal{D}}_{t}\backslash{\mathcal
{D}}_{t-1}}{\bm{I}}_{xy},$
where ${\bm{I}}_{xy}\in\{0,1\}^{V\times V}$ is a binary matrix with ${\bm{I}}_{xy}(x,y)=1$ and 0 0 otherwise.
•
If no such path exists, do nothing.

This mirrors the relation analysis step for the EntiGraph synthetic data augmentation algorithm (introduced in §2.2).
With the setup above, the index $t$ is analogous to the number of synthetic tokens that the model has generated, and the model’s knowledge is captured by how many ones the matrix ${\bm{M}}_{t}$ contains.
To make this connection precise, we define the link density (or accuracy) of ${\bm{M}}_{t}$ to be

$$ $\mathsf{Acc}({\bm{M}}_{t})=\frac{\mathbb{E}[\|{\bm{M}}_{t}\|_{1}|{\bm{M}}_{0}] }{V(V-1)},$ $$

where the expectation is taken over the randomness arising from the synthetic data generation process and not the source documents ${\mathcal{D}}_{\text{source}}$. For a matrix $M$, we use $\|M\|_{1}$ to denote $\sum_{i,j}|M_{i,j}|$. We use the notation $\mathsf{Acc}$ as this is intended to emulate the accuracy on QuALITY test queries studied in the experimental sections (§4 and §5).

### 6.2 Rigorous upper and lower bound

In this section, we derive rigorous upper and lower bounds on the scaling trend of $\mathsf{Acc}({\bm{M}}_{t})$. We show that $\mathsf{Acc}({\bm{M}}_{t})$ as a function of $t$ can be bounded above and below by two exponential functions with different growth rates.
Note that these two bounds do not necessarily imply that $\mathsf{Acc}({\bm{M}}_{t})$ itself grows exponentially. We will provide a precise formula for its growth in §6.3 via an approximation through a Poisson branching process.

###### Definition 1 .

Let $C_{\lambda}=(1-\rho(\lambda))^{2}$, where $\rho(\lambda)$ denotes the extinction probability for a Poisson$(\lambda)$ branching process (i.e., $\rho$ is the smallest solution in $[0,1]$ to the fixed-point equation $\rho=\exp(\lambda(\rho-1))$).
For any fixed $\varepsilon>0$, we further define

$$ $\displaystyle C_{\mathrm{LB}}=1-\frac{1}{V(V-1)},\quad C_{\mathrm{UB}}=1-\frac {(1+\varepsilon)\log V}{V(V-1)\log\lambda}.$ $$

###### Theorem 1 .

For any time $t\geq 1$ and any $\varepsilon>0$, the link density satisfies

$$ $\displaystyle\left(p+C_{\lambda}\left(1-C_{\mathrm{LB}}^{t}\right)\right)(1- \varepsilon)\leq\mathsf{Acc}({\bm{M}}_{t})\leq\left(p+C_{\lambda}\left(1-C_{ \mathrm{UB}}^{t}\right)\right)(1+\varepsilon),$ $$

with probability $\to 1$ when $V\to\infty$.

Even though Theorem 1 provides mathematically rigorous upper and lower bounds on the scaling trend of $\mathsf{Acc}({\bm{M}}_{t})$, the exact growth curve is more intricate, as we will show next.

###### Definition 1 .

###### Theorem 1 .

### 6.3 An analytical formula

For the remainder of the section, we analyze the link density $\mathsf{Acc}({\bm{M}}_{t})$ using a Poisson branching process approximation of the cluster growth of vertices.
This approach yields an approximation of the form

$$ $\displaystyle\mathsf{Acc}({\bm{M}}_{t})\sim p+C_{\lambda}\left(1-\sum_{\ell=0} ^{\infty}\frac{\lambda-1}{\lambda^{\ell+1}}\sum_{k=1}^{\infty}p_{\ell}(k)\left (1-\frac{k}{V(V-1)}\right)^{t}\right),$ $$

where $A\sim B$ means that $A/B$ converges to $1$ in probability as $V\rightarrow\infty$.
We refer the reader to Appendix F for a comprehensive derivation.
Here $p_{\ell}$ denotes the probability mass function of the total progeny $Y_{\ell}$ of a Poisson$(\lambda)$ branching process at level $\ell$.
Qualitatively, for a general representation of source documents ${\mathcal{D}}_{\text{source}}$ beyond directed Erdős-Rényi graphs, we still expect to observe a *mixture-of-exponential* scaling trend:

$$ $\mathsf{Acc}({\bm{M}}_{t})\sim p+C\left(1-\sum_{k=1}^{\infty}\mu(k)\left(1-a_{ k}\right)^{t}\right).$ (2) $$

Figure: Figure 4: A mixture-of-exponential functional form (2) closely fits the scaling trend of EntiGraph CPT with respect to synthetic token count.
Refer to caption: x4.png

In this context, the parameter $C$ governs the link density $\mathsf{Acc}({\bm{M}}_{t})$ as $t\to\infty$.
In our model, $C$ is determined by the proportion of reachable pairs of vertices in the initial matrix ${\bm{M}}_{0}$.
Here, we are essentially filling out the “deductive closure” (i.e., all the facts or relations that can be deduced from ${\mathcal{D}}_{\text{source}}$; Stine (); Akyürek et al. ()) of the original data—if some facts cannot be deduced, then $\mathsf{Acc}({\bm{M}}_{t})$ cannot approach $1$.
The measure $\mu(\cdot)$ is the probability mass function on $k$, which controls the proportion of pairs of vertices with a specific decay rate. The parameters $\mu(\cdot)$ depend on ${\bm{M}}_{0}$ in a more intricate manner.
We find that the formula in (2) accurately fits the empirical scaling trend of EntiGraph CPT accuracy up to 455M synthetic tokens (Figure 4).

#### Sketch of derivation.

Intuitively, the edge $(i,j)$ will eventually be added if and only if $j$ is reachable from $i$ in the original graph ${\bm{M}}_{0}$. This explains the limiting behavior of $\mathsf{Acc}({\bm{M}}_{t})$ as $t$ approaches infinity: the proportion of links will converge to the proportion of connected vertex pairs in ${\bm{M}}_{0}$. To understand the mixture-of-exponential functional form, consider that at the time $t$, the probability of adding each vertex pair follows an exponential pattern, with different vertex pairs exhibiting different exponential growth rates. Specifically, think of a breadth-first search in ${\bm{M}}_{0}$ starting from a vertex $i$. If $j$ is very close to the root, there are many paths from $i$ to other vertices passing through $j$, making it more likely that $(i,j)$ will be included in each iteration. In contrast, if $j$ is far from the root (e.g., at the end of the exploration process), there are fewer such paths, making it less likely for $(i,j)$ to be included in each iteration. This accounts for the mixture-of-exponential shape, where the mixture primarily reflects the distance of each vertex from the root, the number of such vertices, and their corresponding exponential growth rates.

#### Qualitative description.

Figure: (a) Linear regime
Refer to caption: extracted/5897622/figures/Acc1.png

Finally, to help build an intuitive understanding, we provide a qualitative description of the mixture-of-exponential shape.
We demonstrate in Appendix F that this mixture-of-exponential shape comprises three distinct phases: a fast growth phase, a slower growth phase, and a plateau phase. Mathematically, we show the existence of two distinct times, $0<t_{1}<t_{2}$, such that

$$ $\displaystyle\mathsf{Acc}({\bm{M}}_{T})=\begin{cases}\Theta\left(p+t\right), \quad&\text{ for }0\leq t\leq t_{1},\\ \Theta(\log t),\quad&\text{ for }t_{1}\leq t\leq t_{2},\\ \Theta(1),\quad&\text{ for }t\geq t_{2},\end{cases}$ $$

where we use a convenient change of variable $T=tV(V-1)$.
It is important to note that the choice of $\log t$ in the second phase is not necessarily canonical.
In fact, the bound holds for any well-behaved monotone increasing concave function as a replacement for $\log t$.
Our representation here is motivated by two factors: first, it aligns with the performance observed in our EntiGraph CPT numerical results, and second, it reflects the gradual slowdown in growth. We illustrate the three phases in Figure 5, which present a simulation of the toy model with $p=0.03$.

## 7 Discussion

### 7.1 Limitations

Because EntiGraph synthesizes data using a prompted language model, there is a risk that it may hallucinate and fabricate non-existent relations among the entities.
Although our process of generating synthetic data is grounded by the source documents, it is an assumption that $\mathsf{LM}_{\text{aug}}$ is capable enough to generate faithful synthetic data when conditioned on ${\mathcal{D}}_{\text{source}}$.
In our experiment with QuALITY books, we manually read a few books and fact-checked a subset of the synthetic data generated for those books; we did not find factually incorrect synthesized text.
We postulate that this is because we use a sufficiently strong prompted model $\mathsf{LM}_{\text{aug}}$ (gpt-4-turbo).
If EntiGraph were applied to more challenging content like a complex research paper, it is possible that the prompted model could be more prone to hallucination.

On the other hand, since we use a very capable prompted language model gpt-4-turbo to generate synthetic data, one might be concerned that our performance gains come from distilling the prompted LM’s knowledge.
The closed-book results indicate that distillation effects alone cannot explain the performance of our approach (as we exceed GPT-4’s closed-book performance), but our approach does not yet enable bootstrapping, where we use a model to generate its own synthetic data for a small target domain.
We view this as exciting future work.

### 7.2 Future directions

#### Continued scaling beyond real data.

The large but finite body of human-written text is rapidly being consumed.
Villalobos et al. () predict that frontier language models will exhaust all public, human-generated text in 2028.
As we transition from a data-rich to a data-constrained regime (Kaplan et al., ; Muennighoff et al., ), further scaling will require us to extract more knowledge from existing data.
We demonstrated that synthetic continued pretraining with EntiGraph effectively extracts more knowledge from small corpora, which could help us learn from proprietary datasets or tail knowledge that appears only once or twice on the internet.
It is an open question whether synthetic data generation methods like EntiGraph could improve data efficiency more generally on standard pretraining data and without relying upon a stronger prompted model.

#### Alternatives to long-context language models.

Recent work handles long user queries (e.g., 1M-10M+ tokens) using efficient implementations of attention (Dao et al., ; Liu et al., ; Gemini, ) or alternative architectures that are sub-quadratic in the context length (Tay et al., ; Gu et al., ; Gu & Dao, ; Sun et al., ).
In settings where many queries share the same long prefix—e.g., a corporation’s proprietary documents or other use cases with prompt caching (Anthropic, )—one could instead continue pretraining on the prefix to internalize its knowledge, and then perform standard quadratic attention on shorter queries.
This approach pays a fixed training cost to amortize the prefix’s knowledge into the weights of a model, and then benefits from shorter context lengths (Gururangan et al., ; Snell et al., ).
By adapting the continued pretraining paradigm from 10B-100B tokens to as little as 1.3M tokens, our synthetic continued pretraining approach could enable unsupervised learning of shared text prefixes at much smaller and more practical token counts.

### 7.3 Conclusion

Continued pretraining with next-token prediction is remarkably effective in teaching pretrained language models new knowledge, but to date has only been applied successfully in broad, data-rich domains with 10B-100B+ tokens.
We downscale continued pretraining to small, specialized corpora with $\sim$1M tokens using synthetic continued pretraining: converting a small corpus into a large synthetic one with diverse representations of knowledge, and continuing pretraining on it.

We instantiate this approach using EntiGraph, a knowledge graph–inspired synthetic data augmentation algorithm.
Synthetic continued pretraining with EntiGraph demonstrates consistent scaling in downstream closed-book QA performance up to a 455M token synthetic corpus, whereas baselines such as continued pretraining on the small corpus or synthetic paraphrases show no improvement or asymptote early.
Moreover, the acquired parametric knowledge composes with instruction tuning and retrieved non-parametric knowledge in an open-book setting.
Lastly, we present a simplified mathematical model of EntiGraph and derive a functional form for its scaling trend, which closely matches our empirical trend.
We hypothesize that EntiGraph’s “externalization” of the synthetic data generation process to a combinatorial structure—in this case, a knowledge graph over entities—is a generally useful strategy in synthesizing highly diverse data and a promising object for future study.

## 8 Acknowledgement

Zitong Yang would like to thank Samy Jelassi for feedback on a preliminary version of this work,
Ruiqi Zhong for discussion regarding context distillation work, Xiang Lisa Li for discussion about reversal curse work, and the participants of the statistics seminar at Stanford University for their insightful feedback about a preliminary version of this work.
We also thank the Tatsu Lab for constructive feedback and interesting discussions that have helped improve the paper.
Zitong Yang is supported by the Albion Walter Hewlett Stanford Graduate Fellowship.
Neil Band acknowledges funding from an NSF Graduate Research Fellowship and a Quad Fellowship.
This work was supported by gifts from Panasonic Research, the Google Research Scholar Program, and the Tianqiao and Chrissy Chen Institute, as well as the NSF grant IIS-2338866.
E.J.C. is supported by the Office of Naval Research grant N00014-20-1-2157, the National Science Foundation grant DMS-2032014, the Simons Foundation under award 814641.

## Codebase, dataset, and model weights

We provide the codebase for reproducing all results discussed in the paper below:

We release the 455M EntiGraph corpus below:

We release the EntiGraph CPT model weights below:

## Appendix A Additional related work

#### Synthetic data generation.

There is rich literature on using neural nets to generate synthetic data.
Many such approaches were originally developed for semi-supervised learning—self-training and pseudo-labeling methods improve models by iteratively training them on their own predictions (Scudder, ; Lee, ; Yalniz et al., ; Berthelot et al., ; Xie et al., ), and co-training uses two models to supervise each other (Blum & Mitchell, ; Balcan et al., ).
Before language models rose to prominence, few approaches attempted to synthesize inputs.
One exception is membership query synthesis, which explored the synthesis of inputs in a supervised learning context (Angluin, ; Schumann & Rehbein, ).

Contemporary works employ co-training (Lang et al., ) and self-training to improve language model performance, often on mathematical reasoning tasks (Huang et al., ; Gulcehre et al., ; Zhang et al., ), or synthesize input-output pairs for instruction tuning, usually by conditioning on a curated seed set (Wang et al., ; Honovich et al., ; Taori et al., ; Peng et al., ; Yuan et al., ; Li et al., ).

#### Continual learning and pretraining.

Continual learning is rooted in historical work on connectionist networks (McCloskey & Cohen, ; Ratcliff, ) and considers learning with tasks arriving in an online manner (Schlimmer & Fisher, ; Grossberg, ).
The main focus is on mitigating a neural net’s “catastrophic forgetting” of previously encountered tasks (Robins, ; Goodfellow et al., ; Kemker et al., ).
Approaches include regularizing parameter updates to preserve important parameters (Nguyen et al., ; Zenke et al., ; Kirkpatrick et al., ); dynamically modifying the architecture (Rusu et al., ; Golkar et al., ); and recalling or replaying previous experiences (Rebuffi et al., ; Shin et al., ; Lopez-Paz & Ranzato, ).
Modern works in continued pretraining (cf. §1.1) effectively mitigate catastrophic forgetting by scaling parameter count (Ramasesh et al., ) and mixing in updates on pretraining data (Ouyang et al., ).

## Appendix B Details on the QuALITY dataset

We provide additional details on the QuALITY dataset below.
For each book, we execute entity extraction (Step 1, §2.2) and then analyze all pair-wise relations between entities and a subset of all triplet relations (Step 2, 2.2).
We provide summary statistics for the Raw and EntiGraph corpora in Figure 6.

Figure: (a) Raw article tokens
Refer to caption: x5.png

## Appendix C Training details for the main experiments

#### Continued pretraining details.

In all experiments, we continue pretraining the Llama 3 8B Base model with a context length of 2048 and batch size of 16.
We apply a linear learning rate warmup for 5% of total steps, followed by a cosine decay with peak learning rate 5e-6.
We perform full parameter training with Fully Sharded Data Parallelism (FSDP, Zhao et al. ()).

#### EntiGraph continued pretraining details.

To mitigate the forgetting of pretrained knowledge, we perform replay with a rate of 0.1 using 1B RedPajama tokens (TogetherAI, ).
More precisely, for each training batch, we flip a biased coin such that with 10% probability, we load the RedPajama data instead of the EntiGraph synthetic data.

#### Raw continued pretraining details.

Next, we provide details for our continued pretraining directly on the Raw corpus, producing the “Raw CPT” model.
Because the Raw corpus only has 1.3M tokens, we jointly tune the number of epochs (repetition factor) and the RedPajama replay rate on accuracy over a QuALITY QA validation split.
The selected hyperparameter configuration uses 4 epochs and a 0.1 replay rate.

#### Instruction tuning details.

We use the UltraChat instruction tuning dataset (Ding et al., ) filtered by the Huggingface team (Tunstall et al., ) as our instruction tuning data.
We use the chat template of Llama 3.1 8B Instruct (Dubey et al., ) to format the UltraChat conversations, obtaining a 250M token instruction tuning dataset.
We apply a linear learning rate warmup followed by a cosine decay to 0 with peak learning rate 5e-6, and train the model for 1 epoch with a batch size of 512 and context window of 2048.
To sanity check our instruction tuning procedure, we measure the AlpacaEval (Li et al., ) winrate against GPT-4 and find it improves from 0% to 6.25%, comparable to a 7.7% baseline winrate of Llama 2 Chat 13B.

#### Compute resource.

All the continued pretraining experiments are performed with one $8\times$H100 node.
With PyTorch FSDP Zhao et al. (), we obtain throughput of 6090 tokens per second.
Since all experiments use the same model architecture, batch size, and context length, the time to run the experiments can be calculated based on the total tokens seen during training.
For example, the main EntiGraph is trained on 455M tokens with 2 epochs.
Therefore, it should take $455$M$\times 2/6090$ seconds, which is about 41 hours.

## Appendix D Task-specific finetuning for QuALITY Question set

Our work considers *task-agnostic* synthetic data generation and continued pretraining as a way to obtain generalizable knowledge about a domain, in a way that can later be extracted via few-shot prompting (Brown et al., ) and instruction tuning (Ouyang et al., ).

However, if our goal is only to do well on a single task, such as question answering, then we could fine-tune a language model for that particular task. This approach worked extremely well on tasks such as SQuAD (Rajpurkar et al., ) in-domain but suffered from degraded performance outside the fine-tuning data distribution Awadalla et al. ().

We do not extensively perform comparisons to task-specific finetuning due to the more general multi-task goals of EntiGraph, we run preliminary experiments comparing a simple QA SFT baseline to EntiGraph, and find that EntiGraph scaling and synthetic data generation costs are generally favorable even when compared to this strong, task-specific baseline.

#### QA SFT.

We follow the same set as in §2.1 and §3 except that we do not prompt $\mathsf{LM}_{\text{synth}}$ to generate general knowledge about QuALTY articles.
Instead, we prompt $\mathsf{LM}_{\text{synth}}$ to generate QA pairs directly:

We repeat this prompt many times at temperature 1.0, resulting in 28M tokens on synthetic question answer pairs.
We perform the same continued pretraining procedure in §4.1 on Llama 3 8B and refer to this model as “QA SFT”.

Figure: Figure 7: Accuracy on the QuALITY question set ${\mathcal{Q}}_{\text{test}}$ ($y$-axis) as a function of the synthetic token count ($x$-axis). Comparison among EntiGraph CPT, Rephrase CPT, and QA SFT.
Refer to caption: x8.png

#### Results discussion

We plot the QA SFT scaling curve in Figure 7.
We can see that task-specific finetuning demonstrates a very sharp improvement in QA accuracy, consistent with prior results showing task-specific finetuning gains for pretrained models. While QA SFT performance is high, we note that EntiGraph attains similar performance despite being entirely task-agnostic, and the overall dollar cost of creating the dataset is much lower for EntiGraph.

This difference in synthetic data generation cost is hidden in Figure 7, as we plot the number of training tokens rather than dollars spent to generate the synthetic data. For QA SFT, each QA question is generally short, resulting in large inefficiencies in generating this QA dataset. We found that the input token to output token ratio was large compared with Rephrase CPT and EntiGraph CPT, resulting in over  $5k to generate just 28M tokens (^5^55OpenAI API pricing, Sep 2024). This difference in cost means that further scaling became prohibitively expensive, and that EntiGraphs’s performance in Figure 7 is even better than it appears, if we match for total cost rather than token budget.

## Appendix E Additional details on open-book experiments

We provide additional details on our open-book experimental setup below, including our retrieval-augmented generation (RAG, Lewis et al. (); Gao et al. ()) pipeline.
As mentioned in §5, we use a standard two-stage RAG pipeline: first, an offline stage which indexes document chunks; second, inference-time retrieval, reranking, and placement of those chunks in a few-shot LM prompt.

### E.1 Stage 1: offline indexing

The purpose of the indexing stage is to construct an index over all the 265 articles and books from the QuALITY corpus ${\mathcal{D}}_{\text{source}}$.
More specifically, this stage chunks documents from the given corpus, obtains dense vector embeddings for each chunk using an API-based embedding model, and indexes the (embedding, chunk) pairs.

#### Chunking documents.

We first split each document $D^{(i)}\in\{D^{(i)}\}_{i=1}^{n}={\mathcal{D}}_{\text{source}}$ into a set of $m_{i}$ document chunks $\{C^{(i)}_{1},...,C^{(i)}_{m_{i}}\}$.
To perform this splitting, we use the Recursive
CharacterTextSplitter from Chase (), which attempts to keep all paragraphs (and then sentences, and then words) together for as long as possible, in order to preserve the semantics within each chunk.
We use non-overlapping chunks and tune chunk size in characters (chunk_size, hyperparameter values provided below).
Lastly, because we have access to metadata about each document $D^{(i)}$—namely, the title, author, and year of the book or article—we prepend this metadata to each document chunk.
This is analogous to how a corporation building a RAG system over their own document store could include metadata about the document (title, author, year, etc.).
These final chunks with metadata prepended are embedded, and are the ones that are retrieved and placed in-context.

#### Embedding and indexing document chunks.

Next, we obtain dense embeddings for all document chunks using a state-of-the-art text embedding model OpenAI text-embedding
-3-large (Neelakantan et al., ).
Lastly, we index all (embedding, chunk) tuples using a FAISS vector store (Douze et al., ).

### E.2 Stage 2: inference-time retrieval and reranking

At inference time, the RAG system receives a test query $q\in{\mathcal{Q}}_{\text{test}}$.
Each query $q$ is contextualized with the article title and author name, as described in §3, and contains its four possible answer choices (QuALITY is a 4-choice, multiple choice dataset).
In Stage 2, we embed the query with the API-based embedding model, retrieve $K$ document chunks using an approximate nearest-neighbor search, and lastly, select the $k<K$ most relevant chunks using an API-based reranker.

#### Retrieving top- K 𝐾 K italic_K document chunks.

We embed $q$ with text-embedding-3-large, and retrieve the top-$K$ most relevant document chunks from our indexed vector store using FAISS similarity search with a Euclidean distance metric.

#### Reranking to obtain top- k 𝑘 k italic_k ( k < K 𝑘 𝐾 k<K italic_k < italic_K ) chunks.

Next, we use a reranker to filter the $K$ retrieved document chunks to a smaller number of reranked chunks $k$.
Rerankers are known to significantly improve recall (the proportion of the time that the salient article is contained in the top chunks), and indeed, the recall of our RAG pipelines is near-perfect (Table 3 in §5).
Specifically, we pass the query $q$ and the list of $K$ retrieved document chunks to a state-of-the-art reranker—Cohere rerank-english-v3.0 (Cohere, )—which returns a list of the $K$ chunks in order from most to least semantically relevant for the query.
We take the $k$ highest scoring chunks and place them in our few-shot prompt.

#### Few-shot prompt formatting.

Our full few-shot chain-of-thought evaluation prompts for the open-book setting are provided in the codebase.
Similar to the closed-book QA evaluation prompt, we manually write and fact-check in-context learning examples about well-known books, to avoid leaking knowledge from the QuALITY articles.
In early experiments, we found that placing the retrieved contexts first, followed by the question and answer choices after, significantly improved performance compared to question-then-contexts; we use this format throughout the retrieval experiments.
We treat as a hyperparameter whether the reranked chunks are ordered from the best match to worst (best_first) or from the worst match to best (best_last).
When performing few-shot evaluation, we follow the sampling procedure used in the closed-book experiments (Appendix H.1).
Specifically, we generate 64 responses for each question, and filter out responses that do not parse to one of the four choices.
Lastly, we randomly select one of the valid responses as the model’s final answer.

### E.3 Hyperparameter tuning

In our experiments, we compare two LMs used in the RAG pipeline above: EntiGraph CPT and its base model, Llama 3 8B Base.
As mentioned above, we fix the retrieved number of chunks to $K=128$, but vary the number of reranked chunks $k$ which are ultimately placed in the context window.
For each language model + RAG pipeline, we independently tune the following hyperparameters with a grid search on accuracy using a QuALITY QA validation split:

- •
Document $\texttt{chunk_size}\in\{256,512,1024\}$
- •
Rerank top-$k\in\{1,2,4,8,16\}$
- •
Order of chunks $\in\{\texttt{best_first},\texttt{best_last}\}$
- •
Eval temperature $\in\{0.1,0.3,0.5,0.7\}$

We refer the reader to our codebase for tuned hyperparameters.

## Appendix F Proof of Theorem 1 and other analytical formulas

In this section, we prove Theorem 1 and provide the derivations for several other approximation formulas.

###### Proof of Theorem 1 .

Fix the matrix ${\bm{M}}_{0}$, we observe that

$$ $\displaystyle\mathsf{Acc}({\bm{M}}_{t})=\frac{\mathbb{E}[\|{\bm{M}}_{t}\|_{1}| {\bm{M}}_{0}]}{V(V-1)}=\sum_{(i,j)\in{\mathcal{V}}^{2}}\frac{\mathbb{E}[ \mathbbm{1}((i,j)\in{\mathcal{D}}_{t})|{\bm{M}}_{0}]}{V(V-1)}=\sum_{(i,j)\in{ \mathcal{V}}^{2}}\frac{\mathbb{P}[(i,j)\in{\mathcal{D}}_{t}|{\bm{M}}_{0}]}{V(V -1)}.$ $$

For each $(i,j)\in{\mathcal{V}}^{2}$, we define $q_{i,j}$ to be the probability that $(i,j)$ is included in the set $\{(x_{t},z_{t}^{1}),(x_{t},z_{t}^{2}),\dots,(x_{t},z_{t}^{k_{t}}),(x_{t},y_{t})\}$. Note that each iteration of the procedure generates a path $(x_{t},z_{t}^{1},z_{t}^{2},\dots,z_{t}^{k_{t}},y_{t})$ independently identically. So naturally $q_{i,j}$ does not depend on the time $t$. This implies that $\mathbb{P}[(i,j)\in{\mathcal{D}}_{t}|{\bm{M}}_{0}]=1-(1-q_{i,j})^{t}$. Thus we can further rewrite the link density as

$$ $\displaystyle\mathsf{Acc}({\bm{M}}_{t})$ $\displaystyle=\frac{|{\mathcal{D}}_{\text{source}}|}{V(V-1)}+\sum_{(i,j)\in{ \mathcal{V}}^{2}\backslash{\mathcal{D}}_{\text{source}}}\frac{\mathbb{P}[(i,j) \in{\mathcal{D}}_{t}|{\bm{M}}_{0}]}{V(V-1)}$ $\displaystyle=\frac{|{\mathcal{D}}_{\text{source}}|}{V(V-1)}+\sum_{(i,j)\in{ \mathcal{V}}^{2}\backslash{\mathcal{D}}_{\text{source}}}\frac{1-(1-q_{i,j})^{t }}{V(V-1)}.$ $$

The remaining task is to estimate $q_{i,j}$. We say a vertex $j$ is reachable from $i$ and denote $i\sim j$, if there is a directed path from $i$ to $j$ in ${\bm{M}}_{0}$. We define ${\mathcal{R}}=\{(u,v)\in{\mathcal{V}}^{2}:u\neq v,u\sim v\}$ to be the set of all reachable pairs of vertices in ${\mathcal{V}}$. We note that $q_{i,j}$ is non-zero if and only if $j$ is reachable from $i$ in ${\bm{M}}_{0}$. Now, for any $t\geq 1$, the function $1-(1-x)^{t}$ is concave, thus by Jensen’s inequality, we have

$$ $\displaystyle\sum_{(i,j)\in{\mathcal{V}}^{2}\backslash{\mathcal{D}}_{\text{ source}}}1-(1-q_{i,j})^{t}\leq\sum_{(i,j)\in{\mathcal{R}}}1-(1-q_{i,j})^{t} \leq|{\mathcal{R}}|\left(1-(1-\bar{q}_{i,j})^{t}\right),$ $$

where

$$ $\displaystyle\bar{q}_{i,j}=\frac{\sum_{(i,j)\in{\mathcal{R}}}q_{i,j}}{|{ \mathcal{R}}|}.$ $$

For each $(i,j)\in{\mathcal{R}}$, the probability $q_{i,j}$ satisfies

$$ $\displaystyle q_{i,j}=\frac{\sum_{a\neq b\in{\mathcal{V}}^{2}}\mathbbm{1}((i,j )\in\{(a,z^{1}),(a,z^{2}),\dots,(a,z^{k}),(a,b)\})}{V(V-1)}$ $$

where $(a,z^{1},z^{1},\cdots,z^{k},b)$ is the shortest path in ${\bm{M}}_{0}$ connecting $a$ and $b$. If there is no such path, then by default the indicator equals zero. Now we look at

$$ $\displaystyle\sum_{(i,j)\in{\mathcal{R}}}q_{i,j}$ $\displaystyle=\frac{1}{V(V-1)}\sum_{(i,j)\in{\mathcal{R}}}\sum_{(a,b)\in{ \mathcal{R}}}\mathbbm{1}((i,j)\in\{(a,z^{1}),(a,z^{2}),\dots,(a,z^{k}),(a,b)\})$ $\displaystyle\leq\frac{1}{V(V-1)}\sum_{(a,b)\in{\mathcal{R}}}\sum_{i\neq j\in{ \mathcal{V}}^{2}}\mathbbm{1}((i,j)\in\{(a,z^{1}),(a,z^{2}),\dots,(a,z^{k}),(a, b)\})$ $\displaystyle=\frac{1}{V(V-1)}\sum_{(a,b)\in{\mathcal{R}}}\ell_{a,b},$ $$

where $\ell_{a,b}$ is the length of the shortest path connecting $a$ to $b$. To analyze the typical shortest length of paths, we present a few classical results on directed ErdHos-Rényi graphs. For any $a\in{\mathcal{V}}$, let $X(a)$ denote the set of vertices reachable from $a$ and let $Y(a)$ denote the set of vertices from which $a$ is reachable. Recall that $\rho(\lambda)$ is the extinction probability for the Poisson$(\lambda)$ branching process.

###### Lemma F.1 (Lemma 1 and Corollary 1 in Karp ( 1990 ) ) .

For each vertex $a$, with probability tending to $1$ as $V$ tends to infinity, there exists a constant $\beta>0$ such that either $|X(a)|\leq\beta\log V$ or $|X(a)|=(1-\rho(\lambda))V+\Theta(\sqrt{V})$. Moreover, the probability that the latter happens tends to $1-\rho(\lambda)$ as $V$ tends to infinity. The same is true for $Y(a)$.

For each vertex $a$, the set $X(a)$ is said to be small if $|X(a)|\leq\beta\log V$ (in such case we write $a\in{\mathcal{S}}_{X}$) and large if $|X(a)|=(1-\rho(\lambda))V+\Theta(\sqrt{V})$ (we write $a\in{\mathcal{L}}_{X}$). We define ${\mathcal{S}}_{Y}$ and ${\mathcal{L}}_{Y}$ similarly.

###### Lemma F.2 (Theorem 3 in Karp ( 1990 ) and Theorem 2.4.1 in Durrett ( 2010 ) ) .

With probability tending to $1$, the following statement holds for all $a$ and $b$ in ${\mathcal{V}}$: if $X(a)$ is large and $Y(b)$ is large, then $b$ is reachable from $a$. Moreover, if $X(a)$ is large and $Y(b)$ is large, then for any $\varepsilon>0$ and any sufficiently small $\delta>0$,

$$ $\displaystyle\mathbb{P}[\ell_{a,b}>(1+\varepsilon)\log V/\log\lambda]<\exp(-V^ {\varepsilon}\delta).$ $$

With Lemma F.1 and Lemma F.2, we can now give useful estimates of $|{\mathcal{R}}|$.
In particular, for any $\varepsilon>0$,

$$ $\displaystyle|{\mathcal{R}}|$ $\displaystyle=|\{(a,b)\in{\mathcal{R}}:a\in{\mathcal{L}}_{X},b\in{\mathcal{L}} _{Y}\}|+|\{(a,b)\in{\mathcal{R}}:a\in{\mathcal{S}}_{X}\text{ or }b\in{\mathcal {S}}_{Y}\}|$ $\displaystyle\leq(1-\rho(\lambda))^{2}(1+\varepsilon/4)V^{2}+2(1+\varepsilon)V \beta\log V$ $\displaystyle\leq(1-\rho(\lambda))^{2}(1+\varepsilon/3)V(V-1),$ $$

with high probability. Similarly, for the lower bound,

$$ $\displaystyle|{\mathcal{R}}|$ $\displaystyle=|\{(a,b)\in{\mathcal{R}}:a\in{\mathcal{L}}_{X},b\in{\mathcal{L}} _{Y}\}|+|\{(a,b)\in{\mathcal{R}}:a\in{\mathcal{S}}_{X}\text{ or }b\in{\mathcal {S}}_{Y}\}|$ $\displaystyle\geq(1-\rho(\lambda))^{2}(1-\varepsilon)V^{2}$ $\displaystyle\geq(1-\rho(\lambda))^{2}(1-\varepsilon)V(V-1),$ $$

with high probability. By a union bound over all pairs of $(a,b)\in{\mathcal{R}}$, we also have that

$$ $\displaystyle\sum_{(i,j)\in{\mathcal{R}}}q_{i,j}$ $\displaystyle\leq\frac{1}{V(V-1)}\sum_{(a,b)\in{\mathcal{R}}}\ell_{a,b}$ $\displaystyle=\frac{1}{V(V-1)}\sum_{\begin{subarray}{c}(a,b)\in{\mathcal{R}}\\ a\in{\mathcal{L}}_{X},b\in{\mathcal{L}}_{Y}\end{subarray}}\ell_{a,b}+\frac{1}{ V(V-1)}\sum_{\begin{subarray}{c}(a,b)\in{\mathcal{R}}\\ a\in{\mathcal{S}}_{X}\text{ or }b\in{\mathcal{S}}_{Y}\end{subarray}}\ell_{a,b}$ $\displaystyle\leq(1-\rho(\lambda))^{2}(1+\varepsilon/2)\frac{\log V}{\log \lambda}+\frac{1}{V(V-1)}2(1+\varepsilon)V(\beta\log V)^{2}$ $\displaystyle\leq(1-\rho(\lambda))^{2}(1+\varepsilon)\frac{\log V}{\log\lambda},$ $$

with probability larger than $1-V^{2}\exp(-V^{\varepsilon}\delta)$. Combining the above, for any $\varepsilon>0$,

$$ $\displaystyle\bar{q}_{i,j}=\frac{\sum_{(i,j)\in{\mathcal{R}}}q_{i,j}}{|{ \mathcal{R}}|}\leq\frac{(1+\varepsilon)\log V}{V(V-1)\log\lambda},$ $$

with high probability. Therefore, for any $\varepsilon>0$,

$$ $\displaystyle\mathsf{Acc}({\bm{M}}_{t})$ $\displaystyle\leq\frac{|{\mathcal{D}}_{\text{source}}|}{V(V-1)}+\frac{|{ \mathcal{R}}|\left(1-(1-\bar{q}_{i,j})^{t}\right)}{V(V-1)}$ $\displaystyle\leq(1+\varepsilon)\left(p+(1-\rho(\lambda))^{2}\left(1-\left(1- \frac{(1+\varepsilon)\log V}{V(V-1)\log\lambda}\right)^{t}\right)\right),$ $$

with high probability, which completes the proof of the upper bound. For the lower bound, we observe that if $i\sim j$ and $(i,j)\in{\mathcal{R}}\backslash{\mathcal{D}}_{\text{source}}$, then $q_{i,j}\geq 1/V(V-1)$, because when $i$ and $j$ are chosen in the procedure, the edge $(i,j)$ will be added. This implies that

$$ $\displaystyle\mathsf{Acc}({\bm{M}}_{t})$ $\displaystyle=\frac{|{\mathcal{D}}_{\text{source}}|}{V(V-1)}+\sum_{{\mathcal{R }}\backslash{\mathcal{D}}_{\text{source}}}\frac{1-(1-q_{i,j})^{t}}{V(V-1)}$ $\displaystyle\geq\frac{|{\mathcal{D}}_{\text{source}}|}{V(V-1)}+\frac{|{ \mathcal{R}}\backslash{\mathcal{D}}_{\text{source}}|}{V(V-1)}\left(1-\left(1- \frac{1}{V(V-1)}\right)^{t}\right)$ $\displaystyle\geq(1-\varepsilon)\left(p+(1-\rho(\lambda))^{2}\left(1-\left(1- \frac{1}{V(V-1)}\right)^{t}\right)\right),$ $$

with high probability which completes the proof of the lower bound.
∎

To obtain a more precise description of $\mathsf{Acc}({\bm{M}}_{t})$, we employ a Poisson branching process to approximate the cluster growth of vertices, which we now define. A Poisson$(\lambda)$ branching process is a model for a population evolving in time, where each individual independently gives birth to a number of children with Poisson$(\lambda)$ distribution. We denote by $Z_{n}$ the number of individuals in the $n$-th generation, where by default $Z_{0}=1$. Then $Z_{n}$ satisfies the recursion relation $Z_{n}=\sum_{i=1}^{Z_{n-1}}X_{n,i}$, where $\{X_{n,i}\}_{n,i\geq 1}$is a doubly infinite array of i.i.d. Poisson$(\lambda)$ random variables. The total progeny $Y_{n}$ is then defined as $Y_{n}=\sum_{i=0}^{n}Z_{n}$. $Z_{n}$ is often called a Galton–Watson branching process and the associated tree is called a Galton–Watson tree.

As in the previous proof, an accurate estimate of $\mathsf{Acc}({\bm{M}}_{t})$ relies on understanding $q_{i,j}$, the probability that the edge $(i,j)$ will be added in each round. As before, the only edges that will be added are those connected to the giant component (i.e., $i\in{\mathcal{L}}_{X}$ and $j\in{\mathcal{L}}_{Y}$). The proportion of such edges converges to $C_{\lambda}$ as $V\to\infty$. Recall that

$$ $q_{i,j}=\frac{\sum_{(a,b)\in{\mathcal{R}}}\mathbbm{1}((i,j)\in\{(a,z^{1}),(a,z ^{2}),\dots,(a,z^{k}),(a,b)\})}{V(V-1)}$ (3) $$

where $(a,z^{1},z^{1},\cdots,z^{k},b)$ represents the shortest path in ${\bm{M}}_{0}$ connecting $a$ and $b$. Equivalently, if we consider the tree generated by a breadth-first search in ${\bm{M}}_{0}$ rooted at $i$, then since $i\sim j$, $j$ will be in the tree, and the numerator counts the total number of offspring of $j$ in the tree, including $j$ itself. This is the point at which a rigorous mathematical characterization of the tree becomes challenging. Instead, we approximate the tree and analyze its behavior. It is well-known that when $p=\lambda/V$, the cluster growth (or the breadth-first search at a vertex) can be approximated by a Poisson$(\lambda)$ branching process (see e.g., Hofstad (); Durrett ()).
For fixed vertex $i$, we define $T$ as a Galton–Watson tree rooted at $i$ with Poisson$(\lambda)$ offspring distribution with depth $L$. We use $T$ to approximate the exploration process at $i$. For $0\leq\ell\leq L$, the number of vertices at level $L-\ell$ is approximately $\lambda^{L-\ell}$. Given that the total number of vertices in $T$ is approximately $(1-\rho(\lambda))V$, the number of vertices at level $L-\ell$ is also $(1-\rho(\lambda))V(\lambda-1)/\lambda^{\ell+1}$. For each vertex at level $L-\ell$, the number of its offspring (including itself) equals $k$ with probability $p_{\ell}(k)$. In this case, the numerator in (3) equals $k$. Combining the above, there are around $(1-\rho(\lambda))V\cdot p_{\ell}(k)(1-\rho(\lambda))V(\lambda-1)/\lambda^{\ell
+1}$ vertex pairs $(i,j)$ in the graph such that $i\in{\mathcal{L}}_{X}$, $j\in{\mathcal{L}}_{Y}$, $q_{i,j}=k/V(V-1)$ and $j$ is located at the $L-\ell$ level in the tree $T$. Ultimately, we arrive at an approximation of the form

$$ $\displaystyle\mathsf{Acc}({\bm{M}}_{t})\sim p+C_{\lambda}\left(1-\sum_{\ell=0} ^{\infty}\frac{\lambda-1}{\lambda^{\ell+1}}\sum_{k=1}^{\infty}p_{\ell}(k)\left (1-\frac{k}{V(V-1)}\right)^{t}\right).$ $$

Beyond ErdHos-Rényi graphs, the term $q_{i,j}$ may not be as explicit. We can define $C$ as the proportion of vertex pairs $(i,j)$ such that $i\sim j$ in ${\bm{M}}_{0}$, then $q_{i,j}$ is nonzero for $CV(V-1)$ pairs of vertices. In this case, if we write $a_{k}=k/V(V-1)$ and define $\mu(k)$ as the probability that $q_{i,j}=a_{k}$, then we can have a general formula

$$ $\displaystyle\mathsf{Acc}({\bm{M}}_{t})\sim p+C\left(1-\sum_{k=1}^{\infty}\mu( k)\left(1-a_{k}\right)^{t}\right).$ $$

The drawback of this formula is the lack of explicit expressions. For a given ${\bm{M}}_{0}$, it is unclear how to compute the measure $\mu(\cdot)$ easily.

Next, we provide a qualitative description of the shape of such a mixture of exponentials.

###### Lemma F.3 .

For a fixed constant $0<C<1$ and a probability measure $\mu(\cdot)$ on $\mathbb{Z}_{+}$ with finite mean $m$, we define

$$ $\displaystyle f(t)=p+C\left(1-\sum_{k=1}^{\infty}\mu(k)\left(1-\frac{k}{V(V-1) }\right)^{tV(V-1)}\right).$ $$

Then we have that there exists $0<t_{1}<t_{2}$ such that

$$ $\displaystyle f(t)=\begin{cases}\Theta\left(p+t\right),\quad&\text{ for }0\leq t \leq t_{1},\\ \Theta(\log t),\quad&\text{ for }t_{1}\leq t\leq t_{2},\\ \Theta(1),\quad&\text{ for }t\geq t_{2},\end{cases}$ $$

as $V\to\infty$.

###### Proof of Lemma F.3 .

Fix any $1<t_{1}<t_{2}$. Note that $f(t)$ is monotone increasing, concave and always bounded by $1$. We also have

$$ $\displaystyle f(t_{2})\geq p+C\left(1-\left(1-\frac{1}{V(V-1)}\right)^{t_{2}V( V-1)}\right)\geq p+C(1-\exp(-t_{2}))=\Theta(1).$ $$

So $f(t)=\Theta(1)$ when $t\geq t_{2}$. Now when $t\leq t_{1}$,

$$ $\displaystyle f(t)\leq p+C\left(1-\sum_{k=1}^{\infty}\mu(k)(1-tk)\right)\leq p +Cmt.$ $$

Since $f(0)=p$ and $f(t_{2})\geq p+C(1-\exp(-t_{2}))$, by concavity, $f(t)$ is lower bounded by $p+tC(1-\exp(-t_{2}))/t_{2}=\Theta(p+t)$ for any $0\leq t\leq t_{1}$. Finally for $t_{1}\leq t\leq t_{2}$, we note that $f(t_{1})\leq f(t)\leq 1$, so easily, $f(t)\leq\log t_{1}/\log t_{1}\leq\log t/\log t_{1}=O(\log t)$. Similarly, $f(t)\geq f(t_{1})\log t_{2}/\log t_{2}\geq\log t(f(t_{1})/\log t_{2})\geq
\Omega(\log t)$. Therefore, $f(t)=\Theta(\log t)$ for any $t_{1}\leq t\leq t_{2}$.
∎

###### Proof of Theorem 1 .

###### Lemma F.1 (Lemma 1 and Corollary 1 in Karp ( 1990 ) ) .

###### Lemma F.2 (Theorem 3 in Karp ( 1990 ) and Theorem 2.4.1 in Durrett ( 2010 ) ) .

###### Lemma F.3 .

###### Proof of Lemma F.3 .

### F.1 Curve fitting with mixture of exponential formula

To perform curve fitting using the mixture-of-exponential formula, we approximate the infinite sum with three terms in

$$ $\displaystyle\mathsf{Acc}({\bm{M}}_{t})\sim p+C\left(1-\sum_{k=1}^{\infty}\mu( k)\left(1-a_{k}\right)^{t}\right).$ $$

Mathematically, we fit the empirical observation against the formula

$$ $y(x)=a-b_{1}r_{1}^{x}-b_{2}r_{2}^{x}-b_{3}r_{3}^{x},$ $$

where $x$ is the EntiGraph token count (in millions) and $y(x)$ is the QuALITY QA accuracy.
We use the non-linear least squares method implemented by Virtanen et al. ().
As a result of this procedure, we obtain the fitted formula

$$ $y(x)=64.5456-13.8352\times(0.9989)^{x}-8.4705\times(0.8961)^{x}-3.932\times(0. 0546)^{x}.$ $$

For the implementation of this procedure, we refer readers to our codebase.

## Appendix G Synthetic data generation prompts

We generate two synthetic corpora in this paper: EntiGraph (Appendix G.1) and the Rephrase baseline (Appendix G.2).
In our experiments, the ${\mathcal{D}}_{\text{source}}$ is a collection of documents $D$, and our synthetic augmentation procedure is applied to each document $D\in{\mathcal{D}}_{\text{source}}$.
We will focus on a single document $D$ for the remainder of this section.

### G.1 EntiGraph Prompts

The EntiGraph procedure is described in detail in §2.2.
We will recap the three steps below.

#### Step 1: Entity extraction.

The first step is to extract the salient entities from the document $D$ using the entity_extraction operation (Step 1, §2.2).
The complete entity_extraction prompt is as follows:

#### Step 2: relation analysis.

The last step is to generate diverse descriptions of relations among two or more entities.
In our experiments, for each document $D$, we enumerate all entity pairs and generate a description for each.
The prompt for generating a description relating a pair of entities is as follows:

We also generate synthetic data involving three entities, using the prompt below:

### G.2 Rephrase prompts

For the rephrase corpus, we adapt the prompt from Maini et al. () to our setting of books and articles.
We provide four rephrase styles below:

Easy rephrase:

Medium rephrase:

Hard rephrase:

## Appendix H Additional evaluation details of main experiments

### H.1 QuALITY QA question set

In this section, we provide more details of evaluation on the QuALITY QA test queries.
Throughout the closed-book QA experiments, we use a fixed 5-shot prompt below:

If the output of the model correctly follows the format of the few-shot prompt, its last two characters should be “A.”, “B.”, “C.”, or “D.”.
However, the model sometimes cannot successfully follow the few-shot prompting format, particularly for the continually pretrained model.
As a result, in all our evaluations, we sample the response 64 times, and only select the ones that can be parsed in the correct format.
Out of these 64 attempts, we randomly select among the valid answers to give the final answer.
Note that this is *different* from majority voting in self-consistency prompting (Wang et al., ).

### H.2 Closed-book Summarization

#### Automated evaluation metric.

We design a three-stage evaluation procedure:
(i) In the first stage, we use GPT-4(^6^66Specifically, we use the gpt-4-turbo model as of Aug. 19, 2024.) to break the summary into atomic claims, similar to Min et al. ();
(ii) In the second stage, we provide both the list of claims and the source article to a judge model (also GPT-4).
We ask the judge model to determine whether each claim is true or false, based on the source article.
If the claim is true, we further ask the model to determine whether the claim is salient (contributes to the main message of the article) or cosmetic (factual details that do not help understand the main message).
(iii) Finally, for each summary, we obtain its number of false and salient claims and normalize it by the corresponding count from the human summary.
We report the average of these normalized metrics across the QuALITY corpus articles in Figure 3.

#### Prompts to generate summaries.

For summarization evaluation with EntiGraph Instruct and Raw Instruct, we apply the following two prompts to obtain two summaries of increasing length.

**Table 4: Summarization prompt for EntiGraph Instruct, Raw Instruct, and Reprhase Instruct.**
| ➤ | Short prompt: Summarize the article {article title} by {author name} for me. |
| --- | --- |
|  | Give a short summary of ‘‘Cosmic Yo-Yo’’ by Ross Rocklynne. |
| ➤ | Long prompt: Write an extremely long and detailed article regarding the book {article title} by {author name}. |
|  | Write an extremely long and detailed article regarding the book ‘‘Cosmic Yo-Yo’’ by Ross Rocklynne. |

We provide three examples of summarization outputs below.
For each of the three examples, we will first present the human summary for this article to provide context for the example, and then present the short summary from the two summarizers.

#### Example 1.

The first example is “Cosmic Yo-Yo” by Ross Rocklynne.

#### Example 2.

The second example is “Beach Scene” by Marshall King.
We can see that EntiGraph correctly gives a summary of the story while Raw Instruct mistakes the book as a painting.

#### Example 3.

The third example is “The Madman From Earth” by “Keith Laumer”.

**Table 5: Complete instruction following example used in Table 2 from Section 4.3.**
| ➤ | Explicit article reference instruction: Summarize the article “Defining Decay Down” by “David Plotz” for me. |
| --- | --- |
|  | The article ”Defining Decay Down” by David Plotz discusses how the dental industry has changed over the years. In the past, dentists were primarily focused on treating cavities and other dental issues, but as the prevalence of cavities decreased due to the widespread use of fluoride, the industry had to adapt. Dentists began to focus on cosmetic dentistry, offering services like teeth whitening and orthodontics to appeal to the vanity of aging baby boomers. The article also discusses how dentists have become more customer-friendly, with amenities like virtual reality glasses and massage therapy. Despite the shift towards elective procedures, the dental industry has continued to thrive, with dentists earning higher salaries than ever before. |
| ➤ | Implicit article reference: How has the dentistry practice in United States has changed? |
|  | The dentistry practice in the United States has undergone significant changes in recent years. Here are some of the ways in which it has changed:<br>1. Focus on cosmetic dentistry: In recent years, there has been a growing demand for cosmetic dental procedures such as teeth whitening, orthodontics, and veneers. Dentists have responded to this demand by offering more cosmetic services.<br>2. Use of technology: Dentists are now using advanced technology such as intra-oral cameras, 3D imaging, and laser technology to diagnose and treat dental problems. This technology has made dental procedures more accurate and efficient.<br>3. Emphasis on preventive care: Dentists are now focusing more on preventive care to help patients avoid dental problems. This includes regular check-ups, cleanings, and fluoride treatments.<br>4. Increased competition: With the rise of dental schools and the influx of foreign-trained dentists, the dental industry has become more competitive. Dentists are now offering more services and amenities to attract and retain patients.<br>5. Focus on patient experience: Dentists are now focusing on providing a positive patient experience. This includes offering amenities such as virtual reality glasses, massage therapy, and entertainment during procedures.<br>Overall, the dentistry practice in the United States has become more patient-centered, technology-driven, and focused on preventive care and cosmetic procedures. |
| ➤ | Cross article instruction: Compare David Plotz’s commentary style between how he analyze American dentistry and how he discuss the movie Fight Club? |
|  | David Plotz’s commentary style is different when he analyzes American dentistry and when he discusses the movie Fight Club. In his analysis of American dentistry, Plotz uses a critical and investigative approach, highlighting the over-commercialization and unnecessary procedures in the industry. He uses statistics and expert opinions to support his arguments and provides practical advice for readers to avoid unnecessary dental procedures.<br>On the other hand, when discussing the movie Fight Club, Plotz adopts a more satirical and humorous tone. He uses pop culture references and witty commentary to analyze the themes and messages of the movie. He also provides a critical analysis of the film’s characters and their motivations, while also acknowledging the movie’s cultural impact.<br>Overall, Plotz’s commentary style is more serious and investigative when analyzing American dentistry, while his commentary on Fight Club is more satirical and humorous. |

### H.3 Additional quantitative summary evaluation

Our main summarization evaluation focused on testing the summarization ability of our strongest CPT models and the GPT baselines, which made a fair comparison to baselines such as rephrase difficult due to the difference in total token counts.

We perform a controlled comparison between EntiGraph and Rephrase CPT under subsampling the synthetic dataset and find that much like for the QA case, EntiGraph matches or improves upon Rephrase CPT, though the gains here are generally smaller.

Concretely, we apply the same instruction procedure described in §4.3 to the Raw CPT and Rephrase CPT models from §4.1, obtaining two additional instruction-tuned models that have knowledge about QuALITY books.
In addition, we also subsample 29M tokens out of the 455M token EntiGraph corpus to token-match the Raw and Rephrase corpus, and refer to the corresponding instruction tuned model as EntiGraph-29M.

Figure: Figure 8: Closed-book summarization: number of false claims ($y$-axis) versus number of salient claims ($x$-axis) normalized by the human summary. Same setup as in §4.3.
Refer to caption: x9.png

Figure 8 shows that EntiGraph summaries for the short prompt have significantly fewer false claims while having a comparable number of salient claims. The trend holds for the longer summary prompt, with clear separation in the error bars for the false claims gap between EntiGraph and Rephrase baselines, and overlap in the error bars for the salient claims count.

Finally, we also see clear improvements in scaling from 29M to the full EntiGraph model, with significant reductions in false claims for both the short and long prompts, suggesting that much like in the QA case, EntiGraph could bring improvements to knowledge-intensive downstream tasks through additional scale.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: It says “We generate data for pairs D_{Ei, Ej} and triplets D_{Ei, Ej, Ek} in our experiments”. I wonder if the authors have any intuition about how performance changes with the size of subset k.
soundness: 4
strengths: * The problem the work addresses is important.
* Experimental results show that this method scales better than simple paraphrasing or direct pretraining, and that retrieval-augmented generation further boosts performance of this model. 
* The authors also present a theoretical model explaining EntiGraph’s log-linear scaling pattern, providing insights into the mechanics of synthetic data’s impact on learning efficiency.
* Paper is clear and well-written.
summary: This paper addresses the problem of data inefficiency in pretraining language models. Current pretraining corpora may not generalize effectively and models may benefit from structured, repeated, diverse representations of knowledge. 

The proposed is a two-step process that (1) extracts entities from the corpus and then (2) extracts relationship information amongst a subset of the entities.

Experimentation uses the QuALITY corpus and dataset, which is a benchmark for long-document reading comprehension. Evaluation compares with relevant baselines like training on the original QuALITY corpus and a corpus containing rephrasings.
weaknesses: While the experiments focus on the QuALITY corpus, it remains unclear how well this would apply to other domain-specific corpora or more complex fields (e.g., legal or math data).

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: For the data generator, what type of models are necessary to have good performance? (why use GPT4 and not open-source models)
The paper shows that the generated data is useful, but how does it look like? (is it good quality text, factual, natural looking, ...) 
What is the significance of section 6?
soundness: 4
strengths: The paper does a good job at demonstrating the benefit of the synthetically generated data, by including relevant natural baselines. 
The proposed method seem to work well and can be useful for continued pre-training tasks.
summary: This paper proposes a method to continue pretraining LLM with a synthetic data augmentation method. The method is based on expanding the training corpus with many verbalizations of the entity graph present in the training corpus. It moves from a sparsely verbalized entity graph to a more densily verbalized one by using only the source documents and prompting LLMs to generate the new tokens.

The paper shows that the method is beneficial for downstream tasks in closed- and open-book QA as well as RAG. 

Overall, I think the paper is worthy of acceptance, it propose a clean method with good results and the experiments are fairly convincing.
weaknesses: The work relies on commercial and closed-source models (GPT4) for generating the synthetic data, making this work non-reproducible. Since the data generation process is the central contribution, it would have been interesting to have insights about how well different models can perform this data generation task. 

The paper proposes only extrinsic evaluation of the generated data but does not provide intrinsic measures, i.e., how good is the generated text? 

In my opinion, section 6 is not particularly useful. It is unnecessarily mathematical, based on simplistic assumptions and does not bring useful insights (For many continuously increasing lines, there anyway exists a mixture-of-exponential that fit it)

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. How sensitive is the synthetic pretraining process to the specific hyperparameters used for entity extraction and relation analysis? Would tuning these parameters significantly affect the generated corpus quality?

2. How does the synthetic corpus compare to a manually curated dataset in terms of quality and impact on downstream tasks?

3. Could EntiGraph be used effectively in scenarios where entities are ambiguous or domain-specific (e.g., medical or legal texts)?
soundness: 2
strengths: 1. The proposed EntiGraph approach for generating synthetic data is well-motivated and demonstrates clear improvements in downstream performance, as shown by the experimental results.
2. The paper includes comprehensive evaluations, including closed-book QA, instruction following, and open-book settings. The results show a significant performance improvement over baselines, validating the effectiveness of synthetic pretraining.
3. The authors provide a theoretical analysis of EntiGraph's effectiveness, which aligns well with empirical observations and provides a deeper understanding of its scaling properties.
summary: The paper proposes "synthetic continued pretraining" to enhance language models' domain adaptation efficiency, particularly when adapting to a small corpus of domain-specific documents. The authors introduce EntiGraph, a synthetic data augmentation algorithm that constructs a knowledge graph from the entities in the original corpus and synthesizes diverse text to enhance the pretraining process. The approach was evaluated using a reading comprehension dataset, QuALITY, showing notable improvements in model performance with both closed-book and open-book scenarios.
weaknesses: 1. The evaluation relies on the QuALITY dataset, which may not be representative of all types of small corpora. A broader range of datasets, particularly from diverse domains, would make the results more generalizable.
2. Although the authors attempt to mitigate hallucinations by grounding synthetic data generation in the original corpus, the risk of generating inaccurate information is inherent in using a language model for synthetic generation. This aspect needs further empirical examination, such as quantitative metrics to evaluate hallucination rates.
3. The approach relies on using strong language models like GPT-4 for synthetic data generation. The practical feasibility of using this approach might be limited if users do not have access to such models due to their computational cost. What if it was replaced with LLama 3 8B?
4. While the paper includes useful baselines such as "Rephrase CPT," more comparisons with alternative data augmentation or synthetic generation methods from recent literature could strengthen the claim that EntiGraph is an effective strategy.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: The paper could be more robust if you had more than just the QuALITY dataset. It is a perennial problem to find hard datasets to work with, so I understand this may be all there is for now, but given the chance I would attempt to reproduce the results on a different set. The authors mention linear algebra (a much harder topic, I think): is there any corpus for that subject?

The presentation of how exactly you generate the text to train Llama 3 8B with EntiGraph is still a little fuzzy to me, in particular it would be nice to see some examples of what you generated. It is helpful to have the prompts, but some output always grounds the presentation. 

Finally, I imagine GPT-4t made errors in producing the training data--did you search for these? Even at a quick glance how often did it make errors, and what, if anything, did you do to filter them out?
soundness: 4
strengths: The experiments are convincing that the EntiGraph approach improves the LLM's ability to accurately answer questions about a small corpus. In particular the closed-book results in Figure 3 show that the EntiGraph approach leads to far more salient claims per false claim than any of the other models, including GPT-4, or training the LLM (Llama 3 8B). The benefit is substantially less in the open-book RAG case, but there is still substantial improvement. The theoretical model to explain how the model improves QA accuracy with increasing tokens provides some good intuition as to how the model learns. 

Overall the text is clear and easy to read.
summary: This paper addresses how to train LLMs in a data scarce regime, given that LLMs require O(1000) examples of a fact to actually "learn" it. This has applications both to niche corpora (e.g., mathematics textbooks) as well as to training larger models once all human text is exhausted. The authors propose to use a pre-trained LLM to (1) extract entities and summaries from a comparatively small, niche corpus, and (2) use the extracted entities to generate rephrased assertions about those entities, to facilitate learning by a second (here, smaller) LLM. They experiment with a 1.3M token reading comprehension dataset, and test the approach against several baselines, including closed-book tests on the LLM used to extract entities and the rephrased text used to train the second LLM. Finally, the authors present a mathematical model through which they attempt to understand the training behavior of this data augmentation system.
weaknesses: I still have reservations that there is some amount of distillation of GPT-4 into their Llama 3 8B: it seems possible to me that a RAG-prompted GPT4 could generate additional information that is somehow "unlocked" by the RAG prompt, but which the closed-book version was unable to access. At the risk of anthropomorphizing, this is akin to a human getting a visual or audio cue and suddenly recalling whole complex memories. It would make the paper stronger to dig into the results of entity extraction and the generated text to see whether it is rephrasing/paraphrasing, or whether possibly actual new information is injected.

Even so, it would have helped this reader to have pointed out the significance of the closed book experiments earlier on. It isn't stated explicitly until the Limitations section.

I don't feel particularly qualified to check your proofs of theorems, and moreover I think the main value of the theoretical model is to help the reader understand intuitively why the approach works (these may be connected observations). Is all of the theory necessary? Perhaps a simulation would do as well?

Another issue is that much of the benefit of the approach vanishes (though not completely) when using a RAG model directly. Is this approach worth the extra training, given the modest gains? The core problem, really, is how many examples LLMs take to learn anything well. This paper finds a way to side-step that successfully, but doesn't solve it directly.

Meta-review: The paper proposes a data-synthesizing method for continued pretraining for adapting an LLM to a specific domain, where the proposed approach is based on EntiGraph including entities and their relations. In addition, the authors derived bounds for their scaling trends.

All reviewers agree that this is an interesting and solid paper. Reviewers have raised a few minor concerns and questions, which may be addressed in the revision.

Decision: Accept (Oral)