Learning Facts at Scale with Active Reading
Authors: Jessy Lin
Vincent-Pierre Berges
Xilun Chen
Wen-tau Yih
Gargi Ghosh
Barlas Oguz
Venue: ICLR 2026 Poster (2026)
Keywords: factuality
tail knowledge
synthetic data
synthetic continued pretraining
Primary area: foundation or frontier models, including LLMs
TL;DR: We let the model generate self-learning strategies and train on them at scale to learn tail facts more consistently.

Abstract: LLMs are known to store vast amounts of knowledge in their parametric memory.
However, learning and recalling facts from this memory is known to be unreliable, depending largely on the prevalence of particular facts in the training data and other factors which are poorly understood.
Practitioners are lacking tools which will allow them to ensure that the models learn a given body of knowledge reliably and consistently.
To this end, we propose Active Reading: a framework where we train models to study a given set of material with self-generated learning strategies.
First, we demonstrate models trained with Active Reading on expert domains absorb significantly more knowledge than vanilla finetuning and other data augmentations.
We train expert 8B models that achieve 66% on a Wikipedia-grounded subset of SimpleQA (+313% relative over vanilla finetuning) and 26% on FinanceBench (+160% relative over vanilla finetuning) by applying Active Reading to the source documents for each benchmark.
Finally, we show that Active Reading can be utilized at pre-training scale to build more factual models.
As a demonstration of this, we release WikiExpert-8B, a Wikipedia-expert model trained on 1 trillion generated tokens, which outcompetes models with hundreds of billions of parameters on factual QA.

Published as a conference paper at ICLR 2026
LEARNING FACTS AT SCALE WITH ACTIVE READING
Jessy Lin, Vincent-Pierre Berges, Xilun Chen, Wen-Tau Yih, Gargi Ghosh, Barlas Oğuz
FAIR at Meta
jessy_iineberkele..oo edu,
ABSTRACT
LLMs are known to store vast amounts of knowledge in their parametric memory. However, learning and recalling facts from this memory is known to be unreli- able, depending largely on the prevalence of particular facts in the training data and other factors which are poorly understood. Practitioners are lacking tools which will allow them to ensure that the models learn a given body of knowl- edge reliably and consistently. To this end, we propose Active Reading: frame- work where we train models to study a given set of material with self-generated learning strategies. First, we demonstrate models trained with Active Reading on expert domains absorb significantly more knowledge than vanilla finetuning and other data augmentations. We train expert 8B models that achieve 66% Wikipedia-grounded subset of SimpleQA (+313% relative over vanilla finetuning) and 26% on FinanceBench (+160% relative over vanilla finetuning) by applying Active Reading to the source documents for each benchmark. Finally, we show that Active Reading can be utilized at pre-training scale to build more factual models. As a demonstration of this, we release WikiExpert-88 -BB, Wikipedia- expert model trained on trillion generated tokens, which outcompetes models with hundreds of billions of parameters on factual QA.
INTRODUCTION
Large language models (LLMs) have the remarkable ability to store and retrieve vast amount of world knowledge in their parameters. However, it is poorly understood how to teach models to reliably learn and recall facts from this memory. During pretraining, models struggle to learn the long tail of facts that may appear only sparsely in the pretraining corpora (Kandpal et al., 2023; Wei et al., 2024). Incorporating new knowledge into models during finetuning has also been shown to be brittle, leading to increased hallucinations (Gekhman et al., 2024) memorization without the ability to manipulate new knowledge (Allen-Zhu & Li, 2024).
While models seem to learn common facts from pretraining, central challenge in language model training is how to systematically and scalably teach LLMs to learn facts without relying on incidental co-occurrence in large-scale web corpora. In this work, we study the question: given closed body of knowledge, how can we train models to most effectively internalize the information (e.g. to approach perfect factual recall)? Rote memorization of training data neither sufficient nor desirable for robust knowledge integration. Instead, we aim to promote generalizatinn-tie is, enabling models internalize facts in way that supports accurate reasoning, transfer to novel contexts, and compositional inference.
To that end, we introduce Active Reading, scalable and human- -imaan-nsspired framework that enables LLMs to study a given corpus by self-synthesizing data through diverse set of learning strategies. While models are typically trained with a sequential pass through the training text, humans actively study new knowledge to internalize it. Beyond reading piece of text once, we might imagine problems to apply new math theorem to, relate new fact to what we already know about a per- son, or understand concept by explaining it to ourselves in a different way (Brown et al., 2014). Active Reading applies the same principles to synthesize training data for LLMs by allowing the model itself to propose multiple study strategies e.g. paraphrasing, knowledge linking, active call, analogical reasoning- and instantiate these strategies on doc-dmmt basis. Unlike existing synthetic data generation methods that rely fixed templates such question-answer generation

---

Published as a conference paper at ICLR 2026
#cCreste timelineAx
2986:
J. Foge
the recipients and 1d
2307:
Wikipedia
s*treate Song on Rhynet*, of
(Verse 1) In two thousand
IECE Frank Rosenblati Award
ANUse Assaciation with other people events that you are faniliar with....
#Ajohn J. Hopfield (2609)m*: John Hapfictd is associated with the Hopfield network...
Source Document
1. Generate learning strategies to seed data generation
2. Apply strategies to generate diverse synthetic documents
Active Reading as two-stage synthetic data generation pipeline. In the first stage, the model comes up with diverse learning strategies specific the given document. the second stage, strategies are applied independently to generate the eeffttraining data.
Figure I:
simple repetition, this process results in highly diverse and contextually grounded training signal that emulates the way humans actively engage with new information.
First, we show that Active Reading is highly effective at improving factual recall given closed body of knowledge (Section 4. We achieve state-of-the-art factual recall on SimpleQA by studying the associated Wikipedia documents (Wei et al., 2024), improving factual recall from 16% to 66% (50% absolute, 312% relative) compared to naively finetuning on the documents. We also show that Active Reading be used to train expert models in a specific domain of interest such as finance, improving factual recall by 16% (160% relative) over vanilla finetuning on FinanceBench (Islam al., 2023).
We find that Active Reading also exhibits improved scaling trends compared to other synthetic data generation strategies generation plateau downstream recall performance as we generate more tokens, Active Reading continues improve
increase data scale (Figure 2). While paraphrasing and synthetic QA
generate up 4B words.
Finally, we push Active Reading to pre-training scale by generating trillion tokens of synthetic Wikipedia data to train WikiExpert, demonstrating that it be viable approach for build- ing more factual base models. WikiExxprt (8B) substantially improves factual accuracy on Sim- pleQA and NaturalQuestions. outperforming substantially larger models including the 236B param- eter DeepSeekV2 V2 and 405B parameter Llama 3.1 models and competitive with the state-of-the-at DeepSeek V3 (671B) model.
Taken together, our results suggest that Active Reading is an effective and scalable method to teach models knowledge.
As part of this work, we make the following contributions:
We propose Active Reading, a flexible and scalable pipeline for LLMs to study a corpus with self-generated learning strategies, enabling more effective fact learning.
We demonstrate that studying text with Active Reading leads to substantial improvements (160-312% relative) in factual recall expert benchmarks and adversarial fact QA tasks, as well as improved scaling behavior as we scale synthetic data generation up to 4B words.
We scale Active Reading to trillion tokens of synthetic Wikipedia data, demonstrating its viability pre-training augmentation. We release WikiExxert, the most factual 8B parameter model to date, and the full IT-token synthetic dataset to support future research factual language modeling.
Together, these contributions offer step towards building reliably factual language models.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORK
Knowledge Incorporation. Large pretrained language models (LLMs) have been shown to en- code substantial world knowledge, effectively acting implicit knowledge bases (Petroni et al., 2019; Roberts et al., 2020). Their information storing capacity and ability to use this information scales with model size (Brown al., 2020; Tirumala al., 2022: Morris et al., 2025). It has been repeatedly observed that these abilities are inconsistent, and degrade substantially for rare facts (Sun et al., 2024; Wei et al., 2024; Kandpal al., 2023). Generally. it is not fully understood how to teach models new knowledge after pretraining (Zhao et al., 2025; Ren & Sutherland, 2025; Sun et al., 2025; Lu et al., 2024b; Chang al., 2024). Simply finetuning on facts, paraphrases, or documents exhibit limited effectiveness or poor scaling, knowledge conflicts (Sun et al., 2025) or unintended side effects like hallucination (Kang et al., 2024; Ghosal et al., 2024). AlKhamissi al. (2022) presents a comprehensive survey. Our work provides method to incorporate large amounts of new knowledge at pretraining data scale, and finds that factors such as data diversity and mixing in pretraining data are critical to scale knowledge incorporation.
Synthetic Data Generation. Numerous prior works study synthetic data generation for question answering (Du al., 2017; Alberti et al., 2019; Lewis et al., 2021). These explicitly target factuality with a narrow task focus, however do not consider generalization in the context of general purpose language models. Another line of work, which resulted in the Phi series of models (Gunasekar et al.. 2023; Li et al., 2023b; Abdin et al., 2024a;b), investigates the use of synthetic training for L.Ms for wider variety of tasks including language understanding, math and reasoning. However, learning knowledge is an explicit non-goal for this work, and the models do poorly in this area. For knowledge incorporation, several works try direct paraphrasing to improve factual recall, as suggested by Allen-Zhu & Li (2024). Ovadia et al. (2024); Maini et al. (2024); Park et al. (2025) apply this idea at moderate scale. More recently, Yang et al. (2024b) propose synthetic training using the EntiGraph method, which underperformed synthetic QA but showed better generalization. Our method a superset of these specific data generation methods, and we hypothesize that we outperform them due to increased diversity in generation.
Domain Adaptation Domain adaptation of language models for knowledge-intenise expert do- mains is a large area of literature in itself (Ling et al., 2023), particularly in medical (Peng et al., 2019; Lee et al., 2020; Luo al., 2022; Singhal al., 2025), financial (Araci, 2019; Liu et al.; Li et al., 2023a) and legal (Chalkidis et al., 2020; Guha et al., 2023) domains. Most works this area focus on curating large domain-specific datasets, with only few focusing on synthetic data generation (Yue et al., 2021). Our method is both general, in that it can be applied to any domain, and specifie, in that naturally adapts the domain it is applied on, making it unique among prior domain adaptation methods.
3 ACTIVE READING
The prevailing method used by practitioners to improve factual coverage of a given knowledge source is simply to increase the mixing weight of that source the data. However, such passive repetition is known to have limited effectiveness, since too much repetition causes overfitting without generalization. Recent work has established that paraphrasing the data can address this issue some extent, allowing for more repetition of the target semantic content without overfitting to particular surface forms (Allen Zhu & Li. 2024). However, paraphrasing is but one strategy for learning. Humans often learn by employing several diverse strategies: ranging from active recall, spaced repetition, asking and answering questions, using diagrams, and many others (Brown et al., 2014).
Some of these strategies might be more effective than others, and may depend on what knowledge we aim to learn. For instance, one might want to study a historical period by putting events on a timeline. Abstract mathematical concepts might be more robustly understood with concrete analogies, or practice problems. Moreover, learning strategies might be complementary, and combining them might be better than using any one of them in isolation. Instead of trying to engineer the best strategy for learning facts, we ask the model itself to come up with a diverse set of learning strategies, given document want to study. We then apply each generated strategy the associated document,
3

---

Published as a conference paper at ICLR 2026
Table 1: Performance on expert domains. SimpleWikiQ4 tests general tail knowledge, while Fi- nanceBench focuses on domain-specific knowledge. The info. subset of FinanceBench consists of questions which pre-dominantly test for information-extrattti while all corresponds to the perfor- mance on the whole set of questions.
Model Method
SimpleWikiQA Model Grader
FinanceBench info.
all
Llama 3.1 8B Base
7,42
3.93
6,00
repeat (finetune raw documents)
15.92
18.43
10,49
paraphrase
25.74
43.87
17.64
synth QA
47.87
44.23
17.16
Active Reading (task-agnostic)
63.33
66.18
26.83
Active Reading (task-specific)
66.25
61.49
25,16
paraphrase + synth QA + Active Reading
66.66
64.45
26.12
gold context (Llama 3.1 8B Base)
65.85
84.71
44.36
gold context ceiling (Llama 3.1 70B Instruct)
90.55
92.49
57.43
creating multiple diverse augmentations which can then used to train the model. We refer to this simple two-stage data generation pipeline as Active Reading, which we illustrate in Figure 1.
While one can think of many ways in which to prompt a model to generate Active Reading strategies, in this work we instantiate the framework with two prompts. One is generic prompt which asks the model to generate strategies to study the given material generally. without any particular appli- cation in mind (task agnostic). The other is task-specific prompt, where the model is told what downstream task it will be evaluated on (c.g. a trivia competition, or expert financial analysis). This strategy asks the model first imagine questions from the downstream task, and then come up with strategies that would help it master the type of content asked in its self-synthesized questions. This enables generate data that is both (1) more relevant for downstream performance (e.g. focusing on tail facts for trivia competition), as well as (2) more diverse, by focusing on specific aspects of the document at time. We refer this as task-specific Active Reading. We provide both prompts in the Appendix D.3.
Regardless of the particular strategy prompt, the resulting learning strategies from Active Reading are both diverse and context-specific, providing an advantage over any single fixed strategy. In fact, we find that Active Reading recovers many of previously proposed data augmentation strategies, including paraphrasing (Allen-Zhu & Li, 2024), synthetic question answering (Lewis et al., 2021), and concept maps (Yang et al., 2024b).
APPLICATIONS
We will evaluate Active Reading on two overlapping but distinct use cases.
4.1 EXPERT DOMAIN ADAPTATION
First, we investigate whether Active Reading can be an effective strategy to train expert models on given knowledge-intensive domain, such as medical or finance. this setting, given set of documents, we aim to internalize as much knowledge as possible from the documents to solve downstream task (e.g., question answering). We use the following benchmarks for this purpose:
FinchceBench (Islam et al., 2023) is question answering benchmark grounded on financial dis- closure documents of number of publicly traded companies. The benchmark consists of 10,231 questions, which are split into categories such as numerical reasoning, logical reasoning, and infor- mation extraction. Since teaching the model knowledge (rather than reasoning) is the focus of our work, focus the "information extraction" category, but also report overall results.
SimpleWikiQA
is a subset of 3,449 questions from the recent adversarially collected factual ques- tion answering benchmark SimpleQA (Wei al., 2024), where at least reference document for
4

---

Published as a conference paper at ICLR 2026
Scaling Laws for SimpleWikIOA
Artive Reacing cistbsppecifec]
[laskagrostic)
synth DA
paraphrase
Figure 2: Scaling trends with respect to the number generated words for each method. While baseline data aug- mentation strategies like paraphrasing and synthetic QA generation plateau in performance as we scale the amount of synthetic data, Active Reading leads to continued gains in downstream QA ac- curacy up to 4B generated words.
Simplehk Accurayy
0.5
0.4
0.3
0.0
0.5
1.0
1.5
2.0
2,5
3.0
3.5
4.0
Number training
le9
each question is from Wikipedia. Despite the fact that grounding in Wikipedia ensures that the in- formation has been seen multiple times the pre-training corpus of every mainstream LLM, the obscure questions in this benchmark remain challenging for current models. In the expert domain setting, we use the set of Wikipedia documents from SimpleWikiQA as our training corpus, evaluat- ing how effectively models can learn tail facts from this small of documents that have otherwise proven difficult for models to learn.
For baselines, we evaluate training on raw documents with simple repetition, paraphrasing and syn- thetic question generation. We also provide a baseline where we provide the relevant documents as context to the model (gold setting) in addition to the question, which serves as an upper bound of performance for this task. We do not consider EntiGraph Yang et al., 2024b), since it was shown to underperform synthetic QA (see their Appendix therein). For prompts and other implementation details pertaining to baselines, see Appendix D and Appendix A.
For the main experiments this section, we finetune the Llama3.1 8B base model for 20,000 steps. We fix the number of tokens generated for each baseline to ~ 4 billion words. During training, we mix 10% pre-training data from DCLM (Li et al., 2024) to prevent model degradation. For full training details, see Appendix A.
Results are presented in Table 1. We evaluate the answers of the models using GPT-4o (version 2024-06-0I). Compared to the base model performance (Llama 3.1 8B Base), finetuning the raw documents (repeat) provides only small gains in downstream QA performance. Training with data augmentations like paraphrase and synthetic question-answer (synth QA) provides some improve- ments. Active Reading leads to the strongest factual recall performance on both SimpleWikiQA and FinanceBench, model is provided with the documents in context. On FinanceBench. despite good performance on the information extraction subset, there is still substantial gap with the gold ceiling on the overall benchmark, indicating that on questions that require additional reasoning, in-context processing still outperforms our purely parametric method.
matching the gold context baseline SimpleWikiQA, where the base
In Figure 2, we show the scaling trends of each method with respect to the size of the generated data up to billion words. This figure confirms that Active Reading outperforms other methods across data scales, and maintains a stronger scaling trend, likely due to higher data diversity. In particular, paraphrasing saturates quickly, likely due to the limited number of ways a model can find to paraphrase particular document. While synth QA demonstrates strong performance at smaller scales, also plateaus as we scale the number of synthetic tokens.
4.2 FACT LEARNING AT SCALE
How does Active Reading scale increase the size of the corpus of knowledge want the model to internalize? With a sufficiently scalable approach, one could imagine integrating Active Reading into large scale pre-training or mid-training training pipelines, with the hope of mastering
5

---

Published as a conference paper at ICLR 2026
domains with large knowledge corpora (e.g. all medical research) or making base models more factually consistent on a wide set of domains.
To this end, apply Active Reading on the entirety of Wikipedia, generating trillion tokens by augmenting million articles. Training on such a large set of (synthetic) data comes with its own challenges, some of which go into in the next section.
4.2.1 SCALING LAWS FOR FACT LEARNING WITH SYNTHETIC DATA
The SimpleWikiQA reference document set is subset of Wikipedia, covering only about 0.1% all documents. When start expanding this set with more Wikipedia documents in our Active Reading training set, we un- surprisingly see declines in SimpleWikiQA performance. As the pool of documents gets larger, recall of specific facts decrease. This problem is analogous to scaling is- sues faced by dense (Piktus et al., 2022) and generative (Pradeep et al., 2023) retrieval models. In Figure 3, we expand our training set with 4x and 16x more Wikipedia documents and see that the degradation of performance is substantial, even at this moderate scale (less than 2% of Wikipedia documents).
F1 score on SimpleQA
0.5
0.4
Original Data
0.2
With Wikipedia (4x Original) With Wilopedia (16x Original)
SK
10K
15k
20K
Number of training steps
Figure 3: Effect of distractors in fine-tuning setting. Expanding the set of Wikipedia documents 4x and 16x leads to substantial degradation of per- formance on SimpleWikiQA.
We find that two major modifications to training setup greatly improve the scaling behaviour. First, we modify hyper-parameters to more closely resemble (continued) pre-training rather than fine-tuning, by greatly increasing the learning rate (from le 5 to 3e 4). Second, we increase the weight of pre-training data in our data mix.
First, we hypothesized that with the typical learning rates used for fine-tuning (as in the previous section; Figure 3), the model struggles to allocate enough capacity to learn new facts at scale. In- creasing the learning rate forces the model out of its local minima, creating more elasticity for learning. However, the larger learning rate also "breaks" previously learned capabilities, causing decreased performance on guardrail tasks. To encourage the model to recover previous capabili- ties, we increase the weight of pre-training data in the data mix. In Figure 4, we plot scaling laws as we scale up the data size with these changes. We continue training from Llama-3.1-80 on data mix of Active ments), Active Reading-aus Reading-augmented -ugmented SimpleWikiQA documents (a subset of Wikipedia docu- Wikipedia, and generic pre-training data. We keep the mixing ratio of augmented Wikipedia and pre-training data 1:1 and decrease the relative weight of aug- mented WikiQA, so that the relative proportion of SimpleWikiQA data varies between 80% (mostly SimpleWikiQAl to 2.5% (mostly augmented Wikipedia and pre-training data). For the largest scaling run, we do a total of 200k steps with 2.5% augmented SimpleWikiQA (5k steps), 48.75% augmented Wikipedia (97.5k steps), 48.75 75% pre-training data (97.5k steps). To keep the number of gradient steps we do on SimpleWikiQA constant, we increase the total number of steps we decrease the relative proportion of SimpleWikiQU) The detailed configurations, details on training dynamics, and performance are on guardrail metrics can be found in Appendix B.
As seen in Figure 4b, increasing the relative amount of pre-training data recovers guardrail per- DaeeQQuusssioonwiiatttttteee
formance on benchmark. Surprisingly, we also recover performance on our target task of SimpleWikiQA, even as we scale down the relative proportion of augmented SimpleWikiQu training data. The poorly performing intermediate checkpoints still output well-formed and reasonable incorrect answers (e.g. when asked "Who received the IEEE Frank Rosenblatt Award in 2010?", the 40% model responds "geoffrey hinton," an award winner for another year mentioned in the document). This suggests that the typical explanation of catastrophic forgetting of underlying model capabilities does not fully count for the scaling behavior we observe. This poses an interesting question for future work as to why mixing in pre-training data is beneficial for learning, which we discuss in Section 6.
et al., 2019),
Eikipedia-baseee question answering

---

Published as a conference paper at ICLR 2026
0.35
0.40
SimpleQA
NQ F1
Finetune
0.25
Finetune
80%
80%
0.30
40%
40%
20%
20%
10%
10%
0.25
2.5%
2.5%
107
10a
10
107
10*
10a
Number training steps
Number training
(a) Scaling behavior on SimpleWikiQA.
(b) Scaling behavior on NaturalQuestions (guardrail task).
decrease the proportion of SimpleWikiQA training data from 80% to 2.5% by scaling up the amount of Active Reading Wikipedia and pretraining data. We compare the finetuning baseline (Finetune with Ir=le-5, 100% Active Reading SimpleWik- iQA data) against different amounts of Active Reading-augmented SimpleWikiQA, Active Reading- augmented Wikipedia, and pretraining data with higher learning rate (3e-4). We keep the number of steps augmented SimpleWikiQA fixed 5000 but increase the relative amount of augmented Wikipedia and pretraining data, so that the proportion SimpleWikide Surprisingly, when we increase the relative amount of pretraining data, we not only recover guardrail performance, but also recover the original performance on SimpleWikiQA despite doing the same number of steps this data.
Figure 4: Continued pre-training
varies from 80% to 2.5% %.
4.2.2 TRAINING A WIKIPEDIA EXPERT
Based on these scaling experiments, for our final model we train for 4 epochs over IT tokens of Active Reaging-augmented Wikipedia data and 1T tokens of pretraining data, resulting in 8T tokens of training. We do not upweight the SimpleWikiQA documents for this run. We refer to this model WikiExpert, and evaluate 3 factuality benchmarks:
SimpleQA (Wei et al., 2024), and adversarially collected QA benchmark which measures the ability of models to recall tail facts from their pre-training data
NaturalQuestions (NQ) (Kwiatkowski et al., 2019), a popular Wikipedia- based question answering benchmark which covers more natural distribution of facts
TriviaQA (Joshi et al., 2017), a question answering dataset sourced from trivia websites
We summarize results for each benchmark Table For tail-fact recall as measured by Sim- pleQA, we see that Wikižxperi improves over it's base model by 222% (7.2 + 23.5), outperforms much larger models including the 236 billion parameter DeepSeekV2 and the 405 billion parame- ter Llama 3.1 models, and is competitive with the state-of-the- DeepSeekV3 (671B) model. On non-adversarial factual QA benchmarks, WikiExpert also improves over its base model (7.6% on NQ and 6.5% TQA). approaching the performance of SoTA 72B model. These results suggest that Active Reading is scalable approach for learning knowledge and has the potential to be useful more broadly in pre-training mid-training pipelines.

---

Published as a conference paper at ICLR 2026
Table Performance comparison across factual QA benchmarks: SimpleQA, Natural Questions (NQ) and TriviaQA (TQA), Qwen and DeepSeek numbers are reported at bctps://githhr com/
Table 3: Impact of scaling size of the trained
model and data-generating model.
Model
SimpleQA
NQ
TQA
Model
SimpleQA
Llama 8B WikiExpert-88
7.3
29.0
64.3
Llama 8B Base
7.42
23.5
31.2
68.5
with SB-generated data
66.25
with 70B-generated data
62.26
Qwen2.5 72B
9.1
33.2
71.9
Llama 70B Base
75.76
DeepSeck V2 236B
10.2
38.6
80.0
with 8B 8B-generated data
71.52
Llama 405B
17.1
41.5
82.7
with TUB-generated data
77.14
DeepSeek V3 671B
24.9
40.0
82.9
ANALYSIS
5.1
UNDERSTANDING ACTIVE READING DATA
What accounts for the difference between data generated by Active Reading and those generated by other synthetic data generation methods-i.e. why is much more effective learn with Active Reading?
First, we investigate whether different methods differ in their coverage of the answers in Simple Wik- iQA. One hypothesis for why Active Reading is more effective is that generating fong-form synthetic documents incorporates knowledge from the source documents more comprehensively. whereas methods like synthetic QA may only help the model to answer a test question correctly if it happens to generate the exact question during data generation. We sample a subset of 100 SimpleWikiQA questions and ask Llama 3.1 70B Instruct model to judge whether the question is answerable from each synthetic document chunk generated from source document for that question. Figure 5, plot the fraction of answerable questions (i.e. at least one document contains the answer) increase the number of synthetic documents, corresponding to training with more synthetic tokens. Synthetic QA data exhibit the highest coverage, despite underperforming Active Reading. Gener- ally, the differences between methods are small, and most questions are answerable after generating 75 document chunks (corresponding to approximately 1M 2M total generated words on Figure 2). Therefore, we conclude that the differences between the methods are not primarily accounted for by coverage of the target answers.
We hypothesize that Active Reading data is more effective because it is more diverse, which may improve knowledge incorporation and learning. We quantify the diversity of the data by measuring Self-BLEU (Zhu et al., 2018), considering each synthetic document as the hypothesis and all other synthetic document as the references and averaging these BLEU scores. Lower self-BLEU scores indicate more diversity (less n-gram overlap) between different synthetic data samples for the same source document. Figure we compare Self-BLEU for 100 sampled source documents as we increase the number of synthetic words generated. As we generate more synthetic tokens, expect to see Bif-BBLEU scores increase, as becomes more likely that we generate repetitions of the same way study the content. Even with few synthetic augmentations per source document, data generated from synthetic QA and paraphrasing is highly self-similar, while data generated by both Active Reading methods is more diverse. Task-specific Active Reading is also more diverse than task-agnostic, aligning with the improved scaling properties that we observe.
5.2 SCALING MODEL SIZE
We investigate the impact of scaling model size on performance by additionally training and gener- ating data with 70B models. We show the performance of task-specific Active Reading in Table as we scale model size. Surprisingly, training the 8B model with data generated by the 70B model does not outperform using self-generated data, despite intuitions that 70B-geeeetted data may be
8

---

Published as a conference paper at ICLR 2026
1.0
100
Fraction answerable
0.8
80
BLIE (L)
60
0.4
40
Active Reading (task (tack-specffic)
Active Reading (aask-spccifcc
Active Reading (aak-agnosti
70-
Reading (task agnestic
synth
synth QA
0.0
paraphrase
paraphrase
25
50
100
125
150
175
25K
50K
75K
100K
Number of document chunks
Number of synthetic words source doc
Figure 5: Coverage of n=100 SimpleWikiQA an- swers in data synthesized by different methods. Most questions are answerable after generating 75 chunks for all methods.
Figure 6: Diversity of data synthesized by differ- ent methods, measured with self-BLEU between synthetic data documents for the same source document. We observe that the more effective methods generally exhibit higher data diversity.
more diverse or higher quality. One hypothesis that training on data that closer to the model's existing comprehension capabilities and knowledge leads to better learning (i.e. not too challenging or out-of-distributtin for the model), and perhaps that it may important to generate the data from the exact model itself. We leave these hypotheses directions for future work to investigate.
DISCUSSION & CONCLUSION
Understanding the role of pretraining data in knowledge acquisition. In Figure 4, we found that mixing in a large amount of pretraining data was necessary to recover performance on Sim- pleWikiQA. We kept the number of steps on SimpleWikiQA fixed, decreasing the proportion of this data as we increase the pretraining weight. This poses a question for future work as to why mixing pre-training data helps with knowledge acquisition: it because we recover some underlying capability of the base model that is important for task performance, or because some feature of di- verse pre-training data induces more "plasticity," making the model more amenable to absorbing or organizing new knowledge (e.g., by reversing knowledge entropy decay (Kim et al., 2024)?). Understanding the properties pre-training that lead to robust knowledge acquisition may enable practitioners train models that master any domain of interest, and may be key milestone towards lifelong learning settings where the model continually acquires new knowledge.
Parametric knowledge vs. serievallauugenter generation. A common approach to improving factuality, particularly on rare domains and tail queries, is retrieval augmented generation (RAG). In this work, we have opted to focus on the closed-book setting. However, the "gold-context" baselines provide sense of the performance of RAG system with an oracle retrieval. Modern RAG systems are reported to perform close to this oracle baseline on the benchmarks we evaluated in this work. Therefore our results suggest that there is still substantial gap between the RAG setting and what the best methods can achieve with parametric updates.
Nevertheless, there are numerous advantages improving parametric model knowledge (vs. using an external knowledge index). Practically, RAG pipelines are considerably more complex, with larger storage cost, higher memory and compute requirements (due to longer context demanded by additional context) and higher latency, due to the need to query an external index. In the long term, storing knowledge natively in the model's parameters may provide generalization advantages, as the model can relate different pieces of knowledge in its parameters. For example, it is clear coding model that operates primarily by retrieving similar problems fundamentally limited in its capabilities. This is also particularly apparent for complex indirect queries, where simple retrieval augmentation falls short (Yang et al., 2024a). We believe integrating our method with more powerful reasoning models can make progress in these areas.

---

Published as a conference paper at ICLR 2026
Scaling up Active Reading as a training paradigm. Our scaling results suggest that Active Read- ing has potential to be scaled up as a standard pre-training or mid-training procedure. One could imagine augmenting training data with Active Reading by default- ensuring the model always "studies" new information in diverse enough contexts for learning, and providing a solution to con- cerns that pre-training is increasingly limited by data scale (Villalobos et al., 2024).
What would be needed to get to 100% recall on all facts (that we care about) in the pre-training data? Whether not LLMs can eventually replace search engines seems be recurring question (Tay et al., 2022: Bevilacqua et al., 2022). Recent work (Lu et al., 2024a) has suggested that this an altogether infeasible goal. The capacity of LLMs to store facts with respect to the number of parameters remains an open area of research. Architectures which allow scaling this capacity without increasing FI.OPs would be of particular interest in studying these scaling laws (Shazeer et al., 2017; Berges et al., 2024).
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint (rXiv:2404.14214 2024a.
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasckar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, al. Phi-4 techni- cal report. arxiv preprint arXiv:2412.08905, 2024b.
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. Synthetic qa corpora generation with roundtrip consistency. arXiv preprint arXiv:1906.054410 2019.
Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona T Diab, and Marjan Ghazvininejad. A review on language models as knowledge bases. CoRR, 2022.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. In International Conference on Machine Learning, pp. 1067-1077. PMLR, 2024.
Dogu Tan Araci. Finbert: Financial sentiment analysis with pre-trained language models. arXiv preprint arXiv:1908. 2019.
Encent-PPerre Berges, Barlas Oguz. Daniel Haziza, Wen-tau Yih, Luke Zettlemoyer, and Gargi Ghosh. Memory layers at scale. arXiv preprint arXiv:2412.09764 2024.
Michele Bevilacqua, Giuseppe Ottaviano, Patrick Lewis, Scott Yih, Sebastian Riedel, and Fabio Petroni. Autoregressive search engines: Generating substrings as document identifiers. Advances in Neural Information Processing Systems, 35:31668-3166 2022.
Peter C Brown, Henry L Roediger III, and Mark A McDaniel. Make it stick: The science of success- ful learning. Harvard University Press, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in nearal information processing systems, 33:1877-1901, 2020.
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androut- sopoulos. bert: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2898-28880 2904, 2020.
Hoyeon Chang, Jinho Park, Seonghyeon Ye, Sohee Yang, Youngkyung Seo, Du-Seong How do large language models acquire factual knowl- In Advances in Neural Information Processing Sys- URL https: http:s::/pppppppppppeee nips. httppppss.iissscccpppp////////// Pawp::eeee72i--Paa----oooooeeee 3=c5333111o11eeee00088822222---3
Chang. and Minjoon Seo. edge during pretraining? tems (NeurIPS), 2024.
(GGE57771bcifeee2200 4b0.000222222222-----222
.df.
Xinya Du, Junru Shao, and Claire Cardie. Learning to ask: Neural question generation for reading
comprehension. arXiv preprint arXiv: 1705.00066 2017.
10

---

Published as a conference paper at ICLR 2026
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning LLMs on new knowledge encourage hallucinations? In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings the 2024 Conference Em- pirical Methods in Natural Language Processing, pp- 7765-7784, Miami, Florida, USA, Novem- ber 2024. Association for Computational Linguistics. doi: N0.18653/v1/2224.emm URL https::///// httpsr//aaaaaatttaaaoeee org/ 2024 emnlp mttemmain..444... 444/.
Gaurav Ghosal, Tatsunori Hashimoto, and Aditi Raghunathan. Understanding finetuning for factual knowledge extraction. arXiv preprint arXiv:2406. /4785, 2024. URL /tRssttts://arxi..err 0bbs/2406.0 14785.
Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wond Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. Legalbench: collabo- ratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36:44123 2023.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprini arXiv:2306. 1644, 2023.
Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vid- gen. Financebench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.1144,, 2023.
Mandar Joshi, Funsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min- Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, Vancouver, Canada, July 2017. Association for Com- putational Linguistics. doi: 10. 100118653/1/P11-1116 P17-1147/.
URL https:
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference Machine Learning, pp. 15696-15707 PMLR, 2023.
Katie Kang, Eric Wallace, Claire Tomlin, Aviral Kumar, and Sergey Levine. Unfamiliar finetuning examples control how language models hallucinate. arXiv preprint arXiv:2403.05612. 2024. URL /ettps:/aarxte org/ abs 2403. 05612.
Jiyeon Kim, Hyunji Lee, Hyowon Cho, Joel Jang, Hyeonbin Hwang, Seungpil Won, Youbin Ahn, Dohaeng Lee, and Minjoon Seo. Knowledge entropy decay during language model pretraining hinders new knowledge acquisition. 2024. URL https:/ /api- wew.ntisscooooo..ooom org/ CozpusID:27302577
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Lingnistics, 7:453-466, 2019.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jae- woo Kang. Biobert: pre-trained biomedical language representation model for biomedical text mining. Bioinformaties, 36(4):1234 1240, 2020.
Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, and et al. PAQ: 65 million probably-asked questions and what you can with them. Transactions of the Association for Computational Linguistics, 9:1098-111 115,, 2021.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-Im: In search of the next generation of training sets for language models. Advances Neural Information Processing Systems, 37:14200-11409 2024.
11

---

Published as a conference paper at ICLR 2026
Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. Large language models in finance: A survey. In Proceedings of the fourth ACM international conference on Al in finance. pp. 374-382, 2023a.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.0544 05463, 2023b.
Chen Ling, Xujiang Zhao, Jiaying Lu, Chengyuan Deng, Can Zheng, Junxiang Wang, Tanmoy Chowdhury, Yun Li, Hejie Cui, Xuchao Zhang, et al. Domain specialization as the key to make large language models disruptive: comprehensive survey, arXiv preprint arXiv:2305.18703, 2023.
Xiao-Yame Yang Liu, Guoxuan Wang, Hongyang Yang, and Daochen Zha. Fingpt: Democratizing internet-scale data for financial large language models. In NewrIPS 2023 Workshop on Instruction Tuming and Instruction Following.
Xiaonan Lu, Yaqing Li, Xing Lu, Yiming Ma, and Yilun Wang. Scaling laws for fact memorization of large language models. arXiv preprint arXiv:2406. 15720, 2024a.
Xingyu Lu, Xiaonan Li, Qinyuan Cheng, Kai Ding, Xuanjing Huang, and Xipeng Qiu. Scaling laws for fact memorization of large language models. arXiv preprint 27X2::2406.15720, 2024b. URL https: arwii.iiivv///.. org/ abs 2406.15720
Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, and al. Biogpt: Generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 2022.
Pratyush Maini, Skyler Seto, Richard Bai, David Grangier, Yizhe Zhang, and Navdeep Jaitly. Rephrasing the web: A recipe for compute and data-efficient language modeling. In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers), pp. 14044-14072, 2024.
John X Morris, Chawin Sitawarin, Chuan Guo, Narine Kokhlikyan, G Edward Suh, Alexander M Rush, Kamalika Chaudhuri, and Saeed Mahloujifar. How much do language models memorize? arXiv preprint arXiv:2505.22 248332, 2025.
Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine-tuning retrieval? com- paring knowledge injection llms. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 237-250, 2024.
Core Francisco Park, Zechen Zhang, and Hidenori Tanaka. New News: System-2 fine-tuning for ro- bust integration of new knowledge, 2025. URL ptttps://arxite htt.o/g/aaa///...// 01812.
Yifan Peng, Shankai Yan, and Zhiyong Lu. Transfer learning in biomedical natural language pro- cessing: An evaluation of bert and elmo on ten benchmarking datasets. BioNLP 2019, pp. 58, 2019.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models knowledge bases? In Proceedings of EMNLP, pp. 2463- 2019.
Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gau- tier Izacard, Patrick Lewis, Barlas Oğuz, Edouard Grave, Wen tau Yih, and Scbastian Riedel. The web is your oyster- snowledge-intensiing nlp against very large web corpus, 2022. URL
https: arxiv. org/ abs 2212..122.22 09924.
Ronak Pradeep, Kai Hui, Jai Gupta, Adam D. Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, and Vinh Q Tran. How does generative retrieval scale to millions of passages?, 2023. URL /tttps://axxvv...
Yi Ren and Danica Sutherland. Learning dynamics Ilm finetuning. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. URL https:/ / openreview.ner n htt///oormuiitPPoo22e-
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of language model? In Proceedings of EMNLP, pp. 5418-5426, 2020.
12

---

Published as a conference paper at ICLR 2026
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated gated mixture-of-experr layer. In International Conference Learning Representations, 2017.
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulezyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature Medicine, pp. 1-8, 2025.
Chen Sun, Renat Aksitov, Andrey Zhmoginov, Nolan Andrew Miller, Max Vladymyrov, UIrich Rueckert, Been Kim, and Mark Sandler. How new data permeates Ilm knowledge and how to dilute it. arXiv preprint arXiv:2504. 09522, 2025. URL https: //tpiv/aaxivv.a//a org/ abs/2504. 09522.
Kai Sun, Yifan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-tail: How knowledgeable are large language models (Ums)? aka will Ilms replace knowledge graphs? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 311-325, 2024.
Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, al. Transformer memory differentiable search index. Advances in Neural Information Processing Systems, 35:21831-2188 2022.
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35:38274-38290 2022.
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Will we run out of data? limits of Ilm scaling based on numan-generater data, 2024. URL
https:/ //tp/aaxvvv///////////eee//aaav. org/ als/////// 2211 04325.
Jason Wei, Karina Nguyen, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. 2024.
arXiv preprint arXiv:2441.044
Xiao Yang. Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Gui, Ziran Jiang, Ziyu Jiang, et al. corgreoepeeeeniiee rag benchmark. Advances in Neural Information Processing Systems, 37: 17:10470-10990 2024a.
Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candès, and Tatsunori Hashimoto. Synthetic continued pretraining. arXiv preprint WxXiv:2409.07433 2024b.
Xiang Yue, Xinliang Frederick Zhang, Ziyu Yao, Simon Lin, and Huan Sun. Cliniqg4qa: Gen- erating diverse questions for domain adaptation of clinical question answering. In 202/ IEEE International Conference on Bioinformaties and Biomedicine (BIBM), pp. 580-587. IEEE, 2021.
Eric Zhao, Pranjal Awasthi, and Nika Haghtalab. From style to facts: Mapping the boundaries of knowledge injection with finetuning. arXiv preprint arXiv:2503.05919, 2025. URL https:
//arxivv org/ abs /200 05919.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. SIGIR, 2018.
A TRAINING DETAILS
The training runs from Table I and Figure 2 had the following hyperparameters: Trained for 20,000 steps, total batch size of 128 and sequence length of 4,096 with a constant learning rate of 10-5, We report the performance of the best checkpoint after 20,000 steps, evaluating every 2,000 steps.
The training data for the repeat method is the raw documents. For the scaling experiments, generated approximately 4 billion words for each method. The actual word count 3,486,858,22 for paraphrasing, 3,694,833,139 for synthetic QA, 4,000.262,090 for active read- ing task-agnostic and 4,000,330.802 for active reading task-specific. For Figure 2, each dataset was
13

---

Published as a conference paper at ICLR 2026
subsampled from 4B generated words to the indicated number of words. In all runs, mix in 10% of pre-training data from DCLM (Li et al., 2024), although we did not find it strictly neces- sary to prevent model degradation at the scale of these experiments and performance improves SimpleWikiQA by a couple of percentage points if we do not mix pre-training data.
B SCALING LAW TRAINING CONFIGURATIONS
For Figure 4, the following hyperparameters for each method. We vary the proportions of Active Reading-augmented SimpleWikiQA documents, Active Reading-augmenter Wikipedia documents, and generic pretraining data by varying the mixing weights. For all methods, we use sequence length of 4096 and batch size of 4, 4,144,304 tokens.
Table 4: Hyperparameters for scaling law experiments.
Fineture
80%
40%
20%
10%
5%
2.5%
Learning Rate
le-5
3e-4
3c-4
3c-4
3c-4
30-4
3e-4
Weight AR SimpleWikiQa
755.2
125.87
47.20
9.94
4.84
Weight AR Wikipedia
94.4
94.4
94.4
94.4
94.4
94.4
Weight Pretraining
94.4
94.4
94.4
94.4
94.4
94.4
AR SimpleWikiQA
100%
80%
40%
20%
10%.
5%
2.50%
AR Wikipedia
10%
30%
40%
45%
47.50%
48, 18.75%
Pretraining
30%
47.50%
48.75%
# Steps on SimpleWakiQA
5000
5000
5000
5000
5000
5000
5000
Steps
Total Tokens
2.10E+10
2.62B+10
5.24E+10
1,05++11
2,10E+11
4.19E+11
8.39E+11
C PERFORMANCE OF WIKIEXPERT ON GUARDRAIL TASKS
Table 5: Benchmark performance of WikiExpert 8B and Llama3.1 8B on guardrail tasks
Model
OBQA
PIQA
HellaS.
Winog.
ARC-e
RACE-m
MMLU
BOOLQ
GSMSk
MBPP
Wikitixpet
44.4
75.1
78.4 81.5
83.1 83.4
48.6
LlaMA3.1 8B
45.8
81.0
80.7
74.3
65.3
66.4
54.4
48.0
HellaSwag Accuracy
0.78
0.77
Figure 7:
Drop and recovery of performance for guardrail task HellaSwag, mirroring the drop and recovery on NaturalQuestion seen in Figure 4b.
D GENERATION PROMPTS
For all generation, {chunk} is replaced by the corresponding document that we are trying to aug- ment.
14

---

Published as a conference paper at ICLR 2026
D.1 PARAPHRASING
Use the information in the following snippet to write an informational paragraph in your own words. Make sure to cover all the information, including all entities, dates and places the original document. Do not add additional material. Directly output the paragraph and nothing else, c.g. do not say "Hore's s the paragraph".
<document>
{chunk)
</document>
D.2 SYNTHETIC QUESTION ANSWER
Genorate a comprehensive list of all questions and corresponding answers that be answered from this document. Make all entities (including people, dates and locations) covered.
Output one question-answer pair Line, answer seperated from the question by single space. Questions should be unambiguous, have proper capitalization and a question mark. The answers should be as concise as possible. Do not output anything additional, only output question-answes pairs (do not start with "Heer are the questions and ...").
Example lines output: What is the native range the Eurcpean fan worm? The northeastern Atlantic Ocean, the North Sea and the Mediterranean Sea. What countrics are in the European fan worm' s native range? The United Kingdom, Ireland, France, Spain, Pertugal, Italy, Greece, and Turkey. Is the European fan worm found in South America? Yes.
(...)
<document> (chunk </document>
D.3 ACTIVE READING
For active reading, in addition to {chunk} the model is given a strategy. In the following prompt, (strategy) is replaced an active reading strategy. The prompts used create the strategy are in Section D and D.3.2
Here's s learning strategy: (strategy)
Apply this strategy to the follewing document: <document>
(chunk)
</document>
D.3.1 TASK AGNOSTIC STRATEGY GENERATION
Consider the following document. What some strategies specific to this cocument that can use to help me learn and remember all the informat in contained ined? Jse markdown and prefix each strategy with #4
<document>
[chunk)
</document>
15

---

Published as a conference paper at ICLR 2026
D.3.2 TASK-SPECIFIC STRATEGY GENERATION
need to study for a trivia competition. Generate a list of questions that covers every piece of information in this document. After generating all the questiens, for each questien, generate a general study strategy prompt that would help memerize that kind of information (without focusing too much on the particular question). The prempt should outline detailed set of guidelines ines or step-by-step for how should rehearse or exercise the information to most effectively internalize it.
Output all the questions, then <start strategies>, then all the strategies. Prefix each strategy with #t.
<document>
(chunk)
</document>
16


Reviews:
[{"review_id": "1bWQZbDAIL", "reviewer": "Reviewer_yGFJ", "summary": "This paper presents a simple, effective method for generating corpora for continued pretraining. Their method first involves first prompting the LLM to generate a large set of learning strategies, which specify methods of paraphrasing or summarizing information from a source document. LMs then use such strategies to generate synthetic corpera for pretraining. The authors apply this method to wikipedia and financial analysis documents, demonstrating gains on wikipedia and finance-based QA tasks. The authors then perform extensive analysis, looking into scaling behavior, the effect of training hyperparameters (learning rate, data mix ratios), and the impact of diversity in the synthetically generated corpora.", "questions": "1. While performance seems consistent across most guardrail tasks in Table 5, is there any explanation for why performance drops so much on GSM8k and MBPP?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method is straightforward and demonstrates significant gains compared to baseline methods, in particular for the 8B model.\n\n2. The analysis is thorough and examines a variety of questions about their proposed method. Each analysis section provides valuable insights into why their method works and under what settings it does.", "weaknesses": "1. The results with the 70B model have substantially smaller improvements when compared to the 8B -- improving accuracy on SimpleQA <2% versus ~60% according to Table 3. Given this dramatic difference, it would be helpful to see the full results from the primary settings and some of the analysis repeated with this model. Likewise, experimenting with another base model (non-llama) would also help substantiate these results.\n\n2. Including more randomly selected samples of both task agnostic and task specific strategies would help with understanding what the method is actually doing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "14Yj77zseH", "reviewer": "Reviewer_YUiv", "summary": "This paper introduces Active Reading, a novel framework designed to improve the factual reliability of large language models. The problem is that LLMs often struggle to learn and recall facts, especially from the long tail of their training data. The proposed method has a model \"study\" a given corpus by first generating a diverse set of learning strategies specific to a document and then applying those strategies to create varied synthetic training data. This process is inspired by human learning techniques. The authors demonstrate that this method substantially improves factual recall on expert domain benchmarks, showing a 313% relative gain on a subset of SimpleQA and a 160% relative gain on FinanceBench compared to standard finetuning. They also scale this approach to create WikiExpert 8B, a model trained on 1 trillion synthetic tokens that surpasses the factual accuracy of much larger models like DeepSeekV2 and Llama 3.1 405B.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The Active Reading method is intuitive, scalable, and presents a clever way to generate highly diverse synthetic data by leveraging the model's own capabilities.\n2. The empirical results are extremely strong, particularly the performance of the 8B model on Simple WikiQA which nearly matches the gold context baseline.\n3. The release of WikiExpert 8B is a significant contribution, as it achieves state of the art factual recall for its size class and provides a powerful, compact model for fact intensive tasks.\n4. The scaling analysis in Section 4.2 is very insightful. The discovery that mixing in general pretraining data and using a higher learning rate is necessary to prevent degradation when scaling the knowledge corpus is a valuable finding for the community.", "weaknesses": "1. While the method excels at information extraction, its performance on the full FinanceBench benchmark is notably weaker than the synthetic QA baseline. This suggests the generated strategies may not adequately cover complex reasoning, a point the paper acknowledges but does not fully resolve.\n2. The finding in Table 3 that data generated by a 70B model leads to worse performance for an 8B model than its own self generated data is highly counterintuitive. This result is not deeply investigated and raises more questions than it answers.\n3. The evaluation is heavily focused on Wikipedia based corpora (SimpleQA, NQ, and the main WikiExpert model). While FinanceBench provides one alternative, demonstrating this method's effectiveness in another distinct domain like medicine or law would strengthen the claims of generality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0ORZIIKD2v", "reviewer": "Reviewer_oxz6", "summary": "- It is unclear why not simply input all of the four graph representations together as a baseline. Maybe the result will be better?\n- The paper lacks a comparison with human expert performance, even on a subset of the tasks. Including such results would help readers understand how far current LLMs have progressed in graph reasoning relative to human capabilities.\n- Table 4 shows that different models prefer different graph representations, yet the training of the Selector depends on a specific model. This implies that a new Selector must be re-trained each time when the model changes, which limits the generality and practicality of the proposed approach.\n- The description of the baseline in Section 4.2 is confusing. Does Vanilla prompting mean using the same workflow without additional training, or does it refer to using a single model without the agentic framework? Clarifying this distinction would make the experimental setup easier to understand.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The core idea of this work is both innovative and well-grounded. The proposed Active Reading framework, inspired by human learning behaviors, offers a conceptually sound and intuitively appealing approach to improving factual knowledge acquisition through self-generated data augmentation.\n\n- The work is methodologically solid, with clear experimental design, comprehensive ablation studies, and detailed scaling analyses. The authors provide meaningful comparisons with strong baselines such as paraphrasing and synthetic QA generation, which strengthens the empirical credibility of the findings.\n\n- The scalability and practical relevance of the method are convincingly demonstrated. Applying Active Reading at pretraining scale to develop the WikiExpert-8B model showcases its feasibility for integration into large-scale training pipelines and its potential for building more factual and efficient LLMs.\n\n- The work makes a notable empirical contribution by quantifying factual recall improvements across both general and expert domains. The consistent performance gains and smooth scaling behavior suggest that the proposed approach captures underlying principles of robust knowledge integration rather than dataset-specific effects.", "weaknesses": "- In line 240, the authors state that they add mixed pre-training data to prevent model degradation, but they do not provide experiments to demonstrate the occurrence of such degradation.\n\n- The conclusion in lines 274–276 seems rather obvious. Since SimpleWiki can be viewed as representing long-tail knowledge, while the expanded dataset introduces new knowledge, training on new data may naturally interfere with long-tail knowledge retention. This outcome is not surprising.\n\n- Figure 3 is confusing. It is unclear why the curve for the 4× data condition stops at 5k, the same point as the original dataset.\n\n- The conclusion in lines 308–309 is unsurprising, as the model is trained on a related dataset. Recovering performance in this context seems expected and aligns with intuition.\n\n- The experiments in Section 4.1 are conducted only on the LLaMA-3-8B model, which limits the generalizability of the results.\n\n- Lines 242–250 contain several statements that lack proper citations.\n\n- There are a few typos: in line 235, EntiGraph (Yang ...) is missing a space, and in line 308, NaturalQuestions (Kwiat...) is also missing a space.\n\n- In line 288, it is unclear why the learning rate changes from 1e-5 to 3e-4 instead of to 5e-4 or 1e-4. The rationale for this specific choice is not explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3vs6Nok8hf", "reviewer": "Reviewer_2FjT", "summary": "The paper propose Active Reading: a two-stage synthetic data pipeline where the model first generates diverse learning strategies for a given source document (e.g., timelines, analogies, Q&A, active recall prompts), and then applies each strategy to produce multiple augmentations used for training.", "questions": "1. The paper concedes that modern RAG approaches approach gold-context ceilings on these benchmarks. Given (i) the massive synthetic generation + continued pretraining cost and (ii) the modest gains on non-adversarial QA, why is parameterizing all this knowledge inside weights more efficient than maintaining a high-quality retriever+index?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1. The method is clear: first make learning strategies for each document, then apply them to create training data. It covers paraphrase, Q&A, and simple concept linking. It works for general or task-specific goals.\n2. The empirical gains are large. On SimpleWikiQA the score reaches about 66 percent. On FinanceBench the score reaches about 26 percent. Both beat repeat, paraphrase, and synthetic-QA under the same token budget.", "weaknesses": "1. When the document pool expands beyond the target subset, performance on the target recall task drops materially. The proposed “fix” (higher LR + heavier pretraining-data mix) is effectively a training patch, not a property of the method. If simply adding more relevant-but-broader documents hurts recall, then Active Reading does not positively scale in the straightforward way the paper implies.\n2. Most wins of the proposed method are on long-tail factual recall (e.g., SimpleQA subsets). Improvements on broader QA (NQ/TQA) are incremental. There is little evidence that Active Reading scales to domains requiring multi-hop, compositional reasoning, or other forms of tasks like math reasoning or coding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)