Reusing Pre-Training Data at Test Time is a Compute Multiplier
Authors: Alex Fang
Thomas Voice
Ruoming Pang
Ludwig Schmidt
Tom Gunter
Venue: ICLR 2026 Poster (2026)
Keywords: data
datasets
pretraining
pre-training
retrieval
llm
llms
test time compute
Primary area: foundation or frontier models, including LLMs

Abstract: Large language models learn from their vast pre-training corpora, gaining the ability to solve an ever increasing variety of tasks; yet although researchers work to improve these datasets, there is little effort to understand how efficient the pre-training apparatus is at extracting ideas and knowledge from the data. In this work, we use retrieval augmented generation along with test-time compute as a way to quantify how much dataset value was left behind by the process of pre-training, and how this changes across scale. We demonstrate that pre-training then retrieving from standard and largely open-sourced datasets results in significant accuracy gains in MMLU, Math-500, and SimpleQA, which persist through decontamination. For MMLU we observe that retrieval acts as a ~5x compute multiplier versus pre-training alone. We show that these results can be further improved by leveraging additional compute at test time to parse the retrieved context, demonstrating a 10 percentage point improvement on MMLU for the public LLaMA 3.1 8B model. Overall, our results suggest that today's pre-training methods do not make full use of the information in existing pre-training datasets, leaving significant room for progress.

Published as a conference paper at ICLR 2026
REUSING PRE- DATA AT TEST TIME IS A COMPUTE MULTIPLIER
Alex Fang2.3 2.3 Thomas Voice'
Ruoming Pang? Apple work done while at Apple $Stanford
Ludwig Schmidt
Tom Gunter?
ABSTRACT
Large language models learn from their vast pre-training corpora, gaining the ability to solve an ever increasing variety of tasks; yet although researchers work improve these datasets, there little effort understand how efficient the pre- training apparatus is at extracting ideas and knowledge from the data. In this work, way
we use retrieval augmented generation along with rest-time compute quantify how much dataset value was left behind by the process of pre-training, and how this changes across scale. We demonstrate that pre-training then retrieving from standard and largely open-sourced datasets results in significant accuracy gains in MMLU, Math-500, and SimpleQA, which persist through decontamination. For MMLU we observe that retrieval acts training alone. We show that these results can be further improved by leveraging additional compute at test time to parse the retrieved context, demonstrating a 10 percentage point improvement on MMLU for the public LLaMA 3.1 8B model. Overall, our results suggest that today's pre-training methods do not make full use of the information in existing pre- vee-training datasets, leaving significant room for progress.
5x compute multiplier versus pre-
INTRODUCTION
Large language models (LLMs) have consistently improved performance by scaling pre-training compute (Hestness al., 2017: Hoffmann et al.) [022]] Kaplan al.) 2020). In parallel scaling, researchers have also made significant efforts to improve both the architectures and the datasets used by these LLMS (Gu Dao 2023 Li al. 2024 Raffel et al. 2020) (Shazeer et al. 2017). While L.LMs able to solve incredible number of tasks, their current form they have several limitations. For example, they struggle with long-tail knowledge (Kandpal et al. 2023) and have limitations in their ability to generalize, such as for the reversal curse (Berglund et al.] 2023). Additionally, they have been observed to have a log-linear scaling trend, meaning that larger amounts of compute are necessary make the same gains larger scales. To understand whether these limitations come from the quality of the datasets, it is important to explore whether further improvements can be unlocked by using them beyond pre-training.
Taking ad advantage of the effort put into creating these sophisticated pre-training datasets, we explore whether reusing them through retrieval at test time can further improve performance. Additionally, we test if supplementing with additional test-time compute results in further gains. Extra test-time compute can be applied naturally to retrieval augmented generation by running multiple trials for self-consistency while changing the retrieved documents across different trials. We apply this variety of techniques on a set of publicly available pre-training datasets, to measure the potential impact of the knowledge contained in them.
First, we pre-train models at various compute budgets and then use the same dataset for retrieval. We that retrieval further benefits models when evaluating MMLU, Math-500, and SimpleQA, even though the models were pre- mre-trained on the same data. We fit a power law to the base pre-training models across compute budgets and compare against the gain from retrieval, showing that while on average retrieval provides a ~ 5x compute multiplier over pre-training when evaluating on MMLU, the effectiveness degrades with scale.
Next, we use additional test-time compute on the pre- pre-training data with a combination of retrieval and self-consistency techniques. We evaluate our methods variety of downstream tasks that cover

---

Published as a conference paper at ICLR 2026
multiple domains, requiring both knowledge and reasoning abilities. With Llama 3.1 8B reader model, we use retrieval to achieve 74.0% on SimpleQA, as well as a 10.5 percentage point gain on MMLU, 15.7 percentage point gain MATH-500, and a 6.2 percentage point gain GPQA.
Lastly, we analyze our findings to provide suggestions for further improving these datasets. We are able to relate performance gaps between controlled experiments back specific stages of the dataset creation pipeline. Altogether, our work suggests that there is room for improving both pre-training datasets and learning methods using these datasets.
2 RELATED WORK
Classical pre-training scaling law studies established relationships for how loss falls with training compute, data, and parameters (Hestness et al., 2017 Kaplan et al., 2020 Hoffmann et al. (002). More recent analyses factor in deployment costs in high- fech--ffrrenneeeeeee demand settings Sardana et al.
(2024).
Meanwhile, retrieval-augmenne approaches externalize knowledge non-parametrie memory (Guu et al. 2020 Lewis et al. 2020] Ram et al. 2023), trading additional test time compute for improved performance knowledge-intensive tasks. More recently, (haoo al. (2024) demonstrated that scaling the datastore can reliably further improve performance on knowledge-intensiee tasks, while Lyu et al. (2025) demonstrated that compact subset of pre-training data can be used in practical way with minimal retrieval setup to improve performance on reasoning benchmarks.
Recent work (Brown et al.) 2024 Snell et 2024) has shown that scaling test-time compute can be an efficient way of improving LLM performance, Specifically, they suggest parallelizing inference compute across trials, and sequentially iterating on model's output. Results can then be aggregated with techniques like self-consisg consisensst (Wang et al.] [022]] Chen et al.) (2023) or verifiers. Retrieval naturally suited to both as can be parallelized across retrieved documents, while sequentially iterating over the search query improve the retrieved documents.
Large-scale commercial "Deep Research" " systems from companies like Google, OpenAI, and Per- plexity likely apply all of the above by abstracting retrieval behind tool-use APIs and allowing the model direct control over the tools along with additional test-time compute.
In this paper, we first take inspiration from the classical pre-training scaling law studies, and begin to characterize the joint scaling of pre-training and simple retrieval given a fixed and identical data corpus. We also explore which forms of simple test-time compute allow us to most effectively leverage the retrieval stores, and whether these conclusions generalize to settings where the pre- pre-trannin corpus differs from the retrieval datastore.
EXPERIMENTAL SETUP
3.1 DATASETS
We use the exact same datasets for both pre-training and retrieval. We include the standard webcrawl based large-scale datasets DCL.Mbaseeine and WineWed (deduplicated versions for both), fol- lowed by more specialized sources. These include arXiv, peS2o, PubMed Central, Stack Exchange, and Wikipedia. Lastly, we include AlgebraicStack, AutoMathText, FineMath-3+, FineMath-4+, Open WebMath, and StackMathQA to improve mathematics coverage. Additional information on the datasets such as token count and pre-training mixing ratios can be found Table
3.2 RETRIEVAL PIPELINE
We use Qwen3 Embedding 0.6B and Qwen3 Reranker 0.6B (Zhang et al., 2025a) as the embedding and reranker models. We use FAISS FlatP (Johnson et al. 2019) for indexing to retrieve the top 100 documents per dataset (or shard of dataset) before combining across all datasets through sorting by similarity score. This is equivalent to having all datasets in the same index and retrieving the top 100, but is more practical due to compute constraints. Then if applicable, 100 documents across all datasets to achieve our final document ordering. Lastly, we do retrieval augmented generation by concatenating and prepending the relevant documents before the question.
rerank the top
2

---

Published as a conference paper at ICLR 2026
Table 1: Pre-training datasets used. Tokens refers to the actual size of the dataset, all of which is typically used during retrieval. Epochs refers to the number of epochs taken for the largest pre-training run. Epochs for smaller models are scaled down proportionally.
Dataset
Tokens
Epochs
Web Crawl Based
DCLM-baseline (dedup) (Li et al. 2024)
764.9B
0.88
FineWeb-edu (dedup) (Penedo et 2024)
197.6B
0.80
Additional Sources
arXiv
28.7B
1.10
peS2o (Soldaini & Lo 2023
71.9B
1.31
PubMed Central
22.5B
2.72
Stack Exchange Wikipedia
11.8B
7.98
21.7B
3.63
Math
AlgebraicStack (Azerbayev et al.
2023
9.9B
6.38
AutoMathText (Zhang et al. 20256
6.0B
13.11
FineMath-3+ (Allal et al. 2025
36.5B
1.72
FineMath-4+
Alla et al. 2025
10.0B
11.05
OeenWebMath (Paster et al. 2023
13.5B
2.91
StackMathQA (Zhang
2024
0.7B
558.67
4 PRE-TRAINING EXPERIMENTS
0.55
0.50
Math500 (4-shot CoT)
0.6
MMLU (M (S-shot)
0.45
0.70
Simple0A
0.40
0.65
0.60
0.55
0.50
0.0
102z
10
10F2
10a3
102
1022
FLOPs
FLOPs
Decontaminated Retrieval
Retrieval
Figure I: Retrieval on the pre-training dataset can substantially improve upon the performance of the base model. However, the exact benefit depends on the type of task.
We aim measure the knowledge contained in pre-training datasets by first pre-training on it, then also retrieving (with reranker) on it during test time. In Figure we measure performance on MMLU (Hendrycks et al. (2020), Math-500 (Lightman al., 2023), and SimpleQA (Wei al. (024] across compute budgets, comparing the base model with retrieval on all our datasets, retrieval decontaminated version of all our datasets, and retrieval on subset approximately equivalent the unrepeated pre-training dataset. Overall, find that retrieval can help all three tasks, though to different degrees.
Retrieving from the full dataset leads to large gains all tasks, but the model sees more data than the base model sees in pre-training. We account for this by retrieving from a subset similar to the unrepeated pre-traine pre-training dataset, represented by the light blue lines in Figure [ Interestingly, this achieves similar performance to retrieving on the full dataset for MMLU and Math-500. On the other hand, performance SimpleQA is piecewise function because SimpleQA depends heavily

---

Published as a conference paper at ICLR 2026
retrieving from Wikipedia, and the smaller two models only see fraction of Wikipedia while the larger three models see at least one epoch of Wikipedia.
Retrieval as compute multiplier Given the common use of MMLU as a proxy for pre-training quality, we use it to measure the effect of retrieval as a compute multiplier for the base model. We fit a bounded sigmoid function to the base model's MMLU performance as function FLOPs, and then measure the amount of pre-training compute needed to match each existing base model augmented with retrieval. In Table 2 we find that the average compute multiplier across the five models of different scales is 4.86x, and the geometric mean is 4.66x. However, this compute ratio decreases as the model scales, with retrieval providing only 2.88x compute multiplier at the largest scale. While retrieval provides large compute savings, the base model is scaled up. retrieval faces greater diminishing returns than just the base model. It is important to notice that retrieval does not provide a flat or strictly decreasing benefit across compute budgets. There is an initial increase in retrieval efficiency, suggesting that it benefits from better base models.
Table 2: We fit the base model performance to sigmoid function with bounds to get y = 0.25
0.6907
where 0.25 is the random baseline
+ cxp( - 0.7968 (z) - 1o (2.48 x 102
and 0.9407 the maximum achievable accuracy (Gema al. 2024). We use this equation to measure retrieval as a compute multiplier for the base model. The average compute ratio is 4.86x, the geometric mean is 4.66x, and the median is 4.74x.
Baseline MMLU
Retrieval MMLU
Compute for base to match retrieval
Compute Budget
Compute Ratio
5.64 1021
0.4873
0.6063
2.98 1022
5.28x
1.90 x 1022
0.6021
0.6943
1.36 x 1023
7.17x
7.04 X 10a2
0.6623
0.7410
3.34 x 102s
4.74x
1.74 102s
0.7107
0.7775
7.35 x 1023
4.23x
7.34 1023
0.7633
0.8186
2.11 x 1021
2.88x
Decontamination A common question is whether retrieval gains come from retrieving text con- taining exact overlap with the test data. We decontaminate the retrieved documents for MMLU and Math-500 through n-gram overlap with the questions, detailed in Appendix B In Figure 1, the red decontaminated retrieval line is close to the dark blue full retrieval line for MMLU, demonstrating that the gains not attributable simple contamination. Although Math-500 shows signs of more significant contamination, retrieving against a decontaminated training set still shows a very meaningful improvement over the baseline. We do note that analysis shows that 14.1% 1% of MMLU and 32.0% of Math-500 can be found in our commonly used open-source pre-training datasets, highlighting the importance of strictly decontaminated (or held out) evaluation sets for pre-training science. We choose to omit n-gram overlap decontamination analysis for SimpleQA due to the nature of the evaluation task.
4.1 LEARNING FROM PRE-
PRE-TRAINING TRAINING VS RETRIEVAL
In an effort to determine how retrieval can improve model performance, compared scaling up model size and compute budget, we analyzed the accuracy of the trained models on MMLU, broken down into question categories. The results in Figure 2 show that across categories, the addition of retrieval gives comparable boost in accuracy to that of significant increase in pre-training compute budget.
Since retrieval involves memory storage mechanism, we might expect it to provide most benefit for problems requiring good recall of facts, rather than reasoning abilities. However, in Table we see that retrieval is better compute multiplier for STEM than for humanities
social sciences, and in Figure 2 the gap between retrieval and base accuracy is also wider for STEM than for humanities. Such knowledge may be harder to absorb during pre- training, and in contrast to long tail facts in SimpleQA, it may be possible that retrieval expanding the context may also function as additional processing rather than just storage.
4

---

Published as a conference paper at ICLR 2026
Table 3: Summary of pre-training vs retrieval compute ratios across MMLU categories. Values show how many times more compute the base model would need to match retrieval performance. Calculated using category-spece bounded sigmoids (min 0.25; max: STEM 0.9544, Humanities 0.9377, Social 0.9575, Other 0.91 14, All 0.9407).
Compute Budget
STEM
Humanities
Social
Other
All
5.64 10a1
6.82x
2.69x
3.42x
9.80x
5.28x
1.90 10a2
10.23x
3.33x
4.42x
13.78x
7.17x
7.04 X 1022
5.23x
2.57x
4.07x
8.77x
4.74x
1.74 1023
5.01x
2.44x
3.43x
7.53x
4.23x
7.34 1023
3.52x
1.55x
2.22x
6.48x
2.88x
Average
6.16x
2.52x
3.52x
9.27x
4.86x
Geometric Mean
5.78x
2.44x
3.42x
8.96x
4.66x
Median
5.23x
2.57x
3.43x
8.77x
4.74x
10
SUbJCCE grous
Rariewo sctup Bend dul.adi
Socill Suenus
Other
0.8
0.7
0.6
0.4
4.5E 21
1.6E 22
6.0E 22
1BE 22
7:36
Compute (PLOP))
Figure 2: MMLU Breakdown by category of impact of retrieval addition and compute budget. Retrieval provides a strong lift, and the difference between retrieving from random subset of the data store and the full set is small and diminishing with scale.
To investigate this surprising observation, we calculated the increase in MMLU accuracy provided by full retrieval, for different subject areas, with 6.4B model. The top also include the corresponding accuracy increase provided by subset retrieval. Here we can see a mix of differrnt subject types including those that might be expected to require good knowledge recall, such as law and medicine, along with those that require computation, such as physics. A similar mixture of types of subjects may be seen for other model sizes, suggesting no strong correlation between subject type, model size, and the benefits of retrieval.
shown in Table We
Table 4: Top ten categories ordered by change in MMLU accuracy after introduction of full retrieval
Subjects
Impact full retrieval
Impact subset retrieval
Medical genetics
+21.1
+19.
Miscellaneous
+ +19.2
+ 17.8
World religions
+18.5
19,6
Philosophy
+17.9
+17.1
US foreign policy
+17.7
+16.1
International law
+17.0
+17.3
High school physics
+16.9
5.7
Virology
+16.9
+15.2
College physics
+16.8
+14.2
College medicine
+16.6
+11.8
5

---

Published as a conference paper at ICLR 2026
We also compared the MMLU answers between the 6.4B and 12.6B models, with and without retrieval, focusing on problems where the smaller base model gave an incorrect answer. We identified problems where the addition of retrieval corrected the answer, but the increase in model size did not, and vice versa. The main subjects of the former group were professional law, professional psychology. high-school macroeconomics, philosophy, and high-school mathematics. The main subjects of the latter group were professional law, professional psychology, moral scenarios, clementary mathematics, and high school statistics. Both groups have a mix of recall and reasoning problems, with much overlap. This suggests that there is not strong bias for what kinds of problem retrieval can help with compared increasing model size.
In total. increasing model size changed answers for 39.7% of MMLU problems, whereas adding retrieval changed answers for 28. 1% of the problems. similar analysis for MATH-500 showed that increasing model size changed answers for 39.7% of the problems, whereas adding retrieval changed answers for 28.7% of the problems. This suggests retrieval overall has less of an effect on model behavior than increasing model size. Furthermore, it suggests that in areas where retrieval does not help, the problem more that the model ignores the additional context, rather than the additional context being misleading. Future work could involve encouraging models to better utilize retrieved context, possibly through prompt engineering or attention weighting.
TEST-TIME COMPUTE EXPERIMENTS
Table 5: Comparing baseline reader model performance against retrieval with variety of test-time
compute options. All evaluations use shain-of-thoughh reasoning. We use Llama 3.1 8B instruct as the reader model. MMLU results are reported as macro average over subjects. VR refers using variance reduction techniques such as MMR and bagging.
MMLU STEM
MMLU Humanities
MMLU Social
MMLU Other
MMLU
Method
All
Baseline
67.3
71.5
76.6
73.0
71.6
w/ self-consistency
72.3
74.8
79.3
76.4
75.3
w/ retrieval
73.6
74.6
81.8
77.6
76.6
w/ reranker
73.7
76.3
83.4
79.2
77.7
w/ reranker self-cons.
78.7
78.7
85.8
81.9
81.0
w/ reranker self-cons. VR
80.2
79.5
87.4
82.3
82.1
Table 6: Continuation Table
GPQA, and Math-500 results are over 10 trials.
GPQA Bio.
GPQA Chem.
GPQA Phys.
GPQA
Method
SimpleQA
Math-500
All
Baseline
1.5
48.7
46.2
26.4
28.3
30.6
w/ self-consisenncy
N/A
55.9
46.3
28.1
28.4
31.4
retrieval
65.7
56.7
45.1
27.3
34.0
33.2
w/ reranker
74.0
56.8
46.7
28.6
36.0
34.8
reranker + self-cons.
N/A
64.3
48.5
30.1
36.8
36.1
w/ reranker + self-cons. + VR
N/A
64.4
49.7
29.6
38.3
36.8
Knowing the limitations of learning with just pre-training, we attempt to better quantify the knowledge contained in these datasets by applying additional test-time compute on top of retrieval. If the model able to answer a question with retrieval and test-time compute, the knowledge required to do so is likely in the dataset. In this section, use Llama 3.1 8B instruct (Grattafiori et al. 2024) as the reader model due to its performance relative to its size, making it more practical to apply test-time compute. We then augment it with retrieval described in Section 3 and parallel inference with majority voting to select an answer selff-onsistenny). addition to evaluating on MMLU, SimpleQA, and Math-500, also evaluate GPQA (Rein et al. 2024).

---

Published as a conference paper at ICLR 2026
Table 5 and Table 6 show that the effects of self-connistency and retrieval additive across all tasks, with the exception of SimpleQA where consisteey does not help because it is purely a factuality benchmark. Perhaps surprisingly, the two techniques help generally across all other tasks and sub-tasks, with little hint of specialization. Additionally, reranking seems to give a consistent boost top of retrieval across tasks. Lastly, take advantage of retrieving multiple documents and parallelizing trials by using older techniques like MMR (Carbonell & Goldstein, 1998) to increase diversity, and bagging (Breiman 1996) (randomizing over subset of documents) to reduce variance; these techniques give further performance boost for MMLU and GPQA.
we view retrieval as tool for the LLM, then our methods use test-time compute to improve the tool itself. This contrasts with self-consistency by itself. which parallelizes the model without additional enhancement, as well as with deep research, which in addition to parallelizing also uses test-time compute to use the tool for longer rather than to upgrade it. Retrieval is our vehicle through which we can put in additional compute, and do it in a data driven way.
5.1 LEARNING FROM TEST-TIME COMPUTE VS PRE-TRAINING
Table 7:
Compute efficiency gains for each method relative to baseline performance. Values represent
how many times more compute the baseline model would need to achieve the same performance as each method. Calculated using fitted sigmoid equations for each MMLU category.
MMLU
MMLU Humanities
MMLU Social
MMLU
MMLU
Method
STEM
Other
All
Baseline
1.00x
1.00x
1.00x
1.00x
1.00x
w/ self-consistency
2.62x
1.93x
1.63x
2.21x
2.10x
w/ retrieval
3.42x
1.85x
2.70x
3,02x
2.78x
w/ reranker
3,49x
2.67x
3.87x
4.72x
3.56x
w/ reranker self-cons.
10.74x
4.65x
7.18x
11.34x
8.14x
w/ reranker self-cons. + VR
15.72x
5.68x
11.66x
13.15x
11.10x
We can take the MMLU sigmoid fit from Section to analyze Table 5 Although Llama 3.1 8B is trained at much higher tokens per parameter ratio than the models in Section 4 the sigmoid fit could still be reasonable because calculating the compute ratio between retrieval (with reranker) and base performance is within reason for MMLU (All). Additionally, estimates using our previous sigmoid fits would be lower bound for compute multipliers because of diminishing returns at higher tokens per parameter counts.
Despite the different pre-training dataset and Llama being significantly overtrained, we see that retrieval (with reranker) still functions as 3.56x compute multiplier, similar what would be expected of model with the same MMLU base accuracy in our pre-training setup in Section 4 Though we do not know the details of the Llama 3.1 pre-training dataset, likely that there substantial overlap with the data we are retrieving from. Table shoow that, altogether, our methods provide at least an 1lx compute multiplier over the pre-trained baseline.
We also see that the different test-time methods learn utilize data differently from pre-training, as multipliers are different across the categories. Even the different test-time methods have different behaviors, both self-conse caff-consitenccy and retrieval favor STEM and other, while the lift of reranker over retrieval favors humanitics, social sciences, and other.
5.2 A CONNECTION BETWEEN RETRIEVAL AND CONSISTENCY
While the previous results in this section demonstrate that self-consistency is a powerful tool for improving performance, it can also used as an analytical tool for retrieval. As displayed in Appendix H. we can apply self-consistency with it. In Table g we see that inter-document consistency selects better top-1 documents than the reranker. However, this technique also requires multiplicative number of additional trials, as previously we ran a fixed number of trials on all documents combined, but now we are doing it per document. We leave to future work ways distill seff-consistency into efficient reranker.
each individual document and rerank the documents

---

Published as a conference paper at ICLR 2026
Table 8: Inter-document consistency can act as stronger reranker (k=1) than standard rerankers from /Zhan et al. (025a); however, it requires calling the reader model many more times and is not compute efficintt when compared to celf-consi on all documents once.
MMLU STEM
MMLU Humanities
MMLU Social
MMLU
MMLU
Reranker
Other
All
Qwen3 Reranker 0.6B
71.2
72.9
77.7
74.1
73.7
Inter-doc consistency
75.4
75.4
82.7
78.1
77.6
ADDITIONAL ANALYSIS
6.1 BETTER PRE-TRAINING DATASETS ARE NOT NECESSARILY BETTER RETRIEVAL DATASETS
Table While FineWeb-edd is worse than DCLM when measuring pre- tre-iraining performance, is just as good if not slightly better for retrieval. Retrieval is top of Llama 3.1 8B instruct with k=10, Pre-training numbers are 8B models trained for 1T tokens, as reported by Su et al. (2024)
Pre-training MMLU
Retrieval MMLU
Retrieval w/ reranker
Dataset
MMLU
DCLM
53.4
74.5
76.4
Fine Web-ede
42.9
75.2
76.6
Our retrieval datasets were built for the purpose of pre-training, which raises the question of whether better pre-training datasets make for better retrieval datasets. Table 9 suggests that this is not necessarily the case, as FibeWeb-edu is worse than DCLM for pre-training, but is as good if not slightly better for retricval. While DCLM contains more tokens than Fib-Weu, prior work (Muennighoff et al. 2023b Fang et al. (025) would suggest that this is not the reason for the gap in pre-training performance, while the size advantage should not be harmful for retrieval. We leave to future work how determine the qualities that make dataset good for pre-training retrieval specifically.
6.2 IMPORTANCE OF EXTRACTION AND CRAWLING
Table 10: Text extraction and crawling are important for creating good datasets. We vary the extraction done on top of Wikipedia, as well as how we expand the datastore. Retrieval uses reranker and is on top of Llama 3.1 8B instruct with k=6.
Dataset
SimpleQA
Wikimedia Nov. 2023
55.4
OLM June 2025
59.1
Custom June 2025
69.0
Custom + All Sources
73.7
Custom + Golden Links
85.2
We investigate the importance of earlier stages of dataset creation through case study of retrieval for SimpleQA. As constructed, over 70% of the answers in SimpleQA can be found on Wikipedia. However, many works use out-of-date or pre-extracted versions of Wikipedia. This can lead to missing data, especially when the crucial piece of information comes from specialized elements.
We compare Wikimedia (Nov. 2023) (Wikimedia- Fontiiatiii-ouuaaiiiionno and OLM (June 2025) (Thrush et al. 2022), two of the most popular Wikipedia extractions HuggingFace, against custom extracted Wikipedia (June 2025) described in Appendix C Qualitatively, we find that existing extractions often fail to extract elements like bullet points, tables, and info boxes. Table [oobeiIaauttiiiaiiii demonstrates this with up to a 13.6 percentage point difference in SimpleQA performance by simply changing the version of the Wikipedia dataset.
8

---

Published as a conference paper at ICLR 2026
Next, compare expanding the retrieval datastore by adding other sources against adding non- Son-Wikipddia golden links provided by SimpleQA. Table [0 shows that there an 11.5 percentage point difference, and find that only small fraction of the non- non-Wikipedia golden links present in CommonCrawl. This suggests that open-source datasets could be further improved at the web crawling stage.
6.3 ROBUSTNESS WITEN SCALING RETRIEVAL DATA
0.8
SimpleQA
0.6
0.5
0.4
96B
1918
3628
7658
DCLM Dedup Tokens in Datastore
Custom Wikipedia Baseline
Custom Wikipedia + Golden Baseline
Full DCLM Baseline
Custom Wikipedia DCLM
Custom Wikipedia
Golden + DCLM
Figure 3: For SimpleQA, our retrieval system is fairly robust to scaling the retrieval datastore, even if new data does not contain useful information. Our custom Wikipedia contains 22B tokens, and additional DCLM data helps a little, when also starting with additional golden link data, hurts only a little.
Given that most of SimpleQA can be solved with Wikipedia and the additional provided golden links, we investigate the effect of additional web crawl data on the retrieval system. In Figure we see that the additional data has only small distracting effect, as the accuracy on SimpleQA stays close the baseline. However, we acknowledge that SimpleQA measures factual knowledge, and the scaling effect may be different for reasoning tasks.
7 FUTURE WORK
We have shown that pre-training does not fully utilize all the knowledge contained within today's open- source pre-training datasets. This would suggest that there are still many algorithmic improvements left to explore. Additionally, in our process of analyzing data quality, we have also shown that there is room for improving datasets, at the very least in terms of crawling and extraction.
Within this work, we explore limited number of simple test-time techniques on a limited set of evaluations. However, it is quite likely that applying advanced techniques like query rewriting, test-time training, and reinforcement learning for retrieval will further boost the performance on the same datasets (Hardt & Sun 2023; Ma al. 2023). We also believe that these findings apply broader domains. Initial results in Appendix G suggest that retrieving from pre-training datasets also benefits code generation.
Beyond improvements, we would also like to better understand how data is used during pre-training. Our measurement of retrieval compute multiplier parallels that of well-tuned Mixture-of ofre---e-ftttttt models (Clark et al. 2022). Additional exploration in this area could uncover whether these two methods have significant overlap in terms of data usage.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We would like to thank Sewon Min, Rulin Shao, and Anikait Singh for helpful conversations and feedback at various stages of the project.
REFERENCES
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner. Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathicu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big data-centric training of a small language model, 2025. URL htsttps://axii....tttt
abs 2502. 02737
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen McAleer, Albert Q. Jiang, Jia Deng. Stella Biderman, and Sean Welleck. Llemma: An open language model for mathematics, 2023.
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The reversal curse: Lims trained on" is b" fail to learn" b is a". arXiv preprint arXiv:2309. 12288, 2023.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-144 140, 1996.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21778 2024.
Jaime Carbonell and Jade Goldstein.
The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pp. 335-336, 1998.
Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin. Sushant Prakash. Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language model generation. arXiv preprint arXiv:2311.17311, 2023.
Aidan Clark, Diego Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pp. 4057-4086. PMLR, 2022.
Alex Fang, Hadi Pouransari, Matt Jordan, Alexander Toshev, Vaishaal Shankar, Ludwig Schmidt, and Tom Gunter. Datasets, documents, and repetitions: The practicalities of unequal data quality. arXiv preprint arXiv:2503.07877 2025.
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? arXiv preprint 4xX1v:2406.04117, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, arXiv preprint arXiv22312.00752 2023.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming Wei Chang. Realm: Retrieval- augmented language model pre-training, 2020. URL puttps://arxiv holg//bs//20 22022
2002
08909
Moritz Hardt and Yu Sun. Test-time training on nearest neighbors for large language models. arXiv preprint arXiv:2305. 0..2235518466 18466, 2023.
10

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 03300, 2020.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary. Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv: 15556, 2022.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07977 2024.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):33--555 547, 2019.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pp. 15696-157077 PMLR, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alee Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint wwXiv:2001008360 2020.
Mark Lee, Tom Gunter, Chang Lan, John Peebles, Hanzhi Zhou, Kelvin Zou, Sneha Bangalore, Chung- Cheng Chiu, Nan Du, Xianzhi Du, Philipp Dufter, Ruixuan Hou, Haoshuo Huang, Dongseong Hwang, Xiang Kong, Jinhao Lei, Tao Lei. Meng Li, Li Li, Jiarui Lu, Zhiyun Lu, Yiping Ma, David Qiu, Vivek Rathod, Senyu Tong, Zhucheng Tu, Jianyu Wang, Yonggiang Wang, Zirui Wang. Floris Weers, Sam Wiseman, Guoli Yin, Bowen Zhang, Xiyou Zhou, Danyang Zhuo, Cheng Leong, and Ruoming Pang. Axlearn: Modular large model training on heterogeneous infrastructure, 2025.
URL httttttss/aaxiii
hbtg:/bb//555 05411
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. detrieval-augmennt genera- tion for knowledge-intensive intensive nlp tasks. Advances neural information processing systems, 33: 9459-9474, 2020.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, et al. Datacomp-Im: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems, 37: 17:14200-14282, 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe, Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Anton Lozhkov, Raymond Li. Loubna Ben Allal, Federico Cassano, Joel Lamy- LamrePoirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder and the stack v2: The next generation. arXiv preprint 240. 2024.
Xinxi Lyu, Michael Duan, Rulin Shao, Pang Wei Koh, and Sewon Min. Frustratingly simple retrieval improves challenging, -nntesinnginteesivve benchmarks. arXiv preprint arXiv:2507.01297, 2025.
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting in retrieval- augmented large language models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5303-5315, 2023.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.09 07 2023a.

---

Published as a conference paper at ICLR 2026
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36:50358-500 50376, 2023b.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023.
Guilherme Penedo, Hynek Kydlfček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:308 1 1-30849, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. In-context relrieval-auggmented language models. Transactions of the Association for Computational Linguistics, 11:1316-1331, 2023.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling. 2024.
Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: accounting for inference in language model scaling laws. In Proceedings of the 41st International Conference on Machine Learning, ICML'2 24. JMLR.org. 2024.
Rulin Shao, Jacqueline He, Akari Asai, Weijia Shi, Tim Dettmers, Sewon Min, Luke Zettlemoyer, and Pang Wei W Koh. Scaling retrieval-based language models with trillion-token datastore. Advances in Neural Information Processing Systems,
2024.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gan gated mixture-of ex-eers layer. arXiv preprint arXiv: 06538, 2017.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling I1m test-time compute optimally can be more effective than scaling model parameters. arXiv preprint 2401::2200.033114, 2024.
Luca Soldaini and Kyle Lo. peS2o (Pretraining Efficiently on S2ORC) Dataset. Technical report, Allen Institute for AI, 2023. ODC-By, htttpps://gcccbb..ommaaaaaeeeeee
Dan Su. Kezhi Kong, Ying Lin. Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Nemotron-ce: Transforming common crawl into refined Long-horizon pretraining dataset. arXiv preprint arXiv:2412.02595, 2024.
Tristan Thrush, Helen Ngo, Nathan Lambert, and Douwe Kiela, Online language modelling data pipeline. https: //gi thub. com hwoom/huggiii ng face/o m-ddmaaatats... 2022.
Feng Wang, Zesheng Shi, Bo Wang, Nan Wang, and Han Xiao. Readerhm-v2: Small language model for html to markdown and json, 2025. /ttpppss/aaxx....cr////////3 2050000001
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11177, 2022.
Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality large language models. arXiv preprint arXiv:2411.04366 2024.
Wikimedia. Wikipedia structured contents. Available at: https: www www.kagw.e.com.mmmmmm datasets wetseesswwikie http//wiiiiiiaaaaammm wtw.aatiiio//kkieeeeeaatttttaata ort.r cessed: 2025-09-23). Aokimedia-Foundationn Wikimedia downloads. URL wttth::ttppppppsssssessspp wi wikimedii org
[Ac-
12

---

Published as a conference paper at ICLR 2026
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advanc- ing text embedding and reranking through foundation models. arXiv preprint arXtv:2506.05176, 2025a.
Yifan Zhang. Stackmathqa: A curated collection of million mathematical questions and an- swers sourced from stack exchange, 2024. URL http://s///arggnaa....ee//aaaaas math nath-a3/StackMatngg
Yifan Zhang, Yifan Luo, Yang Yuan, and Andrew Chi-Chih Yao. Autonomous data selection with zero-shot generative classifiers for mathematical texts. The 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025 Findings), 2025b.
13

---

Published as a conference paper at ICLR 2026
A PRE-TRAINING DETAILS
We follow a recipe very similar to that open-sourced as the "Honeycrisp" model series the Axlearn training framework Lee et al. (2025).
We train each dense decoder model for around 20 tokens per parameter, and follow a cosine-with linear-warmup learning rate schedule with a peak learning of le-2, decaying 0.01 of the peak Ir. As described in the "Honeycrisp" model definitions, we use a muP-style parameterization to achieve learning rate transfer as we scale up the models. The model architecture similar the LLaMA series, with Swi-GLU FFNs, RoPE positional encodings, and Grouped Query Attention (GQA) using a sey/value-to-query ratio of 1:8.
Compute Budget
Paramters
Tokens
5.64 x 10
6.4B
147B
1.90 x 10a2
12.6B
252B
7.04 x 1022
23.3B
503B
1.74 x 1023
36.8B
786B
7.34 x 1023
77.8B
1573B
B DECONTAMINATION
We perform n-gram decontamination against our test sets using n-grams in token-space (according to our tokenizer). We drop entire documents on collision with single 16-gram from the MMLU test-set or 26-gram for Math-500. Through visual examination we found that shorter n-gram overlaps were too aggressive (noting that our tokenizer e.g. splits numbers into single digits).
C CUSTOM EXTRACTION
implement custom HTML extraction pipeline and apply all pages from the Wikipedia domain found in our general web-crawl. Specifically, we first apply lightweight pre-processing step to remove script, style, unmatched meta tags, HTML comments, links, and images. We then use the ReaderLM-v2 (Wang al. (025)) to extract the coarsely simplified HTML into structured plain-text.
We note that this approach improves on recall (especially for tables and some iiformation-boxee. publicly available Wikipedia extractions, including the recently provided by the Wikimedia organization Wikimedia
D DETOKENIZE SUBSET VS RANDOM SUBSET
Table 1I: In Figure I we show the effect of retrieving from a subset similar to that seen in pre-training for that compute budget. Specifically, we take random subset that is the same size of what is seen during pre-training for each source. In this table, we compare at the smallest 5.64 102 compute budget between random subset and detokenizing the exact pre-training data and using that for retrieval. The results are fairly similar, and the larger gap in Math-500 may be attributed to randomness contamination. Note that this gap will shrink the compute budget increases because the two datasets will have increasing overlap.
Data
MMLU
Math-500
SimpleQA
Random subset
59.0
32.0
35.5
Exact subset
58.6
26.8
33.9
14

---

Published as a conference paper at ICLR 2026
E FITS BY MMLU CATEGORIES
Table 12:
Pre-training vs. retrieval compute ratios for STEM. Sigmoid fit (min=0.25, max=0.9544): 0.7044
0.25+1 +
Average compute ratio is 6.16x, geometric
exp( 0.7351 (log1o (z) 22.9965))
mean is 5.78x.
Baseline MMLU
Retrieval MMLU
Compute for base to match retrieval
Compute Budget
Compute Ratio
5.64 x 1021
0.4247
0.5493
3.85 x 1022
6.82x
1.90 x 1022
0.5367
0.6399
1.94 1023
10.23x
7.04 x 10a2
0.5788
0.6749
3.68 X 1023
5.23x
1.74 x 1023
0.6376
0.7197
8.71 x 1023
5,01x
7.34 x 1023
0.7056
0.7705
2.58 x 1024
3.52x
Table 13: Pre-training vs. retrieval compute ratios for Humanities.
Sigmoid (min=0.25,
0.6877
max=0.9377):
0.25 +
Average compute ratio
1 expl 0.8008 (log10 (z) 22.1259))
is 2.52x, geometric mean is 2.44x.
Baseline MMLU
Retrieval MMLU
Compute for base to match retrieval
Compute Budget
Compute Ratio
5.64 x 1021
0.5277
0.6013
1.51 x 10a2
2.69x
1.90 x 10a2
0.6205
0.6847
6.33 x 1022
3.33x
7.04 x 1022
0.6961
0.7398
1.81
1023
2.57x
1.74 X 1023
0.7420
0.7788
4.24 x 1023
2.44x
7.34 x 1023
0.7871
0.8169
1.14 x 1024
1.55x
Table 14: Pre-training vs. retrieval compute ratios for Social Sciences. Sigmoid fit (min=0.25,
0.7075
max=0.9575) y 0.25 + 1+ exp( 0.9563 (logı0 21.9772)) Average compute ratio is 3.51x, geometric mean is 3.42x.
Baseline MMLU
Retrieval MMLU
Compute for base to match retrieval
Compute Budget
Compute Ratio
5.64 x 10
0.5419
0.6555
1.93 x 10
3.42x
1.90 x 1022
0,6770
0.7538
8.39 x 10
4.42x
7.04 x 10
0.7538
0.8192
2.86
10
4.07x
1.74 x 1023
0.7983
0.8501
5.97 x 10
3,43x
7.34 x 1023
0.8375
0.8828
1.63 x 20
2.22x
15

---

Published as a conference paper at ICLR 2026
Table 15: Pre-training vs. retrieval compute ratios for Other. Sigmoid fit (min=0.25, max=0.9114)
0.6614
y = 0.25 + exp( 0.8008 (I) 22.2759)) Average compute ratio is 9.27x, geometric
mean is 8.96x.
Baseline MMLU
Retrieval MMLU
Compute for base to match retrieval
Compute Budget
Compute Ratio
5.64 x 1021
0.4834
0.6418
5.53 1022
9.80x
1.90 x 1022
0.6046
0.7222
2.62
1023
13.78x
7.04 X 10
0.6601
0.7598
6.17 X 102h
8.77x
1.74 x 1023
0,7006
0.7882
1.31 x 1024
7.53x
7.34 x 1023
0.7516
0.8271
4.76 1024
6.48x
MATH-500 CHECKER FOR USC
Table 16: Math-500 answers are open-ended so we use universal coff-consistency (USC) (Chen et 2023) in Table 6 with the reader model itself as the checker (Llama 3.1 8B). Here, we compare this against using GPT-4.1 mini as the USC checker model.
Llama 3.1 8B checker
GPT-4.1 mini checker
Method
Baseline
48.7
N/A
w/ self-consistency
55.9
62.2
w/ retrieval
56.7
N/A
w/ reranker
56.8
N/A
w/ reranker + self-cons.
64.3
69.7
w/ reranker + self-cons. + VR
64.4
71.8
G LIVECODEBENCH RESULTS
Table 17: Retrieving from the python portions of the Stack v2 (ozzhoo 2024 and CommitPack
(Muennighoff et al. 2023a to augment generation for LiveCodeBench Code Generation (Jain et al. 2024).
Model
Baseline
Retrieval (k=3)
gpt-4o-2024-08-66
0.3793
0,4276
16

---

Published as a conference paper at ICLR 2026
H INTER-DOCUMENT CONSISTENCY
Most Common Answer using
Score:
Most Common
Answer
using
Document
Score:
Document
Most Common
Pre-tr pre-trained Model times per doc
Answer using Score:
obtained using maxik, Xx)
Document
Most Common
Answer Score: x,/t
Figure 4: Inter-document consistency can be used to analyze retrieval and consistency. We apply self-consistency on generating while retrieving from individual documents, and select the answer from the most self-consistent document.
17


Reviews:
[{"review_id": "ydeIxzg9IT", "reviewer": "Reviewer_8hXC", "summary": "This paper studies how much existing LLM models utilize the pretraining data by letting models retrieve them during test time. The result shows that this can greatly improve the accuracy of various benchmarks, and thus imply the potential of the pretraining data might not be fully explored.", "questions": "1. Could the authors show that the performance of existing off-the-shelf models can be improved by the proposed retrieval pipeline? This does not require pretrain so the author should be able to experiment with larger models. This could also rule out the possibility that the models used in the experiment might not be pretrained properly.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Their observation if able to be held to a larger scale might be interesting and could have some valuable guidance to the community.", "weaknesses": "1. The scale of the experiment is limited. It seems less surprising if smaller models could not fully utilize the pretraining data.\n2. If the authors want to propose this as a method to be used rather than just an experiment for some observation, then they should compare it with other test time scaling methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nz7C8Fwulp", "reviewer": "Reviewer_fW4M", "summary": "The paper asks how much useful signal in today’s open pre‑training corpora is left unexploited by pre‑training itself, and if it can be reclaimed at test time. Using a public 8B reader (Llama‑3.1 8B instruct) together with retrieval + reranking + self‑consistency yields +10.5 points on MMLU, +15.7 on Math‑500, +6.2 on GPQA, and 74% on SimpleQA. On MMLU, they fit a bounded sigmoid to the base models and report that adding retrieval behaves like a ~5× pre‑training compute multiplier on average (declining with scale).", "questions": "- Add MMLU‑Pro/Redux, GSM8K/BBH (with grounding demands), and limited‑access tasks where retrieval should not help (negative control)\n- How does the 5× multiplier change under (a) an unconstrained logistic fit; (b) a power‑law fit; and (c) fits trained only on the three middle scales (to reduce anchor effects)? Please report confidence intervals.\n- Provide curves for k, reranker family, query rewriting, and inter‑document consistency at k>1 to situate the proposed VR/SC stack relative to standard RAG toolings", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The design (pre‑train on a corpus, then retrieve from exactly that corpus at test time) probes how much of the corpus’ information is not captured parametrically. This bridges two active lines of work (RAG and inference‑time compute) in a controlled, data‑centric way.\n- Paper does not hand‑wave contamination: it (a) quantifies overlap, (b) shows that decontaminated retrieval still helps (substantially for MMLU), and (c) surfaces the surprisingly large portion of benchmark content present in common open corpora\n- “Inter‑document self‑consistency” diagnostic is clever and novel.\n- Combination of retrieval, reranking, and self‑consistency is methodologically clean and yields strong additive gains", "weaknesses": "- MMLU has documented label and quality issues; using it as the sole basis to translate accuracy into compute multipliers risks reporting an artifact of the fit or of dataset flaws.\n- Decontamination by token n‑gram overlap (16‑gram for MMLU, 26‑gram for Math‑500) is a good start but cannot remove paraphrastic or templatic leakage. Because the retrieval store is identical to pre‑training corpora, any residual overlap inflates the measured gap between “base” and “+retrieval.”\n- The “compute multiplier” compares training FLOPs without charging anything to indexing, storage, retrieval/reranking inference, prompt expansion, or self‑consistency samples. In many deployments, inference cost dominates, and searching over such a large corpora is likely impractical at best and impossible at worst, due to the increased search compute cost. Without this analysis (of a break even point between inference and training flops), the multiplier is misleading operationally.\n- The base models in §4 appear to be evaluated without retrieval but also without self‑consistency, whereas §5 compares an 8B reader baseline w/ CoT to retrieval + reranker + self‑consistency (+ MMR/bagging). While Table 5 includes a “self‑consistency only” baseline for the 8B reader, the earlier compute‑multiplier analysis relies on single‑pass answers for the base models. A more apples‑to‑apples comparison would allow multiple sampled chains and majority‑vote for the base models as well (possibly with no extra context) to isolate how much of the gain is from extra sampling vs. retrieval.\n- The narrative proposes that retrieval may aid reasoning by “expanding context,” but the current evidence is correlational (for the stem benefits more than humanities claim). Are the largest per‑subject gains driven by questions where the retrieved passages contain the answer string (knowledge store) vs. passages that supply definitions/lemmas enabling reasoning (process aid)? A per‑question audit (string‑match / entailment / indirect support) would sharpen the mechanism story.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TCd93SP9sO", "reviewer": "Reviewer_EuQe", "summary": "This submissions studies the benefit of retrieval augmented generation in terms of how it compares to increased pre-training. It shows that retrieval potentially yields benefits comparable to significant multiples of (pre-)training time. A number of experiments analyze various related aspects of pre- and test-time training of LLMs.", "questions": "Gains for SimpleQA are striking (Fig. 1) -- so large that they don't really fit the \"compute multiplier\" narrative. What is going on with retrieval on that benchmark?\n\nHow were the five compute budget points chosen? Also it was not 100% clear whether model size and architecture are kept fixed as compute grows.\n\nWhen retrieval is done, is it done on the full dataset, or only the part used for pre-training?\n\nWhat exactly is \"subset retrieval\"? The description on l.159-160 is not sufficient.\n\nl.161: \"a piecewise function\" -- this by itself is not very informative. Arguably most curves on Math500 are also \"piecewise\".\n\nHow are the variance reduction techniques used in the context of this paper (l.329-330)?\n\nThis is a matter of taste and author choice of course, but there is no conclusion beyond future work?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper argues an interesting and highly-valuable point: well-done retrieval can comparatively benefit performance much more than additional generic model training. The idea of presenting gains in terms of \"compute multipliers\" is convincing (although the underlying simple sigmoid model deserves more attention) . The paper also provides evidence of additive benefits with other test-time procedures, which is promising. There is a massive amount of experiments here, supporting these findings and further analyses. The choice of datasets is clear and reasonable.", "weaknesses": "The main weakness of the paper is the general clarity re. exactly what is done. One blatant example is the Experimental Setup section, esp. 3.2: The short paragraph provides a straightforward and high-level description of the retrieval process, but no description of how the retrieved documents are used -- presumably a RAG-style process, but absolutely no detail is provided. Given that this is mainly an experimental paper, it is absolutely necessary that experimental details are provided, at minima with references, so that the reader knows what is done. There is also some confusion about what model(s) was/were used. The text mentions at various places 6.4B models, 8B models and 12.6B models -- it is not always clear which was used where (and why). \n\nThere is a dizzying amount of experiments done in Section 4, 5 and 6, providing a lot of experimental evidence. Unfortunately, the amount of description for each experiment is minimal, often only a couple of sentences with tables or graphs that are poorly described. It is often unclear what is reported (e.g. what metric is used). And no error bars or assessment of experimental uncertainty are shown.\n\nSeveral interesting findings arise from the many paper's experiments. However some sound more like speculative guesses, e.g \"might require logical/abstract reasoning\" (l.254-255), \"the model ignores additional context, rather than additional context being misleading\"(??, l.284-285), \"it is likely that there is substantial overlap\" (l.362-363). Even the overall message from the paper, that pre-training does not make \"full use\" of the data, is unconvincing and not unambiguously supported by the data. For example, observations may just be the unsurprising consequence of leveraging test-specific data versus using a general-purpose model out-of-the-box.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N4Kt7k3BYe", "reviewer": "Reviewer_xk27", "summary": "The authors pose the question of how well pretrained LLMs can extract information from their pretraining data. To study this, they propose using a RAG setup where the pretraining corpus itself is used for retrieval at test time. Through a broad set of experiments, they report notable accuracy gains on MMLU, MATH-500, and SimpleQA, and argue that these improvements indicate that current pretraining does not fully exploit the information in the data. They further claim that retrieval acts as a compute multiplier, a ~5x gain on MMLU, since, under a fixed FLOP budget, their RAG approach achieves the same accuracy as the base model with fewer FLOPs. They also show that adding test-time scaling via self-consistency leads to additional improvements.", "questions": "- Do you believe that the results would change drastically if the RAG corpus was different than the pretraining corpus?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written, and the methodology is clearly explained.\n- The central question of \"how much of the pretraining data is left unexploited after pretraining\" is interesting and well motivated, and can help understanding the strengths and limitations of current LLMs.\n- The authors conduct a substantial number of experiments, systematically testing different aspects of their claims.\n- The pretraining experiments of Section 4, particularly those after decontamination, indeed suggest that pretraining alone does not always capture all patterns present in the pretraining corpus.", "weaknesses": "- I am not sure if the compute multiplication arguments are fair. To my understanding, the authors do not take into account the FLOPs associated with the retrieval and reranking components of their RAG pipeline when comparing against the base model’s pretraining FLOPs. As a result, the “under fixed compute budget” claim becomes somewhat misleading, and the reported multipliers (sometimes ~10x) are possibly inflated.\n- In addition to my previous argument, I am not fully convinced that the main message of the paper is justified. While the results do show consistent improvements with retrieval, it is not clear whether these gains truly reflect “unused” information from pretraining, or simply the effect of providing additional access to relevant text at test time. Thus, I believe that more experiments in a more controlled setting would be necessary to support this hypothesis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IYQWSeJxJP", "reviewer": "Reviewer_Kf1X", "summary": "This paper explores an interesting idea: reusing pre-training data at test time through retrieval augmentation to boost the performance of large language models. The authors demonstrate that this method acts as a \"compute multiplier,\" meaning it achieves performance gains that would otherwise require significantly more computational resources for pre-training alone. They test this on several benchmarks like MMLU, Math-500, and SimpleQA, showing that retrieval, especially when combined with techniques like self-consistency and reranking, leads to substantial improvements. The key takeaway is that current pre-training methods don't fully squeeze all the juice out of the vast datasets they use, and test-time retrieval is a powerful way to unlock that leftover potential.", "questions": "1.  You showed that retrieval from the pre-training corpus is very effective. How do you think this approach would perform if the retrieval database was a completely different, specialized corpus that the model has never seen before? Would the \"compute multiplier\" effect still hold?\n\n2.  In your analysis (Table 4), the performance lift from subset retrieval is surprisingly close to that of full retrieval for many MMLU subjects. Why do you think a smaller, targeted subset of the data provides such a comparable boost to having the entire dataset available for retrieval?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The core concept of treating test-time retrieval as a \"compute multiplier\" is a really neat and effective way to frame the value of retrieval-augmented generation.\n\n2.  The experiments are quite thorough, covering multiple datasets and model scales, which provides a solid empirical foundation for the claims.\n\n3.  The analysis of how retrieval impacts different MMLU categories (like STEM vs. humanities) offers some cool insights into where this technique helps the most.", "weaknesses": "1.  The study focuses on models pre-trained and retrieved from the *same* dataset, which might not always be the case in real-world applications.\n\n2.  While the gains are impressive, the paper notes that the effectiveness of retrieval as a compute multiplier diminishes as the base model size increases, which could limit its long-term utility for future, even larger models.\n\n3.  The techniques explored, like retrieval and self-consistency, add significant computational overhead at inference time, and the trade-offs aren't discussed in much detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)