Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarks
Authors: Xinxi Lyu
Michael Duan
Rulin Shao
Pang Wei Koh
Sewon Min
Venue: ICLR 2026 Poster (2026)
Keywords: retrieval-augmented language model
RAG
reasoning
datastore
dense retrieval
Primary area: causal reasoning
TL;DR: We introduce CompactDS, a compact, high-quality, and diverse datastore that, with a minimal RAG pipeline, achieves consistent gains on various challenging, reasoning-intensive benchmarks and outperforms commercial search engines like Google Search.

Abstract: Retrieval augmentation has primarily been studied in limited settings, such as factoid question answering; more challenging, reasoning-intensive benchmarks have seen limited success from minimal RAG. In this work, we challenge this prevailing view on a set of established, reasoning-intensive benchmarks: MMLU, MMLU Pro, AGI Eval, GPQA, and MATH. We identify a key missing component in prior work: a usable, web-scale datastore aligned with the breadth of pretraining data. To this end, we introduce CompactDS: a diverse, high-quality, web-scale datastore that achieves high retrieval accuracy and subsecond latency on a single-node deployment, making it suitable for academic use. Its core design combines a compact set of high-quality, diverse data sources with in-memory approximate nearest neighbor (ANN) retrieval and on-disk exact search. Using CompactDS, a minimal RAG pipeline achieves consistent accuracy improvements across all benchmarks and model sizes (8B--70B), with relative gains of 11% on MMLU, 34% on MMLU Pro, 26% on GPQA, and 14% on MATH. No single data source suffices alone, highlighting the importance of diversity of sources (web crawls, curated math, academic papers, textbooks), and a combination of ANN and exact search is shown to be critical for balancing usability and accuracy. Finally, we show that our in-house datastore even outperforms commercial search engines like Google Search. We release CompactDS and our retrieval pipeline as a fully reproducible alternative to commercial search, supporting future research exploring retrieval-based AI systems.

Published as a conference paper at ICLR 2026
FRUSTRATINGLY SIMPLE RETRIEVAL IMPROVES CHAL- LENGING, REASONING-INTENSIV BENCHMARKS
Xinxi Lyu Michael Duan
Rulin Shao wa
Pang Wei Koh wa
Sewon Min ba
UUniversity of Illinois Urbana-Champaign *University of Southern California
Allen Institute for Al
"University of Washington
University of California, Berkeley
seanlyu28illinois.
duanm@use. edu
sewonm@erzkeley.co edu
alzope/CompattD--1222
compactas-retrievg
ABSTRACT
Retrieval-angmented Generation (RAG) has primarily been studied in limited settings, such as factoid question answering, more challenging, reasoning-intensie benchmarks have seen limited success from minimal RAG. In this work, we challenge this prevailing view on established, reasoning-intensse MMLU, MMLU Pro, AGI Eval, GPQA, and MATH. We identify key missing component in prior work: a usable, web-scale datastore aligned with the breadth pretraining data. To this end, we introduce COMPACTDS: diverse, high-quality, web-scale datastore that achieves high retrieval accuracy and subsecond latency on a single-node. The key insights are (1) most web content can be filtered out without sacrificing coverage, and a compact, high-quality subset is sufficient; and (2) combining in-memory approximate nearest neighbor (ANN) retrieval and on- disk exact search balances speed and recall. Using COMPACTDS, we show that a minimal RAG pipeline achieves consistent accuracy improvements across all benchmarks and model sizes (8B-70B), with relative gains of 10% MMLU, 33% on MMLU Pro, 14% on GPQA, and 19% on MATH. No single data source suffices alone, highlighting the importance of diversity of sources (web crawls, curated math, academic papers, educational text). Finally, we show that our carefully designed in-house datastore matches or outperforms web search engines such as Google Search, as well as recently proposed, complex agent-based RAG systems-all all while maintaining simplicity, reproducibility, and self-containment. We release COMPACTDS and our retrieval pipeline, supporting future research exploring retrieval-based AI systems.
benchmarks:
INTRODUCTION
Retrieval-based language models (LMs) enhance generation by retrieving relevant information from a large text datastore and feeding it into the model (Guu et al. [2020) Karpukhin et al.) 2020 Lewis
al. 2020 Borgeaud et al. 20022] (hh al., 2023 Asai al. 2023 Shao et al. (0024). This approach has proven highly effective for tasks like question answering and f'ct verification, especially for
information-ssekking queries that require precise facts that can be found in highly curated knowledge
source like Wikipedia (Kwiatkowski et al., 2019 Lee et al., 2019 Joshi et al. 2017 Petroni et al. 2020). However, much less understood about the utility of retrieval beyond factoid tasks. Prior work suggests that retrieval offers no benefit and even hurts performance on reasoning-intensing tasks (BehnamGhader et al. 2022, Geng al. 2024). Some studies have introduced agentic pipelines to address this gap, but they either rely on web search engines (Li al. 2025b Wu et al. 2025)
are limited to Wikipedia (Jin et al. 2025 Chen et al. 2025 Song et al.
2025 Sun et al. 2025).
In this work, revisit this assumption and ask: How far can we improve performance on challenging, reasoning-intensiee intensive LM benchmarks using minimal retrieval- dense retrieval followed by generation?
"gqual contribution.

---

Published as a conference paper at ICLR 2026
Encode (Offine)
Encode (Offlie)
Datastore
Query
IVFPQ Index
Top Passages
Disk
Top Passages
ANN SEARCH
EXACT SEARCH
Figure 1: An illustration of the two-stage dense retrieval pipeline with COMPACTDS.
To this end, evaluate minimal retrieval setup across diverse set of established benchmarks, including MMLU (Hendrycks et al., 202 la), MMLU Pro (Wang et al. 2024), AGI Eval (Zhong et al. 2023), GPQA (Rein et al. (2023), and MATH (Hendryeks et al. 20211). We identify a critical limitation in prior work: the absence of an accessible general-purpose datastore. Prior work used either Wikipedia-based datastores, which limited in coverage these general-purpose benchmarks (as we show in $4), or large-scale web datastores that contain massive unfiltered low-quality sources and inaccessible in practice (Borgeaud et al.) 2022 Shao et al. 2024). e.g., requiring over 12TB of RAM unless deployed on a distributed serving system.
To this end, we introduce COMPACTDS, 380-billion-word datastore constructed from high- quality, diverse collection of sources-inctuding web crawls, curated math content, academic papers, and educational text text- designed to match the breadth of pretraining data while remaining accessible. COMPACTDS is built on two key insights. First, it is possible to aggressively filter low-qualit, web text while preserving the coverage and diversity of the web, resulting in a smaller yet representative dataset. Second, combining in-memory approximate nearest neighbor (ANN) search (Johnson et al.) 2019) with on-disk exact inner product search enables subsecond retrieval on a single 456GB RAM node far more efficient than prior systems that require several TBs of RAM.
COMPACTDS leads to significant performance improvements on benchmarks that were previously thought not to benefit from mininal retrieval: +10% MMLU, +33% MMLU Pro, +14% GPQA, and +19% on MATH with Llama 3.1 8B Instruct, with persistent gains as the model scales to 70B parameters. Notably, exact inner product search with
expressive encoder following ANN helps compared to using ANN alone, widening gains over no retrieval from 26% to 33% on MMLU Pro and from 14% 19% on MATH. Our ablation study datastore composition reveals several key insights: (1) Dataset diversity is critical: no single data source suffices alone, and even removing weakest contributing sources leads degraded performance; (2) Including educational materials, often absent from web crawls, consistently improves performance; and (3) Quality filtering methods developed for pretraining generally help.
Notably, our in-house datastore matches or outperforms web search engines like Google Search across all benchmarks-44 result that was previously unattainable using standard RAG benchmarks built with web search an oracle (K (Kwiatkowski et al. (019). It also matches or outperforms recently proposed, complex systems based on web search and agentic pipelines (Li al. 20256). demonstrated on GPQA Diamond (Rein et al. 2023) and MATH-500 (Hendrycks et al. 1b using QwQ 32B.
We release COMPACTDS and our retrieval pipeline as a reproducible, academically accessible alternative web search. We hope COMPACTDS facilitates future work on retrieval-based AI systems, including integration with agentic pipelines and training with retrieval.
2 RELATED WORK
Retrieval Augmentation and Evaluation. Retrieval augmented generation (RAG) has shown
remarkable success factoid guestion-answering tasks (Karpukhin et al.) 2020 Lewis et al. 2020).
However, the majority of RAG benchmarks focus fact retrieval, typically supported by highly
curated knowledge sources such as Wikipedia (Kwiatkowski et al. 2019 Lee et al. 2019 Joshi
let al. 2017 Petroni al.] 2020); the applicability of RAG to tasks beyond this scope remains largely
unexplored. In fact, prior work presented that RAG is ineffective for other tasks like reasoning
tasks (BehnamGhader al. 2022 Geng et al. 2024h and open-ended generation (Wang et al. 2023)
2

---

Published as a conference paper at ICLR 2026
In this work, we evaluate retrieval on broad set of benchmarks MMLU, MMLU Pro, AGI Eval, GPQA, and MATH that are well-established and not specifically designed for RAG, and known to be reasoning-intenssin
Concurrent work, ReasonIR (Shao et al. 2025 also demonstrates the value of retrieval for reasoning tasks, but takes an orthogonal approach: ReasonIR focuses on improving the embedding model, whereas we foeus on improving the datastore and nearest neighbor search.
While extensive work has focused on improving training datasets (Raffel et al. 2020 Sollaini et al. 2024 Li et al. 2024 Penedo et al. 2024), little attention has been given to improving data as a retrieval datastore. Beyond Wikipedia, some efforts have used broader web sources, e.g., small random subset of Common Crawl (five billion tokens) (Shi al. 2023 Lin 2021). These efforts were still evaluated on perplexity or Wikipedia-based
Improving Datastores.
et al. 2023 Piktus et al.
benchmarks (except)Shi Shi al. 2023 on MMLU, which we compare against).
We argue that prior datastores are either too narrow or small to be broadly effective, or not practically usable, e.g., MassiveDS (Shao al., 2024) requires over 12TB of RAM avoid multi-minute latency, making deployment infeasible in typical academic settings without distributed infrastructure. This work directly addresses these issues, proposing a datastore that large and broad in coverage, yet compact enough to enable subsecond latency in a single-node deployment.
Agentic RAG. Recently, agentic RAG, which iteratively issues search queries, retrieves information, intensive tasks, has emerged as an active area of research. These approaches can be broadly divided into two categories: (I) prompt-based methods that do not require training (Li et al. 2025b Wu et al. 2025), and (2) graining-based methods that fine-tune reasoning LM to use search, typically via reinforcement learning (Jin et al., 2025| Chen et al. are costly, hard to reproduce, and unstable, making them unsuitable for training, as also noted by (Sun et al. 2025). Consequently, most training-basse based work uses an in-house Wikipedia datastore and only evaluate on Wikipedia- based benchmarks.
and reasons over results to perform reasoning-intensive
2025 Song et al., 2025 Sun et al. (025). Much of this work uses web search engines, which
Instead of optimizing for agentic RAG, our work focuses minimal RAG, which is fundamental building block of any retrieval-basee AI systems that can be easily integrated. This agentic RAG literature, however, highlights an emerging need for high-quality, general- purpose in-house datastores, particularly to enhance reproducibility, improve stability, and ensure cost efficiency.
3 METHOD
We discuss two key ideas that enable a high-quality, high-coverage retrieval datastore: data sources that match the breadth of pretraining corpora while filtering out low-quality web text ($3.I), and approximate nearest neighbor (ANN) search followed by exact search ($3.2).
3.1 COMPACTDS DATA SOURCES
To match the breadth of pretraining corpora while achieving high quality and diversity, we strategically construct COMPACTDS with the following data sources:
Web Crawl. To ensure wide coverage, we start with Common Crawl, which is widely used for pre-training and also constitutes 70% of MASSIVEDS (Shao et al. 2024). However, we hypothesize that much of it is low- low-quality and unnecessary for retrieval. Therefore, we construct compact, high- quality subset (High-quality CC) using a series of filtering steps. We take the union of C4 (Raffel et al.) 2019), a small curated subset, and DCLM- (Li et al. 2025a), which has undergone extensive manual and model-based filtering, and then further filter DCL.
using the Fine FineWeb-Fii Web-Edu classifier (Penedo et al. 2024) with a threshold of 4.0, which filters text based on its educational value. This process reduces the size of Common Crawl from 894B words to 172B words.
Wikipedia and Books. We then augment the web crawl with data sources commonly regarded as high quality for pretraining: Wikipedia, books, and educational text.
Wikipedia: We consider two sources: Wikipedia from DPR (Karpukhin et al. 2020 and 2) Wikipedia in RedPajama-VI (Computer) 2023).
Books: We use the RedPajama- VI Books subset, containing a wide variety of digitized eBooks.
Educational Text: We take the data from Shi et al. (2025) text extracted from digitized PDFs
3

---

Published as a conference paper at ICLR 2026
Expert Data. To broaden the coverage, include more data sources that contain expertise knowledge areas such as math, science, and coding:
Math: We combine OpenWebMare (Paster et al. 2023). filtered math webpages from Common Crawl, and NaturalProof's (Welleck et al. 2021). corpus including theorems, proofs, definitions, and related content.
Academic Papers: We consider three sources: Pes2o (Soldaini & Lo 2023). open-access academic papers, PubMed (National Library of Medicine, 2023). collection of biomedical and life sciences journal literature, and ArXiv
Github:
We use the RedPajama-V V1 GitHub subset, a well-established source for code.
Q&A A Forums. Finally, we include Q&A forums widely used in pretraining:
Stack Exchange: We use the RedPajama-Y V1 subset of Stack Exchange, a collection of high- quality, community-sourced Q&A spanning diverse domains from computer science to chemistry.
Reddit: We take the Reddit data from (Shi et al. 2025).
To ensure the robustness of our evaluation, we filter out paragraphs with an over 70% 13-gram Jaccard similarity with any query in all of our evaluation datasets following Borgeaud ct al. 2022). The impact of this decontamination method on performance is reported in $A.1
Decontamination.
In total, COMPACTDS includes 380.5 billion words from 639 million documents. Each document is split into passages of 256 words, following (Shao et al. 2024), resulting in 1.9 billion passages. More details and statistics are provided in A.2
3.2 DENSE RETRIEVAL. WITH COMPACTDS
COMPACTDS is based on a dense retrieval model over N passages, {pI, PP. ... .NN)- Given an encoder E' that maps text into an fi-dimensional vector, passage embeddings Pı. PN E R are pre-computed and indexed using a nearest neighbor search structure. At test time, a test query is encoded as q Eg)) E R and the top-k passages retrieved according argTophassienq
The key challenge, when it comes to deploying web-scale datastores, is the memory and compute overhead in building and serving a nearest neighbor search index for billions of high-dimensional vectors. For example, with N = 1.9 x 109 and h = 768, storing the passage embeddings full precision requires 1.9 10 768 4 5.4TB of vector data, which is infeasible to store in memory.
Approximate Nearest Neighbor (ANN) Search via IVFPQ. To reduce search latency and storage requirement, we use approximate nearest neighbor (ANN) via Inverted File with Product Quantization (IVFPQ: JJeoou et al. (. IvFPQ partitions the vector space into clusters and quantizes vectors within each cluster to reduce memory usage and improve speed. This allows us to build and serve the ANN index within 456GB of RAM, achieving subsecond retrieval latency. However, IVFPQ typically incurs notable performance degradation due to its lossy nature.
Second-stage Exact Inner Product Search. To recover performance lost from the approximate search, we leverage a second-stage exact inner product search. Specifically, the ANN index retrieves a candidate set of K passages (K > k), which are then re-ranked using their original (non-quantized) embeddings identify the final top-k results. These exact embeddings can be stored on disk, enabling efficient bbsse search under reasonable disk I/O constraints, especially when K is modest (e.g.. 100 < K S 1000). If on the fly, trading off latency. Moreover, the encoder used for exact search can differ from the one used for ANN, enabling the use of a more expressive model that may be difficult to index. This design choice is inspired by DiskANN (Jayaram Subramanya al., 2019). which similarly performs in-memory ANN search followed by on-disk exact search.
search is infeasible, embeddings can alternatively be recomputed
Summary. Altogether, dense retrieval in COMPACTDS proceeds as follows (also shown in Figure I] Offline, each passage p; is encoded using Expprox and indexed with IVFPQ for fast search in RAM (456GB). parallel, passages are encoded using a stronger encoder, Ersact- and the resulting embeddings are stored on disk. At test time, query q is encoded as E (g) and used to retrieve K passages from the IVFPQ index. Then, q is encoded as E (q). and exact inner product search is applied over the K disk-stored embeddings to obtain the final top-k passages: pı," .Pk- We use
4

---

Published as a conference paper at ICLR 2026
EETTRIEVER-MSMAREE (Izaaar et al. 2021) as Eappox. and GRITLM-7B (Muennighoff et al.) 2024) as E
This two-stage dense retrieval resolves the memory and latency issue that single-stage exact nearest neighbor search encounters when deploying web-scale datastores, with performance largely restored.
3.3 AUGMENTATION WITH COMPACTDS
denoted Given P2, ... P& returned by dense retrieval, we feed them into an L.M denoted as generator- to generate the answer. We adopt straightforward augmentation strategy by concatenating the top-k retrieved passages in reverse order, placing the most relevant passage closest to the query g, followed by 9 itself, and feeding it to the LM that generates the answer. We also consider optional, LLM-based reranking; details are provided in |A.3]
Oracle Reranking. Despite the strong capabilities of LLMs generators and reranks, they may still struggle to process and reason over set of retrieved passages, particularly if not explicitly post-trained to do (Jin al. 2024). We aim quantify the upperbound performance achievable from COMPACTDS outputs by defining an oracle reranking method. Given a test query q and its ground truth answer a, we select the top Krerank passages from the K candidates retrieved from COMPACTDS-ANN, where K 2 hiserank >> k. Each passage scored by generator LLM, based on the increase in the model's likelihood of the gold answer a when appended to the query. The top-h passages based on the oracle scores put into the generate stage (sce details in CC.6
EXPERIMENT
4.1
EVALUATION SETUP
We evaluate COMPACTDS using the following five benchmarks (detailed in şB): MMLU (Hendrycks) al. 202 aa and MMLU Pro (Wang et al. 2024), evaluating general knowledge and reasoning over broad spectrum of subjects (MMLU is grouped into four main categories: STEM, Humanities, Social Sciences, and Other); AGI Eval (Zhong et al.) 2023). consisting of questions form human-centric standardized exams (c.g.. LSAT); GPQA (Rein et al. 2023), comprising writtee, graduate- level multiple-choice questions in biology. physics, and chemistry, and specifically designed to be web-search-proof; and MATH (Hendrycks al. 20221)) collection of challenging, competition mathematics problems, sourcing problems from exams such as AMC 12 and AIME. The MMLU, MMLU Pro, and AGI Eval datasets are evaluated multiple choice, while we use chain-of-thought for GPQA and MATH. These are well-established, standard benchmarks in the field, frequently reported by frontier models (OpenAl et al. 2024; Dubey al., 2024). They are known for their domain diversity and reasoning demands, posing greater challenge compared to the simpler factoid- based QA tasks commonly used in RAG literature (K Kwiatkowski et al. 2019 Joshi al. 2017).
In addition, to compare with Search-ol, recently proposed agentic RAG system (Li et al. 2025b). we include two benchmarks used in their work (detailed $5.2 GPQA Diamond (Rein al.
2023). high-quality subset of GPQA, and MATH-500 (Lightman al. 2023), a subset of MATH.
Language Models. We use Llama 3.1 Instruct (Dubey et al. 2024 as the default generator for the main experiments and most of the ablations. To evaluate COMPACTDS across different model sizes and families with stronger capabilities, we also include Llama 3.3 70B Instruct, Mistral 7B
Instruct (Jiang et al. 2023). Qwen3 8B (Yang et al. 2025). and QwQ 32B (Team) 2024).
4.2 MAIN RESULTS
Table T presents our main results using llama 3.1 8B Instruct. COMPACTDS significantly improves performance across all datasets, e.g., .. 1.0% on MMLU STEM, 33.4% on MMLU Pro, 19.2% on MATH, and 36.2% on GPQA Physics. These improvements are particularly notable given the reasoning-intensive nature of these tasks, e.g., GPQA is specifically designed to be web-search-pron proof.
Diversity in COMPACTDS Composition is Critical. Table also reports the performance of datastores constructed from individual data sources. We find that certain sources greatly benefit specific benchmarks, e.g., Educational Text improves MMLU and GPQA; Math corpora boosts
5

---

Published as a conference paper at ICLR 2026
Table 1: Results with Llama 3.1 8B Instruct, comparing retrieval, single-source datastores, and COMPACTDS, k = 3, unless specified otherwise. Best results per source bolded in black, best overall bolded in blue. The relatives gains are computed using the best results with COMPACTDS for each dataset.
MMLU
GPQA
MMLU AGI Eval
MATH
AVG
STEM Human. Social Others
Phys
Bio
Chem
No Retrieval
60.2
72.0
78.7
68.9
39.8
56.2
46.9
26.7
47.4
25.7
48.3
Single-source datastores (All ANN only) Math
63.5
73.1
80.4
70.6
44.1
52.7
31.6
47.4
26.8
51.6
Educattinal Text
62.2
75.7
82.2
75.2
47.4
45.9
35.3
50.0
26.8
$1.3
High-quality CC
62.3
74.0
82.8
74.3
45.1
56.8
45.9
26.7
46.2
26.2
50.0
Books
60.1
74.8
81.8
73.1
44.0
56.5
47.3
31.6
37.2
26.8
49,9
Academic Papers (PeS2o)
59.4
73.5
80.2
69.8
42.3
55.5
45.1
32.6
52.6
28.4
49.4
Wikipedia (Redpajama)
61.6
73.8
80.5
71.6
43.0
54.6
48.4
28.9
46.2
24.6
49.4
Reddit
60.6
72.8
78.9
70.6
41.4
56.5
45.9
31.0
50.0
24.6
49.0
Wikipedia (DPR)
61.4
74.1
80.8
71.1
41.9
55.6
46.1
25.1
53.8
26.8
49.0
StackFxe
63.0
72.0
78.5
69.5
41.0
57.0
46.9
31.0
46.2
21.9
48.9
Github
60.8
72.2
78.8
69.1
40.4
57.0
44.7
32.1
39.7
25.7
48.4
PubMed
60.4
79.6
40.7
56.1
44.9
28.9
25.1
Academic Papers (ArXiv)
59.3
71.8
78.1
39.5
57.5
45.7
25.7
27.9
48.0
COMPACTDS-AN -ANNN only
64.6
76.4
84.3
75.3
47.7
58.9
50.3
26.7
44.9
26.8
$2.2
COMPACTDS
64.4
76.8
83.7
73.9
49.1
60.2
55.1
33.2
39.7
28.4
54.1
COMPACTDS (k 10)
66.8
77.9
83.2
77.0
53.1
58.9
55.9
29.4
47.4
29.0
55.1
Relative gains Retrieval
11.0%
8.1%
7.7%
11.8%
33.4%
7.1%
19.2%
12.8%
14.7%
Table 2: MMLJ results on COMPACTDS MASSIVEDS using Llama 3.1 8B Instruct. ES indicates "Exact Search." Best numbers bold.
Table 3: Oracle performance on COMPACTDS with Koracle 100 and
3. Best numbers in bold.
MMLU MMLU Pro
AGI Eval
AVG
No Retrieval
68.9
39.8
56.2
$5.0
RAM
Method
Accuracy
Llama 3.1 88 Instruct with CompactDS
No Retrieval 12.4TB
68.9
Our Best
75.7
53.1
60.2
63.0
MASSIVEDS
ES
73.6
Oracle
85.6
63.1
65.0
71.2
COMPACTDS
0,5TB
ANN
75.2
COMPACTDS
0,5TB
ANN
ES
75.3
Llama 708 Instruct
No Retrieval 81.5
$7.8
71.1
70.1
performance MATH; Wikipedia from DPR helps GPQA Biology; and Pes2o improves GPQA Chemistry. However, gains from any single source are generally limited.
On the other hand, COMPACTDS-ANN which retrieves across all sources using only ANN search, yields a 8.1% average improvement, highlighting the importance of diverse data coverage for broad task performance. Our preliminary experiments showed that even removing four of the least impact- ful sources (ArXiv, Books, GitHub, and Reddit) reduces performance (e.g., by 1.8% on GPQA), suggesting that long-tail data diversity plays role; see 1 for details.
Among single-source datastores, educational content, which is often absent in web-crawled corpora, and expert content like Math deliver the greatest improvement. Also note that Wikipedia from DPR is the most commonly used datastore in the RAG literature (Karpukhin et al. [2200] Lewis et al., 2020 however here, it offers little benefit on average, and even hurts performance on many datasets.
COMPACTDS with MASSIVEDS (Shao et al. 2024h [0 which, to our knowledge, is the only open source web-scale datastore. MASSIVEDS uses exact inner product search over 12.4TB vector database, requiring sequential shard-wise search and aggregation due to its size, leading to impractical latency for deployment. Building an ANN index at this scale is also prohibitively memory-intensive
First Practical Web-scale Datastore.
For fair comparison, took MMLU retrieval results from MASSIVEDS provided by the original paper and augmentation using laama 3.1 8B Inst.

---

Published as a conference paper at ICLR 2026
Table 4: Comparison among different retrieval pipelines, using LJama 3.1 8B Instruct and K 1000.
ES indicates "Exact Search." The best results are shown in bold.
MMLU
GPQA
MMLU Pro AGI Eval MATH
AVG
STEM Human. Social Others
Phys
Bio
Chem
Retrieval
60.2
72.0
78.7
68.9
39.8
56.2
46.9
26.7
47.4
25.7
48.3
ANN Only (Contriever)
66.4
76.7
85.2
76.7
50.1
57.6
53.3
31.6
48.7
27.3
53.8
ANN (Contriever) + ES (Contriever)
64.8
75.8
83.6
75.5
50.0
59.0
52.9
32.1
$1.3
24.6
53.6
(Contriever) (GRIT)
66.8
77.9
77.0
58.9
55.9
29.4
Table 2 shows that COMPACTDS, built from heavily filtered web crawl data and a few additional sources (,..1) already outperforms MASSIVEDS on MMLU using only ANN retrieval-whit requiring just 4% of the RAM. With exact search, performance improves further, likely offistting any loss from ANN. Overall, these results demonstrate that careful construction of datastore content combined with wwwrooxiaattttee-eeeee retrieval enables COMPACTDS to be the first compact, efficient, and deployable web-scale datastore.
Further ablations on the two-stage retrieval design and memory-performance tradeoffs, including result showing a 4x reduction index size with only small performance drop, are provided in $C]
Ablations Exact Search. Table that COMPACTDS, with Exact Search using GRIT after ANN, consistently improves performance in nearly all cases. Exact Search using CONTRIEVER the same model as ANN-does not improve over ANN-only significantly. This suggests that the use of a more expressive model accounts for the majority of the improvement, showing the benefit of the two-stage design described in $3.2
A natural follow-up question is the comparison to ANN based on GRIT, which is significantly more expensive due to its 0.096-immensional vectors, demanding substantially more RAM and disk space than Contriever-based ANN. $C.3 experiments with GRIT-based ANN on small data subset, showing that the approximate- then-exade pipeline still outperforms GRIT ANN alone, likely because combining two complementary models helps.
Upper-bounding COMPACTDS Performance.
To quantify the upperbound performance achiev- able from COMPACTDS, report results using three oracle passages from pool of 100 retrieved by COMPACTDS-ANN ($3 for details). As shown in Table 3 oracle reranking yields a significant gain, increasing the average improvement over retrieval from 8.0 to 16.2, and surpassing the 70B model. This suggests that COMPACTDS's utility could be significantly higher with better dense retrieval (returning oracle passages) stronger generator that can effectively leverage 100 passages without being misled by distractors.
Qualitative Analysis. Table presents an example of passages retrieved from COMPACTDS for a given query. The query requests the ratio of hydrogen atoms in the second excited state at extreme atmospheric temperatures of Sirius. The retrieved passage contains a similar question and solution concerning the ratio in the first excited state, demonstrating COMPACTDS's ability to return passages relevant and helpful to the query. A more detailed qualitative analysis is given in
Extended Decontamination. Although already applied standard n-gram decontamination, it is still reasonable to ask whether the gains arise from residual contamination not captured by the n-gram method. To this end, we conducted a stricter post-hoc decontamination by prompting GPT-5-mini to identify and remove passages containing the test question and its answer. This can be seen as the most conservative approach: we qualitatively found many false positives but few false negatives. Results, detailed in CC.7 show slight performance drop, but the overall gains remain largely the same, indicating that contamination is not the primary source of improvements. We also note that LMS may already have been exposed to these datasets during pretraining, so the presence of positives in retrieval does not introduce new source of contamination specific to our method.
4.3 COMPACTDS IS EFFECTIVE ACROSS MODEL SIZES AND FAMILY
Gains Persist at 70B Scale. Table shows results from applying COMPACTDS to Llama 3.3 70B Instruct, one of the strongest open- L.LMs at the time of conducting experiments, and compares
7

---

Published as a conference paper at ICLR 2026
Table 5: COMPACTDS results with Llama 3.3 70B Instruct, Mistral 7B Instruct, and Qwen3 8B with
k = 10. Gains are consistent across different model sizes and families.
MMLU
GPQA
MMLU Pro
AGI Eval
MATH
AVG
STEM
Human.
Social
Others
Phys
Bio
Chem
Llama Instruct
No Retrieval
60.2
72.0 77.9
78.7
68.9
39.8
46.9
26.7
47.4
25.7
48.3
COMPACTDS
66.8
83.2
77.0
53.1
58.9
55.9
29.4
47,4
29.0
55.1
IJama 70B Instruct
No Retrieval
74.7
83.7
90.0
81.0
57.8
71.1
72.3
64.2
78.2
50.8
68.8
COMPACTDS 78.6
85.8
89.8
85.7
65.4
72.2
77.0
62.0
73.1
45.4
71.2
Mistral Instruct
No Retrieval 49.6
66.8
73.3
64.4
32.6
48.6
13.4
27.8
44.9
22.4
37.1
COMPACTDS
59.8
72.8
78.6
72.3
42.8
52.9
16.3
33.7
38.5
25.1
42.6
Qwen3 8B
No Retrieval
73.2
75.2
82.8
75.6
49,4
65,1
56.3
38.5
56.4
29.5
57.0
COMPACTIS 79.5
80.5
87.1
81.9
60.6
67.3
60.3
42.2
52.6
27.3
61.6
them with Llama 1. 8B Instruct. Significant gains are observed across most datasets, including 5% on MMLU STEM, 13% on MMLU Pro, and 7% on MATH. Smaller improvements on other MMLU subsets are likely due to performance saturation even without retrieval.
GPQA is the one exception where COMPACTDS offers no improvement at 70B, despite significant gains at 8B. This may be due to the dramatic performance gains of the retrieval baseline from the 3.1 8B model to the 3.3 70B model (c.g., 26.7 + 64.2 on Physics), possibly due to the enhanced CoT capabilities. Again, adapting CoT reranking may help, which we leave for future work.
Significant Gains Across Different Models Families. Table 5 shows results with two widely used LLMs outside the Llama family: Mistral 7B Instruct (Tiang al. 2023) and Qwen3 8B (Yang et al.) 2025). COMPACTDS improves performance consistently across datasets and model families, e.g.. + +10.2% on both MMLU STEM and MMLU Pro with Mistral, and +11.2% on MMLU Pro with Qwen3. GPQA Biology is exception, likely for the same reason in the 70B model.
Later, in $52 we further demonstrate that these gains persist with QwQ 32B, a strong reasoning model, matching outperforming Search-ol (Li al. 2025b).
Table 6: Example of the top retrieved passage GPQA.
Questioon: strius is the brighteat star in the aky. The temperature of this star is around 10000 K. Consider Hydrogen atoms in the atmosphere of sirius. what is the ratio of the number of hydrogan
in the second excited state of
Hydzogen to those in ground scate?
Query
Choices: (A) 8.2 10++-8 (R) 7.26 100-6 (C) 8.11 10"-3 (D) 5.4 10..-9
The temperature of the surface of a certain star is 8000 K. Kos: hydrogen star are
electronic ground state. What appzoximate fraction of the hydrogen atems that are in the first excited state land therefere cculd emit a photon)? The energy the the ground state is (-13.6/22 eV) - (-13.6 aV) 10.2 eV = 1.532e-18 J. 2. Relevant equations Eb 1,38e 23 3/X 1/T B/Kb 3, The attemp: at a solution I don't really know how do this problem. joules by the temperature and then divide by the boltsmann constan: to make it unitiess. (1.632e 18/60000////// (1.38e 231-14.8....
surface
excited state above
guess
to divide the energy in
Retrieved
COMPARISON TO SEARCH ENGINES AND AGENTIC SYSTEMS
Web search engines like Google and Bing are powerful retrieval models, and recent work that uses retrieval as out-of-the-box tool often directly uses web search engines instead of in-house datastores (Li et al. 2025b: Wu et al. 2025). Prior work has rarely compared in-house datastores with web search engines, partly because many RAG benchmarks
constructed with search engines as oracles (c.g.. Natural Questions (Kwiatkowski et al., 2019)), and these commercial engines have been optimized for decades. However, is open question whether search engines necessarily
8

---

Published as a conference paper at ICLR 2026
Table 7: Comparison between search engine and COMPACTDS, all with Llama 3.1 8B Instruct. The best results are bolded.
MMLU
GPQA
MMLU Pro
AGI Evai
MATH
AVG
STEM
Human.
Social
Others
Phys
Bio
Chem
No Retrieval
60.2
72.0
78.7
68.9
39.8
56.2
46.9
26.7
47.4
25.7
48.3
Search Engine
61.8
72.6
80.0
71.6
42.8
59.7
51.4
25.7
46.2
32.2
51.3
L.M Reranking
61.3
73.5
80.3
72.0
44.0
59.8
50.2
32.1
42.3
29.5
51.5
COMPACTDS
66.8
77.9
83.2
77.0
53.1
58.9
55.9
29.4
47,4
29.0
55.1
+ LM Reranking
69.1
77.8
86.8
78.7
54.6
59.5
53.0
33.7
47.4
33.3
56.0
optimal for LLMs, they are not primarily designed for RAG. Furthermore, search engines present challenges for research: they are non-deterministic, costly, and often return noisy results.
In this section, we present RAG pipeline using Google Search for retrieval ($5.1) and compare its performance with COMPACTDS ($5.2).
5.1 A COMPETITIVE METHOD FOR USING A SEARCH ENGINE
We construct RAG pipelines similar to that in $32 replacing COMPACTDS with Google Pro- grammable Search Engine querieed queried through the Custom Search API.
Data Processing. The search engine returns ranked list of ten URLs, which include both web pages and PDFs. Extracting clean text from these sources is non-trivial. We broadly follow prior work using search engines (Li et al. 2025b/c). choosing methods to closely match the processing quality with COMPACTDS. For web pages, we parse the web page content using Resiliparse (Bevendorff et al.) 2018). as in (Li al. 2024), and fall back to BeautifulSoup (Richardson) 2007) when parsing fails, following (Li et al. 2025b(c). For PDFs, we use olmOCR (Poznanski et al. 2025). a vision-language model-based parser that outperforms traditional tools
We find many URLs encountering issues such as request errors, Captchas, or other access restrictions. We also tried craw14AI (UncleCode 2024) and paid services such Jina AI but they did not lead to better performance (see $D.4).
Since evaluation benchmarks are well-established, they appear frequently in web search results. To prevent contamination, we apply strict filtering by removing any paragraph (delimited by "In'n") retrieved document that shares any 13-gram overlap with corresponding query, following Shao et al. 2024), and also block search results from hugg ng face.cc.... co.
Using Search Results in Context. We augment an LLM with search results as COMPACTDS, with one key difference. Web search results in PDFs are often too long fit into the generator. While prior work truncates these texts (Li et al. 2025b), we find this approach suboptimal. Instead, we chunk each document segments with up to words and rerank them using CONTRIEVER; this resembles COMPACTDS's retrieval, with ranking constrained to search results. We use e = 512 and k= ==3 3.
We discuss the impact of decontamination, details regarding augmentation strategies, and ablations on augmentation choices and hyperparameters in D
5.2 RESULTS
All results comparing the search engine and our in-house datastore (COMPACTDS) are provided in Table and T
Search engine improves downstream performance. search engine consistently improves task performance, yielding an average relative gain of 6%,
The first block of Table 7 shows that using
opers..eeppr...... opers.goog googl beoogle... wwm.cmm/cco wtt.ccustssssoooeeoooooo search
For instance, Search-ol (Li et al. 2025b) relies on PDFPlumher, which works primarily machine-
generated PDFs.

---

Published as a conference paper at ICLR 2026
and further minor improvements from LM reranking. Later results (Table show that our pipeline outperforms the Search-ol pipeline (Li et al. 2025b), demonstrating its competitiveness.
COMPACTDS is Competitive Alternative. However, compared to the gains from COMPACTDS (second block of Table 7). web search consistently outperformed. On average, COMPACTIS yields 14% average relative improvement, whereas the search engine achieves only 6%. The gap especially significant on certain datasets, e.g., MMLLU Pro (44.0 vs. 54.6 after LM reranking). Such differences could not be observed in prior RAG benchmarks like Natural Questions, which was constructed using Google Search as oracle.
Later results ( suggest that rel- ative performance may depend on the task and model, e.g., with QwQ. the search engine and in-house datastore perform comparably. Nonetheless, found no case where COMPACTDS was meaningfully worse than the search engine.
Table Experiments based on QwQ 32B, comparing our in- house datastore (COMPACTDS) and search engine pipelines from those Search-ol (Li et al. 2025b).
Self-contained?
GPQA-D
MATH-500
et al (025b))
No Retrivval
58.1
83.2
RAG (w/ search engine)
61.6
85.0
Search-ol
63.6
86.4
Experiments with QwQ 32B and Comparison to Search-ol. We evaluate COMPACTDS with QwQ 32B (Tem 2024) and compare against Search-ol, an agentic RAG system built on web search and QwQ. This comparison serves three goals: (1) to validate our findings with larger, competitive reasoning model; (2) demonstrate the strength of our web search pipeline relative to prior work; and (3) to contrast minimal RAG with more complex agentic RAG systems. Evaluations are conducted on GPQA Diamond (Rein et al.) (023) and MATH-500 (Lightman et al.) 2023 using the same benchmarks reported by Search-ol.
Ours
No Retrieval
58.1
91.0
RAG (w/ search engine)
63311
94.0
RAG (w/ COMPACTDS)
63.1
93.2
Table X highlights three key findings. First, our experimental setup is competitive: our no-retrieval performance matches or exceeds Search-ol 's (e.g., outperforming on MATH-500 by 8%), and our web search pipeline achieves better results (e.g., 63.1 vs. 61.6 on GPQA Diamond). Second, minimal RAG with COMPACTDS remains strong, consistently improving over no-retrieval and matching our own web search results. Third, our minimal RAG with COMPACTDS matches or exceeds the full Search-ol system. This suggest that well-designed retrieval pipeline and carefully constructed in-house datastore can serve as stronger baselines for future agentic RAG research. We additionally study how search engines and in-house retrieval can complement each other in D
6 CONCLUSION & DISCUSSION
We challenge the prevailing view that retrieval is ineffective for established, reasoning- intensive benchmarks. To do so, we introduce COMPACTDS, the first practical datastore that captures the diversity and scale of pre-training corpora while remaining deployable on a single node. Its core design combines compact of diverse sources with a two-stage approximate-then-exann retrieval system. Using COMPACTDS, a minimal RAG pipeline achieves consistent and significant gains across all datasets and model sizes, from 8B to 70B. COMPACTDS also outperforms agentic RAG pipelines based on Google Search, offering greater cost-efficiency and reproducibility.
iessoning-ittesive
Subsequent work (Liu al.) 2026) since the first release of COMPACTDS further improves the efficiency of the retrieval pipeline by applying DiskANN (Jayaram Subramanya et al.) 2019). Fur- thermore, they publicly release a web demo and an API endpoint further increase accessibility. Fang et al. (2025) also concurrently and independently demonstrates that using pretraining data as retrieval corpora is an effective and efficient method to improve base model performance on tasks such MMLU and MATH-500.
We release COMPACTDS to support future research in retrieval, reranking, and agentic RAG systems. For example, COMPACTDS could serve as a search engine within a local agentic framework for both inference and training, providing a strong alternative to commercial search APIs. It is also a realistic testbed for evaluating future reasoning intensive ceasoning-nteensive retrieval and reranking pipelines at scale. Beyond sarieval-augmeent generation, our dense retrieval pipeline can further enable efficient data attribution and contamination detection in pretraining data, which we leave for future work.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
We describe the datastore curation, construction, and usage at high-level in 3 and greater detail in A Search engine methodology can be found in and $D Our experimental setting is detailed in
and SB
We also provide links to our project artilacts and codebase on the first page.
ACKNOWLEDGEMENT
We thank Alex Fang. Yichuan Wang, Jinjian Liu, and Ai2 AllenNLP team members for valuable discussion and feedback.
SM was supported in part by grant from DARPA to the Simons Institute for the Theory of Computing. PWK was supported by the Singapore National Research Foundation and the National AI Group in the Singapore Ministry of Digital Development and Information under the AI Visiting Professorship Programme (award number 11VP-2024-001),, and by the AI2050 program at Schmidt Sciences.
REFERENCES
Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. Retrieval-based language models and applications. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts), pp. 41-46, 2023.
Parishad BehnamGhader, Santiago Miret, and Siva Reddy. Can retriever-augmented language models reason? the blame game between the retriever and the language model. arXiv preprint arXiv:2212.09146. 2022.
Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. Elastic ChatNoir: Search Engine for the Clue and the Common Crawl. In Leif Azzopardi, Allan Hanbury, Gabriella Pasi, and Benjamin Piwowarski (eds.), Advances in Information Retrieval. 40th European Conference on IR Research (ECIR 2018), Lecture Notes in Computer Science, Berlin Heidelberg New York, March 2018. Springer.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean- -aantttttstt Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. International conference on machine learning, pp. 2206-2240. PMLR, 2022.
Mingyang Chen, Tianpeng Li, Haoze Sun, Yijie Zhou, Chenzheng Zhu, Haofen Wang, Jeff Z Pan, Wen Zhang, Huajun Chen, Fan Yang, et al. Research: Learning to reason with search for Ilms via reinforcement learning. arXiv preprint arXiv:2503. 19470, 2025.
Together Computer. Redpajama: An open source recipe to reproduce Ilama training dataset, apr 2023.
URL httthttts://gtthaa com/ htw.aaa....c.oocteeeecoooootttee httpe//rrecooottteeeeaaaaaaaaaaa -amaaaaaaaaaaaa....aaaa
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur' elien Rodriguez, Austen Gregerson, Ava Spataru, Bap tiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cris tian Cantón Ferrer, Cyrus Nikolaidis, Damien Allonsius. Daniel Song. Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriele Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guanglong Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang. Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Ju- Ju-Qing Jia, Kalyan Vasuden Alwala,
11

---

Published as a conference paper at ICLR 2026
K. Upasani, Kate Plawiak, Keqian Li. Ken-591 neth Heafield, Kevin R. Stone, Khalid EI- El-Arini, Krithika Iyer, Kshitiz Malik, Kuen ley Chiu, Kunal Bhalla, Lauren Rantala- Yeary. Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melissa Hall Melanie Kambadur, Mike Lewis. Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Niko lay Bashlykov, Nikolay Bogoychev, Niladri S. Chatterji. Olivier Duchenne, Onur cCelebi, Patrick Alrassy, Pengchuan Zhang. Pengwei Li, Petar Vasić, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Ro main Sauvestre, Ron nie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sa hana Chennabasappa, Sanjay Singh, Sean Bell, Scohyun Sonia Kim, Sergey Edunov, Shaoliang Nic, Sharan Narang. Sharath Chandra Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Vir ginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whit ney Meers, Xavier Martinet, Xiaodong Wang. Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag. Yashesh Gaur, Yasmine Babaci, Yiqian Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zhengxu Yan, Zhengxing Chen, Zoe Papakipos, Aaditya K. Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adi Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonbardi, Po- Yao (Bernie) Huang. Beth Loyd, Beto de Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang. Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai. Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Shang- Wen Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le. Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzm an, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpem, Govind Thattai, Grant Herman, Grigory G. Sizov, Guangyi Zhang, Guna Lakshminarayanan. Hamid Shojanazeri, Han Zou, Hannah Wang, Han Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill. Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang. Kaixing(Kai) Wu, U KamHou, Karan Saxena. Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, A Lavender, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedr, Madian Khabsa, Manay Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo. Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White. Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pe dro Rittner, Philip Bontrager. Pierre Roux. Piotr Dollár, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak
12

---

Published as a conference paper at ICLR 2026
Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shugiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe. Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sung-Bae Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Andrei Poenaru, Vlad T. Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xia Tang. Xiaofang Wang. Xiaojian Wu, Xiaolan Wang. Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vetito, 7ef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The laama herd of models. ArXiv, abs2240..11833 2024. URL api. att.:ppp...matt.....a.aaaammaa.. www.:/aaaanttcccccccccccc wwsccccshhhaoooo Cttp://///oo.1111111111111111110
Alex Fang, Thomas Voice, Ruoming Pang. Ludwig Schmidt, and Tom Gunter. Reusing pre-training data at test time is compute multiplier. arXiv preprint arXiv:25 1,4234,
Shangyi Geng, Wenting Zhao, and Alexander M. Rush. Great memory, shallow reasoning: Limits of kmn-lms. ArXiv, abs/2408. 1115, 2024.
Corpus_D:27191565,
Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. Olmes: A standard for language model evaluations, 2025. htt.:tttt///axi...ooo abs/
2106.08440
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding, 2021a. URL https://
wrxiv.org/abs//20 00300
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardr. Measuring mathematical problem solving with the math dataset, 2021b.
URL
03874
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint /2.1v:2//2.0//// 18, 2021.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning, 2022. URL https: /tte: wwwp:iiswwrgaa///..aa/a.aaabass
httppp////aaa....ra//aa///////1 htt.aa//////ae..ooe. 09118
Suhas Jayaram Subramanya, Fnu Devvrit, Harsha Vardhan Simhadri. Ravishankar Krishnawamy, and Rohan Kadekodi. Diskann: Fast accurate billion-point nearest neighbor search on a single node. In NeuriPS, 2019.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search.
IEEE transactions pattern analysis and machine intelligence, 37 2010.
AQ Jiang, Sablayrolles. A Mensch, C Bamford, DS Chaplot, Ddl Casas, F Bressand, G Lengyel, G Lample, I. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:23 10..23300.000 06825, 2023.
Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. Long-context Ilms meet rag: Overcoming challenges for long inputs in rag, arXiv preprint arXiv:2410.05983, 2024.
Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-rl: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.095 16, 2025.
13

---

Published as a conference paper at ICLR 2026
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scall similarity search with gpus. IEEE Transactions on Big Data, 7(3):535-555 547, 2019.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Annual Meeting of the Association for Computational Linguistics, pp. 1601-1611, 2017.
Vladimir Karpukhin. Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, November 2020. Association for Computational Linguistics. doi: 12t65//////2222mmmmmmmmmomaaa
URL https: www. ac www. org/ /
2020. e.0.0emmlpp-aale 550
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming- Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. TACL, 2019.
Kenton Lee, Ming-Wai Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Annual Meeting of the Association for Computational Linguistics, pp. 608660866999 2019.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Rerrieval-augmented genera- tion for inowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33: 9459-9474, 2020.
Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Kch, Kushal Arora, et al. Datacomp-lm: In search of the next generation of training sets for language models. Advances in Neural Information Processing Systems, 77:14200-142822 2024.
Jeffrey Li, Alex Fang, Georgios Smymiss Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Kch, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby. Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-Im: In search of the next generation of training sets for language models, 2025a. URL URL httthhtts://// http://aavvvvvv. or.g/rrgg/oooo abs/ /2002 11.006...9999 11794
Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-ol: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366. 2025b.
Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:250..21776 2025c.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. Ra-dit: Retrieval-augmentt dual instruction tuning. In The Twelfth International Conference on Learning Representations, 2023.
Jinjian Liu, Yichuan Wang. Xinxi Lyu, Rulin Shao, Joseph Gonzalez, Matei Zaharia, and Sewon Min. Ds serve: A framework for efficient and scalable neural retrieval. 2026.
14

---

Published as a conference paper at ICLR 2026
Niklas Muennighoff, SU Hongjin, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. In JCLR 2024 Workshop: How Far Are We From AGI, 2024.
National Library of Medicine. Pubmed baseline 2023 repository.
html 2023.
OpenAl. Introducing gpt-5. OpenAl website, August 2025. URL 5/|
com/
Accessed: YYYY-MM-DD.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapinn Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Floundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu. Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang. Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, tukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Míshkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichhrr Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng. Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder. Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr. John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam. Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss. Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https: http:////arvvva///aa/ wttg/bb///3at 08774
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text, 2023. WtRLhttps:. ://ariiv......//// orq/abs/2310 06786
15

---

Published as a conference paper at ICLR 2026
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:308 10800811-30844 1-30849, 2024.
Fabio Petroni, Aleksandra Pikuus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thome, Yacine Jernite, Vassilis Plachouras, Tim Rocktaschel, and Sebastian Riedel. Kilt: benchmark for knowledge intensive language tasks. In North American Chapter of the Associa- tion for Computational Linguistics, 2020. URL httppp/s/.a...eemmm......o....e www.ntesscooscaa.nc..oo
CorpusID:221507733
Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Dmytro Okhonko, Samuel Broscheit, Gautier Izacard, Patrick Lewis, Barlas Oğuz, Edouard Grave, Wen-tau Yih, et The web your oyster- knowledge-intensive nip against very large web corpus. arXiv preprint arXiv: 12. 2w..112..9999444 2021.
Jake Poznanski, Jon Borchardt, Jason Dunkelberger, Regan Huff, Daniel Lin, Aman Rangapur, Christopher Wilhelm, Kyle Lo, and Luca Soldaini. olmoer: Unlocking trillions of tokens in pdfs with vision language models, 2025.
025p:///22./////22222.22222 18443
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman, Gpqa: A graduate- google-proof q&a benchmark, 2023. URLptttps://art
wrg/ / 12022
Leonard Richardson. Beautiful soup documentation. April, 2007.
Rulin Shao, Jacqueline He, Akari Asai, Weijia Shi, Tim Dettmers, Sewon Min, Luke Zettlemoyer, and Pang Wei Koh. Scaling retrieval-based language models with trillion-token datastore. Advances in Neural Information Processing Systems, 77:91260-91299 2024.
Rulin Shao, Rui Qiao, Varsha Kishore, Niklas Muennighoff, Xi Victoria Lin, Daniela Rus, Bryan Kian Hsiang Low, Sewon Min, Wen-tau Yih, Pang Wei Koh, et al. Reasonir: Training retrievers for reasoning tasks. arXiv preprint arXiv.2504.20595, 2025.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettle- moyer, and Wen-tau Yih. Replug: Retrieval-augmented black-hox language models. arXiv preprint arXiv:2301. /2652, 2023.
Weijia Shi, Akshita Bhagia, Kevin Farhat, Niklas Muennighoff, Jacob Morrison, Pete Walsh, Dustin Schwenk, Shayne Longpre. Jake Poznanski, Allyson Ettinger, Daogao Liu, Margaret Li, Mike Lewis, Wen tau Yih, Dirk Groeneveld, Luca Soldaini, Kyle Lo, Noah A. Smith, Luke Zettlemoyer, Pang Wei Koh, Hannaneh Hajishirzi, Ali Farhadi, and Sewon Min. FlexOLMo: Open language models for flexible data use, 2025.
Luca Soldaini and Kyle Lo. P2S20 (Pretraining Efficiemtly on S2ORC) Dataset. Technical report, Allen Institute for AI, 2023. ODC-By, https://githube com wtt.:/ccommaaaaaae wwtpm//aaanaaaaeeeaaaaa
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Sehwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannanch Hajishirzi. Iz Beltagy. Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus three trillion tokens for language model pretraining research, 2024. URL htt.tttss::/ aaxxttxxti org/ abs /20022222222 00159
16

---

Published as a conference paper at ICLR 2026
Huatong Song. Jinhao Jiang. Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05590 2025.
Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Fei Huang, and Yan Zhang. Zerosearch: Incentivize the search capability of ilms without search- ing. arXiv:2505.04588. 2025. URL https://appi www.aeeecccccccccacaoooooo.. wwt.:achhh.aaa.oooooooousss.ss.. 278367823
Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is chatgpt good at search? investigating large language models as re-ranking agents. arXiv preprint Stti::2304.095422 2023.
Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, november 2024. URL https://qvenlm. github. (u/blog/(wqg-32-preveees 2024.
UncleCode. Crawl4ai: Open-source Ilm friendly web crawler & scraper. https://github. com/
uncleccoee/craw142 2024.
Shufan Wang. Yixiao Song, Andrew Drozdov, Aparna Garimella, Varun Manjunatha, and Mohit lyyer. Knn-lm does not improve open-ended text generation. In Conference Empirical Methods org/
in Natural Language Processing, 2023. URL https: /aps:://ppii aapi semanticscoolar. org/
CorpusID:258B659
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang. Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: more robust and challenging multi-task language understanding benchmark, 2024. URL htt.::/:///...l..oor/a////////// 01574
Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun
Cho. Naturalproofs: Mathematical theorem proving in natural language, 2021. URL https
//arxiv.. /arxiv.org/bb///222 wttpg////abaaa.aaaa 01112
Junde Wu, Jiayuan Zhu, and Yuyuan Liu. Agentic reasoning: Reasoning Ilms with tools for the deep research. ArXiv, 044444 2025. URI. httpt:tt///// semant. cscho ar. org/
CozpusID:27623535
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 085382250..09388 2025.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang. Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023. URLpttps: //tp://rxiii http:///ww...oo//////////////// 06364
17

---

Published as a conference paper at ICLR 2026
A METHOD DETAILS
A.1 DECONTAMINATION IN PREPROCESSING.
Before Dedupe
To ensure the robustness of our evaluation, we filter out paragraphs in the these data with over 70% 13-gram Jaccard similarity with any query in all of our evaluation datasets following (Borgeaud al. 2022). To quantify the effect of decontamination, we present the aggregated results from running RAG with eight single-source datastores before and after decontamination in Figure?? 2 We observe a slight decline in performance on MML.U, MMI.U Pro, and AGI Eval. Surprisingly, the performance improves slightly on GPQA and significantly suspect that this is due to two factors: (1) both GPQA and MATH are challenging tasks that involves intense
60-
Average Score
m MATH
50-
40
30
20
MMLU
MMLU Pro AGI Eval
GPQA
MATH. We
Figure 2: Effect of decontamination across eight sigle-source datastores.
reasoning and (2) duplicated n-grams in the retrieved passages might affect the chain-of-thought process negatively.
A.2 COMPACTDS STATISTICS
T reports the number of passages, number of words, and number of chunks for each one of the twelve data sources. In total, COMPACTDS contains 639.2M passages with 380.5B words in total. We use a chunk size of 256 words following (Shao et al. 2024), resulting in 1.85B chunks.
Table 9: Statistics of the data source for COMPACTDS.
# Documents (M)
Words (B)
Passages (M)
Math
6.4
7.8
33.7
High-quality CC
407.3
171.9
895.1
Books
0.2
7.8
69.5
Academic Papers (PeS2o)
7.8
49.7
198.1
Wikipedia (Redpajama)
29.8
10.8
60.5
Reddit
47.3
7.5
54.1
Wikipedia (DPR)
21.0
2.2
21.0
Stack Exchange
29.8
9.2
50.5
Github
28.8
17.1
84.3
PubMed
58.6
3.6
60.4
Academic Papers (Arxiv)
1.6
44.9
Total
639.2
380.5
1,851.8
Index Hyperparameter Descriptions.
We adopt FAISS (Johnson et al. 2019) to build the ANN index described in $3.2 The descriptions for each of IVFPQ hyperparameters are listed below:
Number of Clusters is the number centroids that the vector space is partitioned into during nearest neighbor search. It positively correlates with index building speed and negatively correlates with the search time. A suggested value #cector..
Number of Sub-quantizers is the number of subspaces that the original vector is divided into for product quantization. It positively correlates with the performance, and is linear with the size resulting index. Some common values are 16, 32, 64.
Number of Training Samples is the number of training samples used ot train the sub-quantizers and the product quantization centroid set. A suggested value is 0.05 x #vectors.
Number of probes. the number of centroids to search for during search time. positively correlates with the performance and the search time.
18

---

Published as a conference paper at ICLR 2026
Table 10: Comparison between IVFPQ and Flat indices with Math and PeS2o. All results Llama
3.1 8B Instruct with k = 3.
MMLU
GPQA
MMLU
AGI Eval MATH
AVG
STEM Human. Social Others
Phys
Bio Chem
No Retrieval
60.2
72.0
78.7
68.9
39.8
56.2
46.9
26.7 47,4
25.7 48.3
Math
TVFPQ
64.2
73.5
80.3
57.4
50.0
22.4
50.7
Flat
63.5
73.1
80.4
70.6
44.1
58.0
52.7
31.6 47.4
26.8
51.6
PcS2o
IVFPQ
58.8
73.6
79.8
70.0
42.1
55.9
45.7
30.5
53.8
29.0
49.4
Flat
59.4
73.5
80.2
69.8
42.3
55.5
45.1
32.6
52.6
28.4
49.4
Reproducibility and Index Type As we use Inverted File Product Quantization (IVFPQ) index for approximating nearest neighbor search, potential discrepancies could exist among resulting indices from different training runs. To provide reference for reproducibility, we build Flat indices for Math and PeS2o that performs exact nearest neighbor search. We measure Recall @ 10 on GPQA for the IVFPQ indices using the retrieval results from Flat the ground truth, and obtained 0.82 and 0.80. Table Table[[0|shel shows the comparison between the performance Flat and IVFPQ. We see that Flat outperforms IVFPQ by 0.9 on average with Math and performs similarly with IVFPQ with PeS2o. To exactly reproduce our results, the user should use Flat Index to be deterministic.
A.3 LM RERANKING
Dense retrieval can be limited it is based on the inner product without cross-attention between the query and the passage. To mitigate this, we consider an optional reranking stage using an LLM (Sun) et al. 2023) between dense retrieval and generation. The reranker r assigns relevance score to each passage given a query 4. We do so by carefully designing a prompt that presents and LLM and instructs to generate helpfulness score (see details in CC6)). We then select the
top-kreeank passages with the highest scores fI, P2, Phunn = arg TapTTToprrrakke " ( (p,lg) (q), which then replace Φ1. -.. P& the generation stage. our experiments, we make sure that the LM used for reranking is always identical to the generator. See Table 25r the full prompt. and see the ablation using this method Section C.4
B EVALUATION DETAILS
As described in 4.1 We evaluate COMPACTDS using the following benchmarks:
MMLU (Hendryeks al., 2021a) is widely used benchmark for general knowledge and reasoning, consisting of 57 multiple-choice tasks spanning subjects from college-level math, physics, and computer science to professional-level law, medicine, and psychology. These tasks are grouped into four broad categories: STEM, Humanities, Social Sciences, and Other.
MMLU Pro (Wang et al. 2024) is a challenging, reasoning-focusee henchmarks across 14 disciplines, including math, physics, chemistry, business, and philosophy. Each question comes with ten answer choices.
AGI Eval Zhong et al. 2023 includes questions from human-centric standardized exams such as the SAT and LSAT. We evaluate the English multiple-choice tasks with five options each.
GPQA (Rein et 2023) includes 448 expert-written, written, graduate-level multiple-choice questions in biology, physics, and chemistry, requiring deep understanding of topics like quantum mechanics, thermodynamics, and classical mechanics. GPQA is designed to be web-search-proof being difficult for non-experts with unrestricted web access. We evaluate on GPQA using Chain-of- Thought (CoT).
MATH (Hendrycks et al. 2021b) is a collection of challenging, competition mathematics problems, sourcing problems from exams such AMC 12 and AIME, We evaluate using Co'..
19

---

Published as a conference paper at ICLR 2026
GPQA Diamond (Rein al. 2023 is high-quality subset of GPQA with 198 samples where most of the expert annotators answered correctly but most of the non-expert annotators answered incorrectly.
MATH-500 (Lightman et al. 2023 is subset MATH (Hendrycks et al. 2021b) with 500 samples randomly selected by OpenAL.
Table IT shows the specific evaluation setups for the five main tasks, using OLMES (Gu et al. 2025) as our evaluation framework. For GPQA and MATH, we use the GPQA:OSHOT, COT:: LLAMA3. and MINERVA MATH:LLLAM3.. configurations from OLMES, respectively, which reproduces the evaluation configuration used by Llama 3.1 in (Dubey et al. 2024). We use the MMLU:MC: OLMES configurations for MMLU,
HMLU_PRO:MC::: PRO:MC:NONE: and BEL_LEEVA___LIIIIE MMLU Pro, and AGI Eval, respectively.
For all benchmarks except GPQA, randomly sample 100 questions per fine- fine-grained category. Fine- groeegr categories for MMLU and MMLU Pro are defined in their respective works (Hendrycks al. 2021a Wang et al. 2024,, while we OLMES' selected categories for AGI-Eval and MATH. For GPQA, we evaluate on the entire GPQA main set (Rein et al. 2023).
For evaluation GPQA diamond set and MATH-500 (both well- dolumeneddd subsets of GPQA and MATH, respectively), we use the generation parameters from Search-ol (Li et al. 2025b). Specifically, the generation settings 0.8, a top_k of 20, and a repetition penalty of 1.05. However, for our simple retrieval, we use the same prompting style from the SPOA:OsSOT_coor:LL... configurations.
maximum of 32,768 tokens, temperature of 0.7, top_p of
and MINERVA MLNERVA_MATH::LLAN MATH::LLMM33.
For MMLU and GPQA, we report micro-averages over the reported broad categories. We use the caafffi-ccaeeory-oo-ooooo bw-broad-cateoory map for MMLU STEM, Humanities, Social Sciences, and Other reported in (Hendrycks al.) 2021a). For GPQA, questions include metadata mapping them to their respective broad categories (Physics, Biology, Chemistry). For other benchmarks, we report an aggregate micro-average over the whole benchmark.
Table 11:
Evaluation Setup for the five tasks evaluated with COMPACTDS.
# Categories
Samples
Type
Chat Format
# Demonstrations
MMLU Hendrycks et al.
202
57
5,700
MC
MMLU PRO (Wang el al
2024
14
1,400
MC
X
5
AGI Eval Zhong
8
800
MC
X
0
GPQA
Remn | 2023
3
448
COT
MATH (Hendrycks et al.
2021b)
7
700
COT
C ADDITIONAL RESULTS ON COMPACTDS
C.1 DATA SOURCE LEAVE-ONE-OUT
To further investigate the impact of individual data sources from $4.2 we conduct leave-one-out ablations where the retrieved passages from one data source are excluded at time. Table 12 reports the complete results. For instance, MATH drops 2.6% when Math is excluded, MMLU Pro drops 1.6% when Educational Text is excluded, and GPQA Bio drops 3.9% when PubMed is excluded. This implies that the diversity of the data source is crucial. However, the performance can sometimes improve when excluding data sources, e.g., GPQA Bio improves by 5.1% after excluding High-quality CC, indicating that the retrieval can sometimes pick up noisy passages.
C.2 IMPACT OF NUMBER OF RETRIEVED PASSAGES
Figure 3 shows the impact of varying the number of retrieved passages to feed into the generator (k). At = 10, we observe the saturation in performance across the datasets. With GPQA and MATH the two COT datasets- the performance declines significantly after A = 10. We find that this is due to the observed phenomenon that the model begins to generate blank outputs when provided with more retrieved passages.
20

---

Published as a conference paper at ICLR 2026
0.8
0.6
0.4
MMLU
MMLU Pro
AGI Eval
0.2
GPQA
MATH
Average
0.0
0
3
10
20
50 k (# Retrieved Passages)
100
Figure 3:
Effect of number of retrieved passages with COMPACTDS. 'X' represents the points where apply the reordering method from (Jin et al. 2024) at k 100.
Table 12:
Leave-one-out ablation for each data source in COMPACTDS. Numbers that increase the most per dataset are bolded black; numbers that drop the most are bolded in blue.
MMLU
GPQA
MMLU Pro AGI Eval MATH
AVG
STEM Human.
Social Others
Phys
Bio Chem
No Retrieval
60.2
72.0
78.7
68.9
39.8
56.2
46.9
26.7
47.4
25.7
48.3
COMPACIDS-ANN only (k = 3)
64.6
76.4
84.3
75.3
47.7
58.9
50.3
26.7
44.9
26.8
52.2
Leave-one-out (ANN only) (k: 3)
Math
64.0
76.6
84.1
57.8
47.7
25.1
25.1
51.0
Educational Text
65.0
75.0
83.0
73.6
46.1
58.1
50.9
27.3
43.6
25.1
51.5
High-quality CC
64.4
75.7
84.0
74.2
47.6
58.2
49.3
27.8
43.6
26.8
51.8
PubMed
64.6
76.4
84.3
75.2
47.9
58.9
50.3
26.7
41.0
26.8
52.1
StackExchange
64.1
76.5
84.2
75.3
47.4
59,0
50.6
26.2
44.9
26.8
52.1
Reddit
64.9
75.7
84.2
74.9
47.6
58.8
50.6
26.7
44.9
26.8
52.1
Books
64.4
75.9
84.2
75.2
47.5
58.8
50.3
26.7
46.2
27.3
52.2
Wikipedia (DPR)
64.5
76.5
84.3
75.2
47.9
58.9
50.3
26.2
44.9
26.8
52.2
Wikipedia (Redpajama)
64.7
76.4
84.3
75.2
47.7
59.0
50.1
26.7
44.9
26.8
Github
64.7
76.4
84.3
75.3
47.7
58.9
26.7
44.9
26.8
Academic Papers (ArXiv)
64.5
84.3
75.3
58.8
30.5
26.8
52.5
Acudemic Pupers (PeS2o)
64.6
76.0
84.2
75.7
47.8
58.9
50.9
31.0
50.0
28.4
53.0
COMPACTDS (I: 10)
66.8
77.9
83.2
77.0
53.1
58.9
55.9
29.4
47.4
29.0
55.1
Furthermore, we explored the reordering technique at = 100 from (Jin et al. 2024) designed to improve the performance with long-context retrieval, where the most relevant passages are put the beginning and end of the list of retrieved passages and the most irrelevant ones are hidden the middle of the list. However, we find this technique ineffecctvee in our setting according Figure
C.3 EFFECT OF TWO-STAGE PIPELINE COMPACTDS
[Iacard et al., 2021 for ANN and GRITLM- 7B (Muennighoff et al.) 2024) for exact nearest neighbor search. We experiment with different permutations of these two models over the two stages of the pipeline with 1% of COMPACTDS. Results reported in Table B GRIT, as the larger model, results in higher ANN-only performance than CONTRIEVER, Using Exact Search with CONTRIEVER does not improve or even decreases the performance. Surprisingly, ANN with Contriever Exact Search with GRIT outperforms using
As discussed in 3 we use CONTRIEVER
21

---

Published as a conference paper at ICLR 2026
Table 13: Comparison between Contriever and GritL for ANN and Exact Search I 1% of COMPACTDS with & = 3. ES stands for Exact Search. The best numbers are shown in bold.
MMLU
GPQA
MMLU Pro AGI Eval MATH
AVG
STEM Human.
Social Others
Phys
Bio
Chem
No Retrieval
60.2
72.0
78.7
68.9
39.8
56.2
46.9
26.7
47.4
25.7
48.3
ANN only with Contriever
60.5
73.8
79.8
71.0
43,9
56.6
44.7
32.6
42.3
25.7
49.4
with Contriever
60.3
73.4
80.4
70.5
42.6
56.9
46.4
24.6
47.4
30.1
49.4
+ES with GritLM-7B
61.1
74.6
79.5
71.5
42.8
56.2
47.1
30.5
59.0
34,4
50.8
ANN only with GriLLM-7B
59.9
72.2
79.1
71.5
41.4
57.1
47.0
32.6
47.4
30.1
49,9
+ES with Contriever
61.2
73.2
80.2
71.0
43.0
56.6
47.1
29.4
42.3
25.7
49.5
+ES with GritLM-7B
59.8
73.7
79.8
71.1
42.9
56.5
45.7
35,8
52.6
29.0
50.2
Table 14: Comparison between two levels of approximation for ANN with COMPACTDS. The results
k 10. The best results shown in bold.
Performance
MMLU
MMLU
AGI Evall
MATH
GPQA
AVG
No Retrieval
68.9
39.8
56.2
46.9
24.9
48.3
Subquantizers =
COMPACTDS-AN Only
125GB
COMPACTDS
74.4
51.7
59.2
$4.6
30.8
54.1
Subquantizers 256
COMPACTDS-ANN Only
75.2
50.1
57.6
53.3
32.8
53.8
456GB
COMPACTDS
53.1
55.9
55.1
GRIT for both ANN and Exact Search. We hypothesize that this is because different embedding models complement each other.
C.4 IMPACT OF RERANKING
Table 15 quantifies the impact of LM reranking described in $3.2 LM reranking further improves several datasets, particularly MMLU and MMLU Pro, but struggles with MATH and GPQA. We hypothesize that LM reranking demands tailored reranking instructions for each downstream task: for instance, MATH and GPQA use CoT during generation, where reranking does not, and using CoT for reranking could further improve performance. Another possible factor is the limited capacity of our generator (Llama 3.1 8B Inst).
C.5 4x SMALLER INDEX CAN BE ACHIEVED WITH 1 % PERFORMANCE DROP
To assess the effect of approximation in ANN, we build two versions of COMPACTDS by varying the number of subquantizers in IVFPQ. The more compressed version (125GB) trades accuracy for memory, while the less compressed one (456GB) offers higher fidelity. As shown in Table [4] 4x compression reduces index size by factor of 4, with only a 1% drop in average performance. We use the 456GB index as the default in this paper, but the 125GB index still preserves most gains over no retrieval, and this tradeoff can be adjusted based on available memory.
C.6 ORACLE RERANKING IMPLEMENTATION
formalize oracle reranking method discussed 4.2 Given a test query g, ground truth answer and K candidate passages from COMPACTDS-AND the oracle passages are
P Pi.P2
PP = argTopkkeaakksjsK (PMaa (Pum(alp,, :) Pam(alg)).
where (a) represents the probability that the LM assign to the gold answer a. These oracle passages then replace 9ı, 12,2.... P& in the generation stage.
22

---

Published as a conference paper at ICLR 2026
Table 15: Comparison among different retrieval pipelines, using Llama 3.1 8B Instruct and K 1, 000. ES indicates "Exact Search." LM reranking uses Kyerank 100. The best results are shown in bold.
MMLU
GPQA
MMLU Pro AGI Eval MATH
AVG
STEM
Human, Social Others
Phys
Bio
Chem
No Retrieval
60.2
72.0
78.7
68.9
39.8
56.2
46.9
26.7
47.4
25.7
48.3
ANN (Contriever)- ES (GRIT)
66.8
77.9
83.2
77.0
53.1
55.9
29.4
47.4
29.0
55.1
LM Rerunking
69.1
77.8
86.8
78.7
54.6
59.5
53.0
33.7
47.4
33.3
56.0
Table 16: Deduplication with GPT-5 Mini on COMPACTDS with K 1000, 10, and Llama 3,1 8B Instruct.
MMLU Pro
MATH
GPQA
No Retrieval
39.8
46.9
29.9
Before Dedup
53.1
55.9
32.4
Rolative gains from No Retrieval
33.4%:
19.2%
8.4%
After Dedup
48.0
48.0
33.9
Relative gains No Retrieval
20.6%
2.2%
13.4%
C.7 A LOWERBOUND WITH LM DEDUPLICATION
To consider potential duplicated text with minor format difference with the query, which would be neglected by the 13-gram filtering, we adpot GPT-5 Mini (OpenAl 2025) identify retrieved passages by COMPACTDS (See Table 27 for our contamination detection instruction). Despite instructing the model to only identify the passages that contain the exact same question and answer, the model still marks the passages that contain either the question and multiple candidate answers without identifying the correct one or information that can be used to infer about the answer to the question (e.g., shown in Table 17 and Table T8 As result, consider this decontamination to be overstrict and treat it as a lowerbound of COMPACTDS's performance. We conduct such deduplication on three datasets that in Table [6]
likely contain deuplicates, and the results shown
23

---

Published as a conference paper at ICLR 2026
Table 17: Example of a False Positive from MMLU Pro in LM Dedupli- cation.
The engineer Camillo Oliver was 40 years old when started the company in 1908, Àt his factory in Ivrea, he designed and produced the firet Italian typewrizer, Today the compeny's s head cffice a still in ivrea, near Turin, but the company much largar than it offices all around the world, 1935 there was a ataff of Y00 and the company Lurned out 10, machines a year, but olivetti exported more typewriters to other countries, countries.Camill's sen, Adriano, started working for the company in 1324 and later he becane the boss. le introduced standard speed for the preduction Line and he employed technology and design spacielists, The company developed nev and batter typewriters produced the ELEA computer ayatem, This was the fi:st mainframe wtw.aa/leTTaaaaaaeeooooooooaa
in those days and there are
Italy, went customers
then salculaters (Su8hal (17677166733)
1959 it
designed Adrianc died in 1960, the company had period of financial prcblens. Other companies, especially the Japanese, made faster progress electroric tecanclogy than the Italian company, in 1978, Carlo de Benedetti became the new heas, Olivatti inereased its marking and service networks and made agreements with other compantes equipment. Soon it became one of the world's leading companies in information technelogy and commnications. There are new five independent companies in the olivetti grouplu2014ene for parsonal computers, one for syst and services, and two for telecommunica wo...oo.....ntttooor......... Anquestior From the learn that
Italy.After
design and produce
advanced office
..An A. by 1930 olivetti produced 13, 100 typewriters yearln 8, olivatt earned more in the 1960s than in the 1950s\n some Olivetti\u2019s 750 Italy'n D. olivetti set offices in other countries From the very beginning \nAnswer:
reqularly visited customers
Query/Answer
the mest successful and prosperous companies in the quite inmature Italian industrial scenario. The first fastory in Ivres enlarged, fleor space was addad and a considerable number of new empl cyses were hi red; between 1914 1929, the production typewrizers rose times from 1,000 13, 000. New typewriter models were added to the line, such as the X20 in 1920 and the M40 10 years later, and Clivetti started International expension, opening sales offices in six foreign countries. Fig. 4.1 The clivetti x1, Source: ww.olivetti. in 1924, Adriano olivetti started working in the family bisiress with Domenico Burzio, be travelled around the USA and visited many factories. This tour opened he decided to put in practice, as he recurned to Italy, to modernize Olivetti, through innovative and advanced protects: function based management, decentraaized staff organizatien, rationalizatien of assembly work, davelopment of an efficient national and internationsl sales network. organization produced very positive affecta, increaaing productivity and sales. In 1931, Adciano travelled to the Soviet Union with del egat
apprentico. ahe year later, tagether
offered hin a sange new idaas
Iea' lan irdustrfalissa later that year, he eatabl ished an advertiaing department in Glivetti, whien stazted working straightaway with famous actists and designers and he eventually se: up the organizstion effice. Juring this period, olivetti incerested in the wal and trying to improve the living conditions of ita workera, 1926, company started building houses for and managed to set
active the secial front,
Retrieved
24

---

(Izacard et al. 2022 Published as a conference paper at ICLR 2026
Table 18: Example of False Positive from MATH in LM Deduplication.
Problem: inin how many ways can people be seated around squaze teble with 2 people
considered equivalent if one ta a rotat on of snother. Inn.er...nnnnooo in ion: There are $814 ways to place the people
side? (Tro configurations
the table, but this counts each (if you nove each person 2, 4, er 6 places clockwise you get the seme arrangement). The answer 1a SYNatraci811:4:
acrangement times
Query/Answer
\1boxed/1088085.
0: Arranqements around aquare, People not alt across d'agonally: - Excension The first tue parts this question are exact same as Counting the arrengements of people around a zquare table? and have already been answered which is an extension of the problem cited. Rasically, the final part of the queation ask how to arrange people around a square with 2 people on each side such that people who hate each other won't sit the or on pazallel side, L,e, they cannot sit acrose directly or diagonally. I parts, where part a my answer for the second part my ansver was 3$2Vizimes (7!-61)88 the second answer might be wrong but can eight people be seated at square tanle if there are two
my struggle
the final part my problam
side
solved the first
$871\\time225
confident about the anewer to the first part As
how nany people per side up to rotations by $90*Me,ire,, $180"Mieire9, $270*\ecrre,, er 93c0"Vicires? Your As we preceed clockwise around the tabla from Ann, the others can he seated in 9718 ways. Dividing by W8Virac Wadet (:] 2 Wedat 7!93 as you found. In how many says can eight people seated at square table Ann and Rob do not sit the aide? Again, anawer is correct. Ann can seated in
is correct. Ann
be seated in eight ways,
to acccunt
equivalent rotations yields
Retrieved
D ADDITIONAL RESULTS ON SEARCH ENGINES
D.1 SEARCH ENGINE AUGMENTATION METHOD DETAILS
We set up a default Google Custom Search Engine for web retrieval (except for blocking results from the huggingface.c domain). The search engine returns 10 results (one page) per query, with paid daily limit of 10,000 queries at a rate of $5 per 1000 queries. Similar restrictions/costs exist for other search engine providers like Bing.
We limit retrieval to 10 results per query due to the aforementioned costs and rate- rate-limiting.
Ranking Details. As with COMPACTDS, we use CONTRIEVER-
referred to as CONTRIEVER, as the default chunk embedder. GRITLM-7B (Muennighoff et al. 2024 used for optional second-stage reranking (generally referred as Grrr-reranking).
Strategies to use Search Results in Context. We explore two methods for using web search results in context:
(1) Rank Extract identifies the most relevant chunk per document using CONTRIEVER Izacard al. 2022) reranking, preserving search result order over the selected chunks.
(2) A Aggregate-RRan aggregates all chunks across search results and reranks them using CON-
TRIEVER; this resembles COMPACTDS's retrieval, with ranking constrained search results.
As some test queries were overly long or complex and caused search failures, we also consider Break-down, which decomposes the query into up to three subqueries, performs a search for each, and aggregates results (following the Aggregate- the break-down method decompose queries into at most three subqueries prior search engine retrieval.
strategy). Table shows the prompt used for
In $ we use Aggregate- Rann our main strategy. Ablations over the different strategies are available in $D.3
D.2 IMPACT OF DECONTAMINATION ON SEARCH ENGINE RESULTS
As described in $ to prevent contamination, we apply striet filtering by removing any paragraph (delimited by "Inin") J in a retrieved document that shares any 13-gram overlap with its corresponding query, following Shao et al. (2024), and also block search results from huggingface.co. Table T9
25

---

Published as a conference paper at ICLR 2026
Table 19: Impact of Decontamination of Search Engine Retrieval Results. We bold the best results between before and after decon- tamination.
Table 20: Impact of data sources in search results.
Note that the first row is equivalent to No Retrieval.
Web
PDFs
MMLU
MMLU
AGI
GPQA
x
68.9
39.8
56.2
29.9
MMLU
MMLU Pro
GPQA
69.3
42.4
56.0
34.6
pipeline (Decon.)
70.5
42.8
31.9
70.4
41.0
60.1
31.9
Before Decon.
73.1
46.4
33,3
70.5
42.8
59.7
31.9
Table 21: Results of methods using Google Search for retrieval augmentation, using Llama 3.1 8B Instruct. The best results are bolded.
MMLU
GPQA
MMLU Pro
AGI Eval
MATH
AVG
STEM
Human.
Social
Others
Phys
Bio
Chem
Retrieval
60.2
72.0
78.7
68.9
39.8
56.2
46.9
26.7
47.4
25.7
48.3
Rank -Extract
62.4
72.8
79.3
71.4
42.9
59.4
51.6
29.9
43.6
26.8
51.1
Aggregate- Rank
61.8
72.6
80.0
71.6
42.8
59.7
51.4
25.7
46.2
32.2
51.3
Break-down
72.5
75.5
81.8
73.8
44.8
58.4
50.4
30,5
38.5
31.1
51.6
quantifies the impact of decontamination, showing that it can reduce downstream performance by up to 3.6%. This underscores the importance of rigorous decontamination when using search engines for evaluation.
D.3 ADDITIONAL ABLATIONS OVER SEARCH ENGINE METHOO
We first compare the different strategies outlined in D.1 to prepare search results for in-context use. Table 21 shows that incorporating search results consistently improves accuracy. Aggregate-<Rat outperforms Rank->Exrrae suggesting that ranking from search engines not necessarily better than those from CONTRIEVER. Break-down performs slightly better overall, particularly on MMLU STEM, MMLU Pro, and GPQA Physics. However, due break-down costing substantially more and only yielding marginal performance gains average, adopt Aggregate Sagnegatee+Rank method. We suspect that query decomposition may require datasct-specific tailoring and stronger decomposition models for more pronounced gains.
default
We conduct a brief investigation over chunk size and the number chunks to feed into the generator k. Table 22 suggests that a larger chunk size generally yields slightly better performance, while the optimal k varies by task. As the variants for (c, k) yield similar performances, we default to using (c = 512, A 3) due it being the cheapest hyperparameter configuration (i.e., smaller number of chunks to embed and shorter prompt length).
We also investigate the impact of GRIT- and LM-reranking over the top K 100 CONTRIEVER- ranked chunks. Table 22 indicates that, unlike with COMPACTDS, GRIT-based reranking on web retrieval results generally yields similar or slightly worse performance than the baseline Aggregate-RRand strategy with CONTRIEVER. However, we do observe gains on GPQA Biol- ogy (46.2%-445 +4).%)) LM reranking improves performance on tasks such as MMLU Pro and GPQA Physics, but leads to degradation on others, such as MATH and GPQA Biology. Similar to COM- PACTDS, this likely demonstrates the weakness of smaller reranker on more reasoning-intensie tasks, especially without enough CoT generation before scoring.
In general, we suspect that reranking may not see as much benefit in the case of web retrieval because of the decreased breadth in retrieval results. Although there may be more than 100 chunks per query to rerank, these chunks come from at most 10 documents; chunks retrieved from COMPACTDS don't necessarily adhere to such restrictions. The performance of oracle reranking further supports this hypothesis, as we see the approximate upperbound gain in Table 22 considerably less than that seen by COMPACTDS with oracle reranking in Table 7
26

---

Published as a conference paper at ICLR 2026
Table 22:
Results of different (c, hyperparameter choices and reranking methods over Google Search retricval results with Llama 3.1 8B Instruct. indicates the chunk size, and k indicates the number of chunks fed into the generator. We use the Aggregate Rank method for all variations. For reranking strategies, we rerank the top K = 100 CONTRIEVER-ranKe chunks. The best results across c, hyperparameters and the ablations over reranking strategies, respectively, are bolded (excluding oracle).
MMLU
GPQA
MMLU Pro
AGI Eval
MATH
AVG
STEM luman.
Social Others
Phys
Bio
Chem
No Retricval
60.2
72.0
78.7
68.9
39.8
56.2
46.9
26.7
47.4
25.7
48.3
Ablotions over kc) Hyperporaneters
256,
60.9
72.5
79.3
71.5
42.6
58.3
50.2
27.8
41.0
28.4
50.3
512,
61.8
72.6
80.0
71.6
42.8
59.7
51.4
25.7
46.2
32.2
51.3
c= 512, 10
62.8
72.7
80.1
71.7
43.4
59.3
51.8
26.7
46.2
29.5
51.3
Remonking Strategies with (c 512,
=
3)
Aggregate-<Ram
61.8
72.6
80.0
71.6
42.8
59.7
51.4
25.7
46.2
32.2
51.3
GRIT Reranking
61.1
72.5
79.5
59.5
51.1
27.8
29.5
51.1
L.M Reranking
61.3
73.5
80.3
72.0
44.0
59.8
50.2
32
29.5
51.5
Oracle Reranking
65.9
77.0
84.2
76.6
48.2
61.5
Comparison between different webpage URL parsing strategies with (c 512, 3) and using the Aggregate->te Rank strategy. Static parsing is the default method The best results are in bold.
Table 23:
Table 24:
Comparison between using COMPACTDS only and merging COMPACTDS and search engine retrieval results at k 10. The first row equivalent
to using retrieval. Best results are in bold.
described $5.1
COMPACTDS Search Engine
MMLU Pro MATH GPQA
MMLU Pro
MATH
GPQA
X
39.8
46.9
29.9
No Retrieval
39.8
46.9
29.9
X
53.1
55.9
32.4
Static (Ours)
42.8
51.6
31.9
/
53.5
56.4
33.0
Craw14AI
41.9
46.7
33,7
JINA
43.4
48.6
32.4
D.4 SEARCH ENGINE PIPELINE RESULTS WITH DYNAMIC URL PARSING
We perform ablations using dynamic URL parsing methods alternative the static scraping presented in $5.1 We explore two options: 1) Craw14AI (UncleCode) 2024) and 2) JINA Reader AP Both frameworks offer high-quality URL content parsing by rendering pages within browser and then formatting the data into Markdown format. In addition to higher- more complete parsing, the dynamic page rendering with browser sessions allows for increased chances avoid bot detection such as Captchas. Crawl4Al is free, open-source web-scraping framework, while JINA F7 Reader a paid service charging on number output tokens that is simple to sct-up (c.g., prepend https: / r. jinjiaaaa.... ai/ to URLs before performing GET requests). We only replace the static parsing webpage URLs and continue using olmOCR for PDFs because 1) Crawl4AI has more limited PDF parsing than olmOCR and JINA Reader would be much more expensive due the longer PDFs resulting in a far greater amount of output tokens.
After gathering the parsed results, additionally filter out any embedded URLs (e.g., the Markdown conversion preserves URLs on the page) (following (Li et al. 20255)) and parsed texts that contain keywords such "cloudflare" or "captcha" indicating bot detection page was parsed instead of the protected content. We then follow the rest of the pipeline detailed in $5.1 using the Aggregate-<Rank method.
Table 23 demonstrates that the best URL parsing strategy varies across benchmark, but no single method consistently outperforms the others, not even the paid service (JINA). We suspect one possibility could be that the parsing quality, while non-trivial, is not as impactful as the significant
https: / /jinannn.. iaaaaaaa a ataaaaaiii/arerrr/e reader/
"There is free option that avoids API key, hut is heavily rate-limited 20 requests per minute.
27

---

Published as a conference paper at ICLR 2026
amount of unparsable URLs due to preventative measures such as bot detection (e.g., employed on qui griilett.. com), which greatly reduces the number of retrieval candidates. We note that even one were to deploy stronger web-scraping services that could bypass these protections, there are still ethical concerns regarding non-permissive data usage that should be approached with caution.
static parsing as it is the fastest method while also not demonstrating significant performance degradation.
For our experiments,
D.5 COMPLEMENTARY STRENGTHS OF SEARCH AND IN-HOUSE RETRIEVAL.
While we demonstrated that in-house retrieval is competitive alternative search engines in 55.2 we believe that search engines and in-house retrieval offer complementary benefits. COMPACTDS is simple (as they rely solely on vector similarity), self-contained, reproducible, and robust; it avoids the cost and noise associated with web search (see $5.I). In contrast, search engines incorporate complex rule-based and ranking algorithms that complement vector-based retrieval. They also access a broader and more diverse set of documents, including content unavailable to crawlers, such as many web-hosted PDFs.
PDFs, in particular, are notable advantage of web search. While COMPACTDS includes PDFs, they are primarily academic papers and educational text. contrast, the web contains many valuable PDFs, such as lecture notes and problem set solutions, that are largely absent from COMPACTDS. To assess the impact of different source types, we compare performance using only web pages vs. only PDFs (Table 20). PDF-only retrieval can significantly improve performance MMLU Pro, using only PDFs performs nearly as well as using both PDFs and web pages, and for GPQA, PDFs alone even outperform the combined. Overall, we think:
tasks, e.g., on
1. PDFs are valuable source of information complementary to web crawl, similar to how educa- tional text and academic papers were effective in 442
2. Common Crawl does not represent the full scope of web content; expanding COMPACTDS with additional sources such as high-quality PDFs on the web found by search engines could enhance its performance.
D.6 IMPROVING COMPACTDS WITH SEARCH ENGINES
As search engines naturally retrieve over a larger data distribution (c.g., the entire indexed web), we hypothesize that retrieved search results could augment COMPACTDS performance in a hybrid search fashion.
As preliminary hybrid search strategy, aggregate the top- K results from both COMPACTDS and search engine retrieval, re-ordering them using their GRIT-based retrieval scores. We compare this to just GRIT-rerankinn the top K 1000 results from COMPACTDS only. For fair comparison (only reranking 1000 total candidates). we select the top K jocal = 900 results from COMPACTDS and top Kweb 100 results from our search engine. Table 24 shows our merging strategy yields small gains on all benchmarks (53.1% +535555% for MMLU Pro; 55.9%=556.... for MATH; and 32.4%-33... for GPQA, respectively).
Our preliminary merging method is likely not optimal. However, we emphasize that our qualitative observations on the domain diversity of search engine results motivate further exploration into how search engines can complement local datastores outside of treating them as independent retrieval flows. For example, to expand COMPACTDS to specialized benchmarks in other domains (e-g.. law) while also maintaining space efficiency. identifying a concise, relevant, and high-quality subset of data to build on efficiently is a natural first step. One could use search engine results over a benchmark validation subset (potentially using agentic search flows to improve search quality) to identify relevant URLs. Metadata from these URLs (e.g., domain, subfolders) could then be used as a pre- pre-processing filter to reduce web crawl dumps to much smaller, higher-quality subset (similar to how we currently use Fine Web-Edu filtering), or help with targeted crawls to collect data not typically found in pre-training data such as the web PDFss discussed in DD55lan other multimodal media.
28

---

Published as a conference paper at ICLR 2026
E QUALITATIVE ANALYSIS
We provide examples of the retrieved passages for a query from MMLU Pro in Table 28 GPQA in Table 29 and MATH in Table 30 respectively.
With ANN-only, the retrieved passages are relevant to the domain of the query but can sometimes be irrelevant to the query itself. Table 28 asks for the an incompatible application for hash tables, while the retrieved passage only mentions the mechanism of hash table without providing useful information mention for answering the question. In Table 29] the query asks for a calculation of ratio of numbers hydrogen atoms in the second excited state in the extreme temperature in the atmosphere of Sirius. The retrieved passage mentions the temperature in Sirius but are about wavelength, without mentioning hydrogen atoms at all. Table (30 shows another example where the query is about probability of Michael rolling three dice and getting at least two 1s. The retrieved passage with ANN only mentions Michael rolling dice, but it is about the probability distribution of the sum of two dice.
When using GRIT for Exact Search, the retrieved passages become more relevant to the queries themselves. For example, the retrieved passage in Table 29 includes question and solution that is similar to the original query asking about the ratio of the number of hydrogen atoms in the first excited state. The one in Table 30 also mentions the joint probability of two dice throws.
LLM Reranking inconsistently provide more relevant passage than Exact Search. For instance, the retrieved passage in Table 28 specifically mention that hash table is not good choice for range search. Passages that including such direct answer for the query is the most effective ones suggested by the oracle retrieved document also shown in the table. However, in Table 29 the retrieved passage using L.MM Reranking only provide a relevant high level background of hydrogen atoms' excited state around Sirious in the extreme temperature without providing a specific solution the query.
Comparing Retrieved Passages between COMPACTDS and Search Engine. We addition- ally compare examples of retrieved passages between COMPACTDS-LLMN and search engine Agggregate-RRena for queries from MMLU Pro, GPQA, and Math in Tables 31 and 33 respectively. We also include comparison between oracle-reranked retrieved passages from COM- PACTDS and search engine for MMLU Pro in Table 34
We first observe that search engine retrieval typically results in documents that differ significantly in format and nature from the text retrieved from COMPACTDS and are often atypical to the pre-training corpora that COMPACTDS is composed of. For example, Table 33 shows retrieved passages for query from MATH which involves computing the volume of a sphere when given the volume of the cireumseribing cylinder; while COMPACTDS yields a Stack Exchange post discussing how to derive the exact relationship between the volumes of a sphere and its circumscribing cylinder, the search engine yields PDF workshee from K-12 textbook which also derives the same formula alongside many other relationships. Table 34 shows retrieved passages from a query in MMLU Pro about deed validity. Both passages capture the necessary information regarding participation and intention in deed delivery. However, COMPACTDS's 's passage is from related court case on the legitimacy of a property sale, while the search engine result comes from guide about estate planning for Oklahoma farm and ranch fa
We also note that the lexical matching strength of search engine retrieval is inconsistently beneficial. Table 32 shows question about the interpretation of the commutator of two gamma matrices. While the result from COMPACTDS is closely related to the given problem, but not completely sufficient to solve the problem, the search engine result exactly contains the commutator expression. Furthermore, it identifies how it contributes to the angular momentum of the Dirac field and derives the Lorentz transformations well. On the other hand, Table 31 gives an example query that uses Euler's polyhedron formula. While COMPACTDS DS yields a result that exactly contains the required formula and how to use it, the search engine result is lexically noisy and contains numerous unrelated formulas for different 3D shapes. One possibility is that the short question and large number of answers could be distracting during the chunk CONTRIEVE BeREvvR--eeean reranknng stage. Nonetheless, the inclusion of such a document within the top-10 search engine results raises caution regarding search engine susceptibility to lexical distractions.
nttpa: h//ptt////atttt bf
http:///eeeeeeettoooeee////aa/ee /tctt////ttteeeeeeeeeeeeaoeeeeee www.eettttnngggeeeo con//eoo_aaaae pee _0_11______11__11111 11_01.1___11____ 08.por
tttps// on. edu/fao: sheets/estate anning simp if ed- ed--g-ddee-e
and-
mtm
29

---

Published as a conference paper at ICLR 2026
Table 25: Instruction for generating helpfulness score for LLM Rerank-
ing.
Instruction
You are an expert evaluator. Your task evaluate hew much - the context can help solve the question and arrive at the correct answer.
We will provide you with the question and the context. You should first read the question carefully, and then evaluate context based on the scoring criteria
the helpfulness
-
proviced below.
# Question { (full_text]
Context
|retrieval_text:
# Scoring Criteria
Before outputting the score, provide a short reason for the decision, citing specific chunks of text from the context if applicable. Output the score in the range of 1-10, where means the response is extremely unhelpful and means the + response is extremely helpful.
-
Here are more detailed criteria for the scores:
Score 1-2: The provided context is largely off-topic and provićes minimal or helpful information. Its content is very distant from the question at hand.
+
Score 3-4: The provided context has a weak connection to the problem. While it may mention related concepts or offer minor insights, it does not contribute meaningfully to solving the question.
-
Score 5-6: The provided context centains some relevant information, but it doesn't directly help in solving the question. It may provide background context or partial +
information that needs further clarification.
Score 7-8: The provided context is highly relevant and addresses most aspects the question. It provides clear
- and acticnable information, though there may still be minor gaps missing details.
Score 9-10: The provided context is entirely relevant and offers thorough, accurate, and comprehensive information
that directly solves the question. covers all aspects
necessary to fully address the question with precision.
Please output your reason and score JSON object.
30

---

Published as a conference paper at ICLR 2026
Table 26: Instruction for generating sub-query breakdown.
(query\)n\nnewriite the above questicn up three unique
+ search queries to use with search engine to find helpful
relevant information to solve the above problem. Only output
the generated search queries as ison dict with key
seearch_queries pointing to the list generated search
queries. Do not exceed three search queries.
Table 27: Instruction identifying duplication.
Below is question and answer pair and retrieved passage. \nAnswer 'Yes' if: the passage contains the exact same question and answer, ignering minor format
difference, \nAnswer 'No' it: the passage does not contain either the exact same question or the answer, or the answer has to be summarized or inferred from the passage, or the answer has to be selected among other options. \nYou answer should only be 'Yes' 'No' Inlargraa>///qqestiio and answer) /a<///////////// cappnnagapspeeeeeeaooa (ni /ttpa/aassgaaaaa/aaaaaaa/aaaaaaa
answer:
31

---

Published as a conference paper at ICLR 2026
Table 28: Examples of the top retrieved passage for MMLU Pro from COMPACTDS.
Questior: Hash tables can contribute an efficient average-case solutier. for all of the problems described below EXCEPT; sey comparisen: Siven two keys, determine they are Ldentical different.
Count distinct values: Siven set of keys, determine the number of distinct key values.
e, bynamiz dictionary: Support the operatiens of insert, delete, dictionary,
and search
u. Aange search: Giver values and b, find all the records whose the ranga
value
E, Symbol table lookup: Siven program identifier, find ita type address.
F. Frequency count: Given a set of n keys, determine how many
times each key appears.
G. Unique key identification: Given key, find if it is unique set.
Query
H, Sequential search: Given key value, fird the first occacrence the key in the set.
Answer
Making use of the whole key gives better chance of limiting the number of collisions. try spread the keys unimerlly over the table. Avoid function which deliberately biases keya to certain portion of the table. The efficiency of the hashing technique depends mainly the hashing function resolving collisions. The method described above is called the (linebr()inear) method table 18
aethed
gives good results provided the more than about 9014 full...
ANN only
computo the average number of comparisons needed perform succesaful search for value in the table, 13, Khen computer program is comptlldd storing information about the identifiers used in the progran. & scheme is needed to quickly decide whether a giver identifier has siready been stored in the table and, if identifier A hast function is often used to locate a pesition item...
compiler builds a aymbal table
the
+Exact Search
the table which
information
If you to quickly the keys given range, implement any of the other operationa *r the ordered aymbal-table APT on page 366, ther hasting la not linear time...
mirimum key, find
+L.M Reranking
appropriate,
these opecations
... The search efficiency of the range very 1ow, For example, to find the value between key001 and key002, you aeed traverse all the keys to find the correspending answer...
Oracle
32

---

Published as a conference paper at ICLR 2026
Table 29: Examples of the top retrieved passage for GPQA from COM- PACTDS.
Question: Sirius the brighteat star in the sky, The temperature of this star is around 10000 K. Censider Hydrogen atoms in the stmosphere of sirius. what is the ratie of the number of hydrogen the second exeitad atate of Hydroger those ground state?
Choices:
(A) 8.2 + 1
7.26 10--6
(C) 8.11 -0--7
Query
5.4 10+--9
Answer
c
The brightest star in the sky is Sirius, the nog Star. It is actually a binary system: of two stara, the amaller one (Sirius 5) being white dwarf. Spectral anslysis of Sirius B indicates that its surface temperature is 24,00 000 and that it rediates energy at total of M1.0 stimes (25)))
that behavas like an ideal blackbody. (a) What is the total radiated intensity siriva B? wavelength visiale to humans? (c) What is the radiva of Striua B7 Express your enswer kilometers and as a fraction of our sun'a radius...
the peak-intensity wavalength?
ANN only
... temperature Kest hydrogen atoms at the surface of the star are in the electronia ground state, What the approximate fraction of the hydrogen atoma that could emit a photon) 7 The energy of the first axcitad state above (-13.5/22 6,232e 18 J. 2, Relevant equations Rb 1.38e 23 3/X 1/T - B/Kb attempt at solution problem. My guass was to divide the energy in joules by the temperature and then divide by the ba taman constant to make unitless. (1.632e (aste-18/80000 18/8000: /1.000 (1.36e 23) 2. 28, 2010 444 nickjar Amm.... You are right so for, but you might want to use for this problem well. 3. Apr t+t burg25 What do you mean? Could you clarify? 1. Apr 28, 2010 burg25 Wait think understand. Are you saying take raised to this valua? 5. Apr 28, 2010 ### burg25 Yeah I got it. nickjer Yes, using the Boltzmann factor you get: $5Vfrac(P (BV (P e e"(AAtrac (R_1-__R) electron
surface
is
In the first excited atate lard therefore
1-13.6
10.2
really
Boltamann
Thanks 6, Apr 28, 2010
where SP (8)5 18 the probabil the energy SES.
+Exact Search
.. Can we conclude that the atmosphere of Sirius made up largely of hydrogen while the solar atmosphere consists largely of metal atoms? since the early 1920s astronomers have understood that the arise from
canspi diffarences betweer stallar speatra layers differences in the temperature of the atmoapheric specaral di ferreeees in the relative abundances of the chemical elements. To absorb light at the frequency of one of the 3alme: lines, a hydrogen. atem hydroger atoma in this stare dapenda on the temperature (and, much weakly,
formed,
be in ins first excited
The fraction
the temperature of solar shoLospheze (the visible layer of the Sun's almospheze), nearly sll hydrogen azoms aze in ground state, where they can abeorb lines of the ultraviolet Lyman series, The spectrus of Sirius is forned at a tempezature of eround 10,000 X. & much larger (though still nunerically small] fraction
preasure).
+LM Reranking
33

---

Published as a conference paper at ICLR 2026
Table 30: Examples of the top retrieved passage for MATH from CoM- PACTDS.
Problem: If Michael rolls three fair dice, what is the prebebility that he soll fraction.
1's? Express your answer as common
Query
Solutior:
Answer
(22a5
brother Kichsel went to borsow their father's car on Friday night. determine who gezs use car, Carmen wants father to roll a pair of fair dice. If the aum of the two dice is 2, 3, 11, or 12, Carmen gets to use the car, It the sum of the two dice is other number, then the dice will be rolled again. Kichsel thinks that this is Michael is wzong. The probabilizy of zelling sum of 2, 3, 11, $81 (21)) (9) 3S)frao[4] (2)1)ac[1]
10, then Michael
the
If the
is
fair
decido. he correct? Explain. No.
sum of a 4 or 10 88) The probabt lty of rolling $8VIrac12) (21)85 equally
(8) (21)88. Because both Cermen and Michasl likely to get the cas, this would yield a fair decision, Question technology glizch, airlira number of passengers in economy class en a flight from New zork city to los Angales. Currantly, there are 150 passengera who have economy class tickets, but there There are two seats available in first class and one sest svailable in busiress class. ixplain how the ticket agent could use a random number generator to make a fair desision in moving passengers either firat of the plane and to
2. Due
overboaked the
only 141 seats the plane,
sections
ANN only
Problem: Nathan will roll six-sided dice. what the probaniligg that he will roll a number less than three on the die
three on Express your answer as common fractien. Selutien: For the first die with probability #1frac(1] than 3, it must be 4 or a or probabslity {Xérac(1] (2).. The probability bath of these evants accuring, aa they are independent, 1a s\rrac[1] (3) Vodot \Irao(HH2) (boxes/_ccacc(])6))3..
greater
die?
three, it must For the second die to ba greater which occurs with
or which occurs
+Exact Search
0: If you roll three dice, what the probability of getting at least two number seme? If roll three dice, What is the probability of getting at least two numbers the same? A: Hint: compute the chance they all differert, then aubtract from 616.
+LM Reranking
34

---

Published as a conference paper at ICLR 2026
Table 31: Example of top retrieved results from COMPACTDS and search engine retrieval for query from MMLU Pro (Computer Science).
Question: Suppose convex 3d jd-object has 15 vertices and 39 edges. Row many faces does it have?
A. 25
B. 26
c. 20
D. 31
E. 22
F, 28
G. 32
B. 30
I. 24
J. 27
Query
Answes:
Answer
B
Eech trisngular prism has a base area of W
(i (5.5) WV 22 emku0062 and height of 10 41a. Sample answer: 49, semicircla; 180 51, mojor arc; 270 53. 73 mn, 180.3 mm\uc062 41b, Greater than; aquare with side length of 6 a has an ares of 36 mlu00b2. A circle with diameter of 6 a has an area of 9)u03e0 miu00b2. Since the heights are the same, the volume of the aquare priam is greater, 4lc, Multiplying the redius by x; since the volume is represented by \u0Se0rtn00b2h, miltiplying the haight by makes the volume times grester. Multiplying the radius by x makes the volume xhu00b2 times greater.
in. by if., height 4'u0se0 in. per side, height 12Nu03e0 in. 43c. base with legs messuring 3 in. and in., height 10\u03e0 in, 45. Sample answer: 47, Both formulas involve multiplying the area of the base the height. The base of a prism is a polygon, so the expression representing the varies, depending of cylinder is circle, so its area 1s \u03c0r/ub002 49, r em\u00b2;
base
the type of polygen it is. The base
C 53.
cm\u01b2 55.
in\u00b2
11.4
59. 9.3 in. 61. 378 m\ud052 Lesson 12-5 1. 75 inlu00b3 3. 62.4
m/u0cb3 5. 51.3 inku0ch3 7. 28.1 mm\u00h3 9. 513.333 fevu00b3
105.8 mmku00b3 13. 233.8 emtud0e3 15. 35.6 cm/u00b3 17. 235.6
inkut0b3 19. 1473.1 amtu00b3 21. 1072.3 in\u00b3 23. 234.6
emluc063 25. 32.2 tt 27, 3190.6 m/u0003 29. about 13,
BTUs 31s. The volume is doubled. 31b, The volume is multiplied by 2vu00b2 or 4.
volume multiplied by 2 14 in. 35a. Sample answer: 35b
33.
COMPACTDS
Q: A formula to describe the relation of faces, edges and vertices in three-dimensionss yes, what is 15, to describe the relation of races, edgea and vertices in three- dimensional convex bodies. for regulaz shapes: A tetrakedron has faces, 6 edges and vertices Cube: 6 saces, 12 edges, 8 vertices Octshedzen: 8 faces, 12 edges, vertices reataçonsl dodecahedzon: about a n-facad polyhadron? n faces, but how many adges and vertices? Ts there formula to calculate the number of vertioes and edges, given specific number of [aces? Or possible numbers of vertices and edges? Add-on: What happens under the assumption of irregular shopes with that formala? Yes, there is such a formula, It is called Buler's characteristic formule, and stazes that the numbar of edges, and $F$ the number of faces of palyhedron, then ESV-B1F=288 For example, the cube has 585 vertices, $66 faces and $129 edges, and $8 1215-23. The cetahedron has $68 vertices, 989 faces and $128 edges, and again 36-12+6=25. A: suler's formula allows dimensional regression. let $F=\M(f_0, f_1, f_2, Au2026\iJ5 be the facet vector of your polytope SPE, 1.e. SP$ has $f_0s 60S-facats (vertices), $f_18 s16-facets (edgas), $$_25 #25-raceta (facea), etc. Then you have
convex bodses Ta thare formular, and 1f
faces, edges, vercices what
of
the number of vertices, SE$
(-1)
I_ - 1 (11) That formula aleo holds for non zeqular polytopes, provided do nos encounter holes and other odd stuff. ssp. it is valid
Search Engine
35

---

Published as a conference paper at ICLR 2026
Table 32:
Example of top retrieved results from COMPACTDS and search engine retrieval for a query from GPQA (Physics)
Question: Which of the following statements is correct physical interpretation of the commutator of two gamma matrices, 1/2 [ganna"ma "ma, gammarnu]?
1. gives contrihution the angular momantium of the rac field.
2. gives contribution the four-momentum of the Diras field.
2. It generates all Poincer\ud0es transformations
Dirac
field.
It generates all Loreatz transformations the Dizae field.
Choices:
(A] and
(2) 2 and 4
(C) and
Query
(U) 1 and 3
Answer
A
#_Ving M/frac/ (Mfarci) (Vppartial x*Viuu.. Vitag(3.51) From the relations derived it follows that W WhMLk() is the orbital angular momentun projection operator whereaa the translation operator, multiplied (aY)) represents the four-dimensienal enezgy-momenter vestor WPp.immuVV. one can be easily convinced (VLm) MMLLMMM and (P_VVmmMM satisfy the commutaticn
fact, that the opezaLors
relations (3.45) (12013 (3.48)) Relations (3.47) and (3.48)
demonstrate, that the generatora (MMM and WPPM commote with the Mamiltonian WPP_OW, what allows us classity physical states according to eigenvalues of these generators. The eigenvalues of the Lorentz boosts generaters WMN cannot be (MP_OW)
used for this purpose because they de not commute in case of the "imize-dimmesion 2oincare transfermetions. the fields tranaformatien (Li")(\Aaabbda
the form: WI Vipai' _ (x)
a)),, VV where the set of matrices W (B(IAaaaaaa)) W) ferms repzesentations of the Foincere group. Oace again classify all irreducible unitary representations of the group one mast find all the reprasentatians permutazion relations (3.46) (32033 sermitian operators. To achieve this goal fix Casimir operators. let va defina the 50-called Pauli-inbbnskd peeudovector: AV M_Vlsigma (\frac(11
of
Mmu'AamMMMembdde X*(Maau\Nuu) p"Vilambda. Mlvarepsilon_/ (VVeegal
(1tag(3.52) WM Eq. (3.52) W (1 Mom//mmmmn//// is generator Of reprasentation of the proper Lorentz group, and consequant it already a sum In the three-dimeniional vector rotations (X W has the form: WI ghg = p Wedot M, Aiquad (3Lag(3.33) WJ It is obvious that the four dimensional pseudovestor WMLLNssggaa(V
orbital and spin moments
field.
P_0 - [p Wtimas N).
COMPACTDS
Pauli matrices with two spacetime indices \u2022 #1 Cohn Corn 2 ("Fauli matrices with exo spanetime indices Hi all. This my first post so forgive ve ma if my latex doesn't show up correctly. famillar with defining identity matrix to construct four-vector of 2x2 matrices, Mlliggma \mmu. I'm trying to
matrix
2x2
paper which uses $ (Vlsigna" (VAmm Annus.. This i5 betweer a 4-spinor and a gamma matrix. Can someons please enlighten me about what this notation meang? Thanks much. Physies vaguely rememoer it to be the (anti-?) commutator of two gamma matrices. \u2022 anticommutated of gamra matrices is just $2 Vetaa (\ Veu Vanu L_4 Atimes 419, which hardly calla doesn't discusa commutatora in relation to ciifford algbbra, but con't rule that out. \=2022 #4 As for as remember [tex]
notation
Phys.org \u2022 $2
for quick response Dr.
The
new notation. One usually
WSiigra (1Amm Wrract (1) (2/AMeft (1 (VAm)) Vigamma (Maui/eeight)) [/tex] has zo with spin operator massive Dirae field. \u2022 +5 the Sigma matrices are usuelly used during the derivation
the quantized
the Lorentz covariance propertiea of the Dirac equationn Later it is usually shown how to represent the sigma matrices using thre gamma matrices. So strictly apeaking you
transformation
need them for you only need intermediate scep) Back Top
Engine
36

---

Published as a conference paper at ICLR 2026
Table 33:
Example of top retrieved results from COMPACTDS and search engine retrieval for a query from MATH, truncated for viewing.
Problem:
The volume of a cylinder is 60 cubie ceatimeters. What La the numbes of cubic centimeters in the volume of the sphere it circumscribes?
Query
Solution:
Answer
40
eylinder is circumseribed about sphere. their volumes are denoted by $C$ and sa6, find SCS as a function of $8$ llere is problem. eylinder
circumecribed about sphere. If their volumes are dencted by $C$ and sss, find SC$ as a functior. of $85 My (Amended) Actempt: [Based on the correction suggested by herbSteinbergl Let 8:8 be the radius of the sphere. Then the height of the cylinder is 82r3, and the radius of the base is $x8. So the volume SCS of the eylinder is given by 39 C = Vpi r12 [2r]
2 Vpp r*3. Wtagl1) $5 And, the volume $5$ of the aphere ia given by 35 s Wfrred3 Api 2-3, Atagi()
From 12), we obtain
66 r33 a (3rrac33) Apil = Virrac 35 1(4 Wpt 58 and hence es [egrt(33] W3rac Apit J. Wtag(3] $8 Finally,
putting the value of 828 from (3) into (11,
get 8$ C - Mpi
Weeef (3]grt W/rran Apil } Mright]" = Vkii Wlett( Wirro((3) Mpi)) Wright) Mirra322 s.
my
solution correct in each and every detail? or, are there any errors of approach radius of the ey inder 1a just srs, Otherwise your work appears to be correct, bus you've made things harder on yourself than you
answer? A: noted
comments, the
secessary. Nete that Sr*3$ appears in both formilas, have solved for Sr*3 Vifrac (38) (4 you can immedictely use this expresaton
COMPACTDS
Volumea Spheres Resential Question Row you find the volure of a sphere? EXPIORATION 1 Finding the Volune of Sphere Kork with partner. sylinder shown, Write formula for the volume svs of the cylinder in terms radius $rs. 45V Wttxxt (Volume cylinder]35 When half the sphere la hemisphere) is filled with sand and poured into the cylinder, it takes threa hemispheres to fill the cylindar. Use this information
circumscribed about sphere, as
volume in Lecms of the radius $zξ. S$V Vitext une of a sphere Use the formuls you wrota for the volume of a sphere is correct. EXPLCRATICN 2 Sphere Arother Way Work with a partner. The figure shows hemisphers, and cylinder with A plane parallal to their basas intersents the solids $x8 units above their bases, Using
write formula
sphere
Internet scme other resource confirm
Finding the Volure
removed.
AA Similarity Theorem, the radiua of the cross section of the cone at height $2$ is $28. The ares of cross section forned by plane (1)pi 2
both solids, Recause the sclids have the same height and the some cross prosa-sectional area at every level, they have the aame volume by DavalieriNu2013 Principle. volume of the semisp, the volume of the eyl gylinder, and the volume the cone. the formule for the volume of sphere with a radius $28. Search Engine Communicate Your Ansver 3. ROw can you find the
Krite the relationship between the
the relatienship
part (a) derive
37

---

Published as a conference paper at ICLR 2026
Table 34: Example of top retrieved results from COMPACTDS and search engine retrieval for query in MMLU Pro (Law), with oracle reranking applied. Documents are truncated for viewing.
Question: Two brothars owned a parce: of re:l astate as joint tenants. Both brothers signed deed as grantora conveying the land to the buyer, The first brother headed the deed to the second brother with instructicns to take approvel pricr to delivery. Brother two, without his brother's 'a permi ssion, took the deed rerettly to the buyer and collected the azle price, is this good deed to partner one?
to their lawyez for
A. Yes, the deed was algned by both partners, which proved their intent to sell Le bayet, and it was delivered at the Lime of signing by the fact of affixing the'r aighstures to the document.
B. Yes, the deed is valid the buyer
not
of the
internal agreement between the two brothars.
C. No, the deed invalid
partner one did not give explicit permission for partaer two to deliver the deed.
D. Yes, the deed is valid as partner two had the autherity to finalize the deel.
is invalid because partner two collected the sole price without partner one's consent.
g. No, the
F. Yes, the deed is valid as both brothers had signed it, fying their agreement to the sale.
G. Yes, the transfer is valid from both partnerss because partner
two partner ene's apparent agent for purposes of delivering the deed.
R. the deed two atepped outside his scope of authority.
invalid as to both grantors bacause partner
I. the deed cannot bind partner
because he did not
participate in the deed delivery to the buyer and did not intend to deliver the deed up to the grantic at that tima.
No, the deed is not valid as the lawyer did net approve i: before delivery. Anawer:
Query
Answer
I
The absence of tornal doed of csnveyance is strong indication that the parties aid not incend immesiste transter ot ownership. Fourth, patitioner rataiaed possession of the carcitiaata of title of the lst, This is an additional indication tast the agreement did not transter to private respendenta, either proparty. Finally, respendent Juarizo admitted during trial that they have not firalizes the sale in 1992 becsuse there were miner cwners such that when they conatrusted their house tharesa, thay sought the pernission of petitiones.
csastruetive delivery, evnarship of the
whether the suspensive ccadition, 1.e., judicial appsoval ot the sale ot the ninor owerss stares, upon which the ablication of the sellars to execute a dead of sale depands, is ruirilled, Artieie 1186. The condition shall 20 dosmed fulfilled when the obligor voluntarsly prevents its lnent. This prevision reters constructive felfillment fer tvo rsqusites,, nancly: (al the inzent of the caligor to prevent the fulfillment of the condition, and (b) the actuel prevent ion of the fuirillment. Here intention of the debtor ccadition, or to place incifective obstacios to its complianes, without actually preventing the fulfillaent, is insufCiciunt....
the
questior.
resolved
suspeasive condition, whose application calls
the happaning of the
COMPACTDS
... Dunda are recorded the Coatly Clecklu2019e officw the county in which the land is located, Nu2022 Inportance of Delivery, Sonetimes a grantor will xign deed covering land which he or she wants to keep unt il death expeeting ing the deed be the nsthod by shich tille the land transferred. Tratead of delivering the deed direstly to the grantee, the owner planes the deed in a sa safety depoxit box or aning pr private papers. Undelivered deeda 1ike theae are commanly called (u20ledresse: draver Ad201d
deeds are invalid because they grantes or his or her agent daring the lifetine of the grantor. belivery is necessary to aczorp a conveyance by deed, Unless the deed is dravt and exeruted ed in the forn of will (whigh would be extranrdi nary and urosual) ), could not
delivered
legally effective withous delivery of the deed grantee or his sgent... The key to ettective delivery of a deed is that the grantor nust presently ly transfer legal concral over the deed to the granten or Independent third pasty, I2 the yranter retains right to change Eix mind, then control has potential legel problens that may ari in using this type of device sxtale glaming, it is especially important to oblain legal advicw...
the
been presently transterred. Becsuse
Search Engine
38


Reviews:
[{"review_id": "wy3DJ2s222", "reviewer": "Reviewer_DAeh", "summary": "This paper introduces CompactDS, a diverse datastore retrieval. The paper discusses the pipeline on generating this datastore and the method to efficiently serve the system. The paper claims that LLMs can perform better on reasoning-intensive tasks with RAG on this datastore.", "questions": "1. In Table 4, Qwen3-8B has MATH accuracy 56.3. However, from Qwen3's technical report (https://arxiv.org/abs/2505.09388) the accuracy for MATH-500 is 97.4 (with reasoning; Table 17) or 87.4(without reasoning; Table 18). While MATH-500 is a subset of MATH, I wonder why there is such discrepancy?\n2. In Table 10, what does \"chat format\" mean? Does it mean that the first three evals are selected based on model's perplexities on the choices and the last two are answered in QA?\n3. Table 7 is confusing. Why the upper half table has MATH-500 \"No Retrieval\" accuracy 83.2 but lower half table with MATH-500 \"No Retieval\" accuracy 91.0?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper release a datastore for retrieval.\n2. The paper includes a method to efficiently retrieve passages that reduce RAM needed.\n3. The experiments are conducted on different model sizes.", "weaknesses": "## Improvements on reasoning-intensive benchmarks\nI am not sure if the experiments fully support the claim, especially claiming that the proposed RAG system improves **reasoning extensive** tasks.\n1. MMLU and MMLU Pro are more knowledge related. These two benchmarks ask more about model's knowledge. I am not familiar with AGI Eval but from Table 10 it seems like it is MC and no CoTs are allowed when the model answers. While the model still needs to reason internally even without CoT, I think \"reasoning-intensive\" means those benchmarks where the model need to produce very long CoT (?) Usually those are math and coding benchmarks like AIME, AMC, HMMT, LiveCodeBench for reasoning models.\n2. The author includes MATH and GPQA where CoTs are allowed. While there are some improvement, how much improvement comes from the retrieval? For non-reasoning model like Llama-3 8B, perhaps a better baseline is to use a fixed, detailed prompt curated by a strong model like GPT-5 that teaches the model to plan, reflect, backtrack (and maybe some examples).\n3. There should be more experiments on reasoning models. The paper includes Qwen3-8B in Table 10 but I am confused about the result (see Questions section below). Reasoning models achieve non-trivial scores in benchmarks like AIME so I wonder how does reasoning model perform on those challenging benchmarks.\n\n## Other clarifications needed.\nPlease see questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GbsumNpMCY", "reviewer": "Reviewer_GFMa", "summary": "The authors claim that past works have shown little success in leveraging minimal RAG pipelines to improve performance on reasoning-intensive benchmarks because there are no practical, web-scale databases that capture the breadth of the pretraining distribution. At the same time, a RAG database must be compact to minimize decoding latency. The authors introduce CompactDS to meet these constraints. CompactDS is a curated mix of data that is initially drawn from a wide variety of sources and filtered to reduce redundancy and data contamination. In addition, the authors introduce a two stage retrieval pipeline: a course-grained lossy nearest neighbor search to narrow down the candidate set of passages followed by an exact inner product search conducted on-device to minimize retrieval latency.", "questions": "-\tWhen would you use RAG over hand-curating examples per topic? You could curate examples for each of these baselines that might boost performance further. Is the claim that RAG allows an agent to operate autonomously? Could you illustrate what a system like this would look like? The total dataset size is still hundreds of GB of RAM, and the course grained search probably can’t be done on a small edge device.\n-\tAre you concerned that filtering heavily for contamination directly conflicts with the goal of RAG (retrieving relevant passages).\n-\tWhat would MASSIVEDS performance look like (efficiency and accuracy) when using the two stage retrieval pipeline?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "-\tWell written and motivates why current RAG pipelines are insufficient due to lack of scope and high retrieval latency\n-\tDemonstrate improvement on a variety of benchmarks using RAG\n-\tConsiders problems in actual deployment (ie on device fine grained search after coarse-grained candidate search to minimize latency)\n-\tUpper-bounding the RAG performance with the oracle baseline makes sense and demonstrates how close this “autonomous” RAG system is to a curated ICL baseline", "weaknesses": "-\tThe paper introduces CompactDS as a means to improve reasoning performance using RAG but focuses mainly on multiple-choice question style evals that aren’t particularly reasoning intense. The MATH results are particularly interesting, but I would have liked to see the upper bound performance on this task (few shot prompting/ICL). My main concern is that the authors claim RAG helps with reasoning performance, but it seems hard to separate the boost from fact retrieval from the boost in reasoning performance, especially on MCQ style questions with non-reasoning models (it seems like in Table 4, retrieval seems to hurt GPQA performance across the board).\n-\tIs there a better way to gauge the accuracy of the two-stage retrieval pipeline beyond comparing downstream performance to the oracle? I am concerned because the oracle passages are chosen from the initial pool provided by the lossy retrieval.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cxvd4pse6e", "reviewer": "Reviewer_Uhfh", "summary": "This paper revisits the assumption that retrieval-augmented generation (RAG) is ineffective on reasoning-intensive benchmarks. The authors build a large and diverse datastore called COMPACTDS, combining academic, mathematical, and educational sources, and evaluate a simple retrieval-augmented generation setup without complex pipelines. To make it deployable on a single node, COMPACTDS integrates in-memory approximate nearest-neighbor (IVFPQ) retrieval with on-disk exact search to balance recall and latency. The proposed approach consistently improves accuracy across reasoning benchmarks and model scales, showing that even a minimal retrieval system can enhance reasoning performance when supported by a sufficiently rich and diverse datastore. COMPACTDS also matches or outperforms commercial search engines (e.g., Google Search) and complex agentic RAG systems such as Search-o1.", "questions": "- How well does COMPACTDS generalize to new reasoning datasets or more general tasks outside the benchmarks used in this paper?\n- How sensitive is the final performance to the strength and relative similarity between $E_{Approx}$ and $E_{Exact}$? Does the consistency of their retrieved results significantly influence performance?\n- Is there a quantitative analysis of retrieval quality (e.g., precision and recall) to verify that improvements stem from better retrieval?\n- Have the authors examined whether retrieval mainly aids factual grounding or intermediate reasoning steps?\n- Did the authors attempt a fine-tuned retriever trained directly on reasoning tasks, and how would it compare to COMPACTDS?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is clearly written and presents strong empirical evidence that retrieval remains useful for reasoning tasks when the datastore is comprehensive and high quality. The motivation is well grounded in an important and practical question. The retrieval design is simple yet effective, showing that careful data construction and indexing choices can yield significant improvements over prior baselines. Experiments are systematic, covering multiple model families and evaluation settings, and the results convincingly support the paper’s main claim. The analysis provides useful insight into how retrieval complements reasoning models. Releasing COMPACTDS as an open, reproducible resource would represent a meaningful contribution to the community.", "weaknesses": "The main limitation of the paper lies in its limited conceptual novelty. The contribution is primarily an engineering improvement through large-scale data construction rather than a new retrieval or reasoning algorithm. Although the results are strong, it remains unclear how well COMPACTDS generalizes beyond the specific reasoning benchmarks used in evaluation. Since the datastore composition is closely aligned with those benchmarks, the observed gains might partly reflect domain overlap rather than general reasoning improvement. It would be helpful to understand how the system performs on more general or non-reasoning tasks, and how flexible the datastore is when retrieval data need to be updated frequently, which is an important advantage of RAG systems in practical use.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UsioNhx6M6", "reviewer": "Reviewer_41Bx", "summary": "This paper introduces CompactDS for RAG in LLMs, a web-scale datastore built from filtered web crawl CommonCrawl plus specific math, academic, and educational texts to better cover content relevant to the evaluation tasks. To efficiently access the datastore, the paper proposes a two-stage retrieval mechanism: on-memory approximate nearest neighbor search using an build index followed by on-disk exact search. The aggressive content filtering enables single-node low-latency RAG use because the full datastore can fit <500GM RAM device. The paper evaluates a simple RAG pipeline on CompactDS on reasoning benchmarks like MATH and GPQA and language understanding (MMLU), and releases the datastore for public use.", "questions": "1. Line 68: Why does removing weakest contributing sources degrade performance? One would expect that the most similar documents would still be retrieved, and so the performance would be unaffected. Why is \"long tail diversity\" important (line 284)?\n2. Line 116: While this work does not focus on agentic RAG, do you think it can still work with RAG in an interactive setting? Say when the agent requests a tool call/search string but we retrieve non-exact document hits (e.g., in cases where the search string doesn't exist in the document corpora verbatim)?\n3. Line 192: What is the time and space overhead at training time, for indexing with IVFPQ and exact embeddings?\n4. Line 197: Need some clarity in understanding the 2-stage process. After q is encoded as E_exact(q), which K disk-stored embeddings are used? Are these the K passages retrieved in the 1st stage of IVFPQ? If that is so, wouldn't the lossy IVFPQ retrieval hurt the second stage? If that's not the case, then finding K passages with exact search in the full index would be extremely time-intensive.\n5. Table 1: CompactDS vs No Retrieval on Bio is worst (39.7 vs 47.4) but the relative gain is still positive? This seems incorrectly computed, or a typo.\n6. Line 250: good to point to Table 7 with the search-o1 results.\n7. Table 4: GPQA scores are consistently worse for CompactDS compared to NoRetrieval for all LLMs except Llama3.1 8B. Is there any insight into this? To me, arguing that CoT is extremely competent at 70B scale to improve NoRetrieval is not a great argument.\n8. Line 303 and Table 3: What is the oracle reranking? This needs explanation here.\n9. Line 410 and Table 6: If the PDFs cannot fit in context at all, how do we get evaluation results for the rows without LM Reranking?\n\n### Typos and editorial suggestions\n1. Line 171: spacing in argTopk...\n2. Table 1: it would make sense to only present No Retrieval, CompactDS rows, and any other baselines in this table. Then another table for ablation on single-source datastores.\n3. Line 248: demends -> demands?\n4. Line 256: point to the appendix with other LLM results.\n5. Fig 5: the font is extremely small even on my large monitor. I cannot read the math expressions at all. I can barely read the plots elsewhere, but this plot is really hard even after zooming in.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. The paper is generally well-written and clear.\n2. Open-sourcing and publicly releasing the datastore will be a plus for the research community when the industry datastores are proprietary and behind paywalls.\n3. MassiveDS and other non-filtered datastores can be impractical due to large memory footprint; a compact datastore like CompactDS will help the research community.\n4. The paper compares various approaches for retrieval-based generation, not just embedding-based RAG but also agentic systems based on classical search engines. This grounds the paper's contribution in several popular approaches.", "weaknesses": "1. While the paper explicitly describes the data sources in Sec 3.1, the justification for each filtering step is unclear. For example, in line 134... why take the union of two datasets then filter using a classifier with a certain threshold? Is the target to reduce to a maximum number of words in the corpora, or to maximize some validation split in the evaluation setup? This problem persists in the whole of Sec 3.1, which is a key contribution of the paper. Without this justification, design choices for data filtering seem arbitrary and do not help build intuition on what makes a datastore \"good\" or \"bad\".\n2. The summary in Sec 3.2 is not fully clear, but could improve significantly with a visual diagram spanning the top of this page. Specifically, it's unclear to me which K disk-stored embeddings are retrieved in the 2nd stage of the 2-stage process.\n3. It is difficult to be convinced that the 2-stage process is significantly better than the 1-stage (ANN only) process, comparing rows CompactDS-ANN only and CompactDS in Table 1. MMLU doesn't really improve and GPQA varies with different parts. There is improvement in MMLU Pro, AGI Eval, and MATH though. It would be useful if the authors discuss and interpret these results further.\n4. Since the paper argues that MassiveDS is too massive, which makes it impractical, it makes sense to evaluate it in Table 1 and report all benchmarks (not just MMLU in Table 2).\n5. Significant drops in performance on GPQA and MATH after decontamination (Sec A.1) suggest that CoT is more important for reasoning than retrieving passages. This counters the paper's argument that retrieval could be important for math reasoning tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)