Craw4LLM: Efficient Web Crawling for LLM Pretraining
Authors: Shi Yu, Zhiyuan Liu, Chenyan Xiong
Accepted at: acl25

CRAW4LLM: Efficient Web Crawling for LLM Pretraining

Shi Yu12*   Zhiyuan Liu1   Chenyan Xiong2
1Department of Computer Science and Technology, Tsinghua University
2Language Technologies Institute, Carnegie Mellon University
yus21@mails.tsinghua.edu.cn; liuzy@tsinghua.edu.cn; cx@cs.cmu.edu

Abstract

Web crawl is a main source of large language models’ (LLMs) pretraining data, but the majority of crawled web pages are discarded in pretraining due to low data quality. This paper presents CRAW4LLM, an efficient web crawling method that explores the web graph based on the preference of LLM pretraining. Specifically, it leverages the influence of a webpage in LLM pretraining as the priority score of the web crawler’s scheduler, replacing the standard graph-connectivity-based priority. Our experiments on a web graph containing 900 million webpages from a commercial search engine’s index demonstrate the efficiency of CRAW4LLM in obtaining high-quality pretraining data. With just 21% URLs crawled, LLMs pretrained on CRAW4LLM data reach the same downstream performances of previous crawls, significantly reducing the crawling waste and alleviating the burdens on websites. Our code is publicly available at https://github.com/cxcscmu/Craw4LLM.

1 Introduction

Massive in size and diverse in topics, web data usually serve as the primary source of pretraining data for large language models (LLMs), providing an extensive and heterogeneous corpus that captures a wide spectrum of human knowledge and real-world information (Baack, 2024; Dubey et al., 2024; Penedo et al., 2024). Pretraining datasets are typically built from large-scale web crawls such as Common Crawl (CommonCrawl, 2007), which may contain TBs of data spanning billions of webpages (Penedo et al., 2024; Weber et al., 2024).

Despite their vast scale, most of the data collected from web crawls are not used in the pretraining of LLMs. Existing work often discards over 90% of the raw data collected from the web (Li et al., 2024; Penedo et al., 2024; Tang et al., 2024), highlighting the inefficiency of current web crawlers in collecting LLM pretraining data. Common web crawlers like Common Crawl prioritize pages based on graph connectivity metrics like PageRank (Page et al., 1999; Cho et al., 1998) or harmonic centrality (Boldi and Vigna, 2014; Baack, 2024), which favor documents with a high number of inlinks (indegree) (Fortunato et al., 2008) rather than those most relevant for pretraining. This misalignment not only leads to waste in computational resources during excessive data processing for LLM developers, but also incentivizes over-crawling, which burdens website operators with redundant traffic and increases ethical and legal risks related to fair use of data and copyright (Longpre et al., 2024; New York Times, 2023).

To bridge this gap, we propose Web Crawling for LLM Pretraining (CRAW4LLM). Instead of relying on traditional graph-connectivity-based signals, CRAW4LLM improves crawling efficiency by prioritizing webpages based on their influence on LLM pretraining. Specifically, during each crawling iteration, all newly discovered documents are scored with a pretraining influence scorer derived from data-filtering pipelines for pretraining (Li et al., 2024; Penedo et al., 2024), and docu-

![Graph traverse process of a traditional graph-connectivity-based crawler (green) and CRAW4LLM (red) starting from a same seed URL (star).](page_1016_563_377_246.png)

Figure 1: Graph traverse process of a traditional graph-connectivity-based crawler (green) and CRAW4LLM (red) starting from a same seed URL (star).
Figure 2: Correlations between pretraining influence scores from DCLM fastText (Li et al., 2024) and PageRank to indegrees, on randomly sampled ClueWeb22-B documents (Overwijk et al., 2022). Spearman correlation coefficients are reported in parentheses.

ments with the highest scores are used to discover new documents. By prioritizing webpages with high influence scores, as illustrated in Figure 1, CRAW4LLM explores the web graph in a fundamentally different manner from traditional graph-connectivity-based crawlers, uncovering a distinct subset of the web more useful for pretraining.

We conduct large-scale crawling simulations on ClueWeb22-A (Overwijk et al., 2022), a snapshot of the web containing 900 million English webpages obtained from the central index of a commercial search engine. Results show that, by crawling only 1× of the pretraining dataset size, CRAW4LLM can outperform traditional crawlers which collect 1x, 2x, and 4x data followed by data selection. Compared to the baseline crawler that achieves the same performance, CRAW4LLM crawls only 21% of the webpages. Further analysis reveals that during crawling, CRAW4LLM quickly discovers documents that align with the oracle selection, which selects from the full web graph. As a result, it achieves over 95% of the oracle performance while crawling only 2.2% of the data.

2 Methodology

In this section, we introduce Web Data Crawling for LLM Pretraining (CRAW4LLM), an efficient crawling method that integrates LLM pretraining preference into the crawler. The algorithm of CRAW4LLM is presented in Algorithm 1.

Similar to traditional crawlers (Cho et al., 1998), CRAW4LLM starts with a set of seed URLs. For each unvisited outlink of them, CRAW4LLM assigns a score using a pretraining-oriented URL scoring function SCORE_URL(; \( \mathcal{M} \)), where \( \mathcal{M} \) is a pretraining influence scorer which rates a document’s influence for pretraining. \( \mathcal{M} \) can be derived from data classification models for pretraining data, which have been used to decide whether a document should be retained in or filtered out from the raw dataset (Li et al., 2024; Penedo et al., 2024). Formally, given a pretraining influence scorer \( \mathcal{M} \), the score s of a URL u is calculated as

\[
s \leftarrow \text{SCORE\_URL}(u; \mathcal{M}) = \mathcal{M}(\text{FETCHPAGE}(u)),
\]

where FETCHPAGE(u) gets the page content of u and \( \mathcal{M}(\cdot) \) returns the score. Once all outlinks have been scored, following the standard procedures of existing crawlers, they are inserted into a priority queue, which automatically orders them based on their scores. The top n highest-scoring URLs are then dequeued for pretraining and serve as the sources for the next round of crawling. This process repeats until N documents have been collected, forming the final pretraining dataset \( \mathcal{P} \).

In contrast, traditional crawlers typically rely on graph connectivity metrics, such as PageRank (Cho et al., 1998) and harmonic centrality (Baack, 2024), which basically assign higher priority to pages with higher indegrees (Fortunato et al., 2008). As shown in Figure 2(a), the indegrees of webpages exhibit a poor correlation with the scores assigned by the DCLM fastText classifier, a pretraining influence scorer for identifying high-quality pretraining data (Li et al., 2024). This confirms that graph connectivity-based crawlers are inefficient in crawling pretraining data.

By incorporating a pretraining influence scorer, CRAW4LLM traverses the web graph in a way

Algorithm 1 CRAW4LLM Algorithm

Input: Seed URLs \( \mathcal{U}_{seed} \), number of pages to be crawled \( N \), number of pages to be crawled in each iteration \( n \), pretraining influence scorer \( \mathcal{M}(\cdot) \)

Output: Crawled page set \( \mathcal{P} \)

1: Initialize URL and score priority queue \( \mathcal{Q} \leftarrow \emptyset \)
2: Initialize crawled page set \( \mathcal{P} \leftarrow \emptyset \)
3: Initialize visited URL set \( \mathcal{V} \leftarrow \mathcal{U}_{seed} \)
4: \( \mathcal{U}_c \leftarrow \mathcal{U}_{seed} \)
5: while \( |\mathcal{P}| \leq N \) do
6:   \( \mathcal{P}_c \leftarrow \text{FETCHPAGES}(\mathcal{U}_c) \)
7:   Merge \( \mathcal{P}_c \) into \( \mathcal{P} \)
8:   \( \mathcal{U}_{out} \leftarrow \text{EXTRACTURLS}(\mathcal{P}_c) \)
9:   for all \( v \in \mathcal{U}_{out} \) do
10:     if \( v \notin \mathcal{V} \) then
11:       ENQUEUE(\( \mathcal{Q}, v, \text{SCORE\_URL}(v; \mathcal{M}) \))
12:       ADD(\( \mathcal{V}, v \))
13:     end if
14:   end for
15:   \( \mathcal{U}_c \leftarrow \text{DEQUEUE}(\mathcal{Q}, n) \)
16: end while
17: return \( \mathcal{P} \)
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Selection Pool Size</th>
    <th colspan="3">Commonsense Reasoning (4 tasks)</th>
    <th colspan="3">Language Understanding (6 tasks)</th>
    <th colspan="3">Reading Comprehension (3 tasks)</th>
    <th colspan="3">Symbolic Problem Solving (5 tasks)</th>
    <th colspan="3">World Knowledge (5 tasks)</th>
    <th colspan="3">Core (23 tasks)</th>
    <th rowspan="2">% of Oracle</th>
  </tr>
  <tr>
    <th>Selection</th>
    <th>Reasoning</th>
    <th>Understanding</th>
    <th>Comprehension</th>
    <th>Problem Solving</th>
    <th>Knowledge</th>
    <th>Core</th>
  </tr>
  <tr>
    <td colspan="21">Using the DCLM fastText (Li et al., 2024) classifier as the pretraining influence scorer</td>
  </tr>
  <tr>
    <td>Oracle</td>
    <td>4x</td>
    <td>0.2438</td>
    <td>0.2209</td>
    <td>0.1483</td>
    <td>0.2039</td>
    <td>0.2403</td>
    <td>0.2239</td>
    <td>100%</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>1x</td>
    <td>0.1906</td>
    <td>0.1890</td>
    <td>0.0244</td>
    <td>0.1834</td>
    <td>0.1930</td>
    <td>0.1748</td>
    <td>78.1%</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>2x</td>
    <td>0.1896</td>
    <td>0.1967</td>
    <td>0.1260</td>
    <td>0.2000</td>
    <td>0.2024</td>
    <td>0.1964</td>
    <td>87.7%</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>1x</td>
    <td>0.1730</td>
    <td>0.1680</td>
    <td>0.0326</td>
    <td>0.1616</td>
    <td>0.1668</td>
    <td>0.1556</td>
    <td>69.5%</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>2x</td>
    <td>0.1845</td>
    <td>0.1856</td>
    <td>0.0970</td>
    <td>0.1958</td>
    <td>0.1953</td>
    <td>0.1865</td>
    <td>83.3%</td>
  </tr>
  <tr>
    <td><b>CRAW4LLM</b></td>
    <td>1x</td>
    <td><b>0.2116</b></td>
    <td><b>0.2311</b></td>
    <td><b>0.0826</b></td>
    <td><b>0.1979</b></td>
    <td><b>0.2486</b></td>
    <td><b>0.2133</b></td>
    <td><b>95.3%</b></td>
  </tr>
  <tr>
    <td colspan="21">Using the FineWeb-Edu (Penedo et al., 2024) classifier as the pretraining influence scorer</td>
  </tr>
  <tr>
    <td>Oracle</td>
    <td>4x</td>
    <td>0.1899</td>
    <td>0.1973</td>
    <td>0.1081</td>
    <td>0.2117</td>
    <td>0.2786</td>
    <td>0.2133</td>
    <td>100%</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>1x</td>
    <td>0.1906</td>
    <td>0.1890</td>
    <td>0.0244</td>
    <td>0.1834</td>
    <td>0.1930</td>
    <td>0.1748</td>
    <td>82.0%</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>2x</td>
    <td>0.1797</td>
    <td>0.1888</td>
    <td>0.0931</td>
    <td>0.1586</td>
    <td>0.2100</td>
    <td>0.1807</td>
    <td>84.7%</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>1x</td>
    <td>0.1730</td>
    <td>0.1680</td>
    <td>0.0326</td>
    <td>0.1616</td>
    <td>0.1668</td>
    <td>0.1556</td>
    <td>72.9%</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>2x</td>
    <td>0.1720</td>
    <td>0.1709</td>
    <td>0.0840</td>
    <td>0.1783</td>
    <td>0.1842</td>
    <td>0.1724</td>
    <td>80.8%</td>
  </tr>
  <tr>
    <td><b>CRAW4LLM</b></td>
    <td>1x</td>
    <td><b>0.2122</b></td>
    <td><b>0.1867</b></td>
    <td><b>0.0368</b></td>
    <td><b>0.2055</b></td>
    <td><b>0.2837</b></td>
    <td><b>0.2043</b></td>
    <td><b>95.8%</b></td>
  </tr>
</table>

Table 1: Downstream LLM performance. Either the DCLM fastText classifier (Li et al., 2024) (top) or FineWeb-Edu classifier (Penedo et al., 2024) (down) is used as the pretraining influence scorer for CRAW4LLM, and to select documents for oracle and crawl-then-select runs (Section 3). All models are pretrained on 1x data (20M documents, 32.9B tokens). The evaluation metric is centered accuracy (0 = random guess) (Li et al., 2024). Best/2nd best in the last two groups are bolded/underlined. See Appendix D for detailed results.

that prioritizes high-quality pretraining documents. This makes the crawling more efficient and enables the discovery of documents dramatically different with connectivity-based crawlers.

3 Experimental Methodology

In this section, we introduce our experimental setup, with details on the crawler implementation and LLM training provided in Appendix A and B.

CRAW4LLM. To run experiments in our limited computational budget, we run a simulation of CRAW4LLM on the ClueWeb22 dataset (Overwijk et al., 2022), a snapshot of the web with graph information from a commercial crawler. We use the English subset of ClueWeb22-A, which is a web graph containing 900M webpages with links. We randomly sampled 10K URLs as our seed URLs. We set the number of total crawled documents \( N \) to 20M and crawled documents each iteration \( n \) to 10K. In our experiments, we consider using the DCLM fastText classifier (Li et al., 2024) or the FineWeb-Edu classifier (Penedo et al., 2024) as the pretraining influence scorer \( \mathcal{M}(\cdot) \), whose details can be found in Appendix C.

Baselines. We emulate traditional graph-connectivity-based crawlers by replacing the LLM-oriented URL scoring function (Eq. 1) with a function that returns the indegree for a given URL, since a node’s indegree closely correlates with PageRank, a common graph connectivity metric, as shown in Figure 2(b) and previous findings (Fortunato et al., 2008). We also introduce a random crawling baseline, where the scorer assigns random scores. We run both of them in a crawl-then-select setting, first crawling 1x or 2x documents and then selecting the top 1x (20M) documents based on scores assigned by the DCLM fastText classifier or the FineWeb-Edu classifier. This process mimics existing data-filtering pipelines, which begin with crawled documents and then apply filtering (Li et al., 2024; Penedo et al., 2024).

Oracle. We introduce an oracle selection run in which we directly apply the pretraining influence scorer to the ClueWeb22-A dataset and randomly sample 20M documents which are scored as the 10% for pretraining, serving as the upper bound.

LLM Training and Evaluation. For all runs, we use the final set of 20M crawled or selected documents to pretrain a 411M Transformer on 4x Chinchilla-optimal tokens (Hoffmann et al., 2022), totaling 32.9B tokens. The pretraining is conducted using the DCLM codebase (Li et al., 2024). To evaluate the pretrained LLMs, we follow the DCLM evaluation recipe, assessing performance on 23 (22 unique) core tasks.

4 Evaluation Results

In this section, we first present the overall performance of CRAW4LLM (Section 4.1), followed by further analysis (Section 4.2).
Figure 3: Efficiency of crawlers. (a) shows the performance of LLMs trained on selected data crawled by CRAW4LLM and extended baseline crawlers. (b) presents the number of crawled (\( P \)) and visited (\( V \)) documents for CRAW4LLM, along with the estimated number of crawled documents required for indegree-based crawler to match CRAW4LLM’s performance.

4.1 Overall Performance

In this experiment, we compare the performance of CRAW4LLM with baseline crawlers by evaluating LLMs trained on their respective crawled data. As shown in Table 1, when all methods crawl the same amount of training data (1×), CRAW4LLM significantly outperforms random crawling and indegree crawling using either DCLM fastText or FineWeb-Edu classifier as the pretraining influence scorer. In the crawl-then-select setting, where traditional crawlers are allowed to collect twice as much data (2×) for later selection, they still underperform compared to CRAW4LLM. This suggests that incorporating pretraining-oriented signals early in the crawling process is more beneficial than relying on post-selection. With only 1× of the data, CRAW4LLM retains 95.3% and 95.6% of the performance achieved by the DCLM fastText and FineWeb-Edu oracle run, respectively, which directly selects from the entire ClueWeb22-A dataset, a substantially larger 45× data pool.

4.2 Analysis

In this subsection, we further analyze the efficiency of CRAW4LLM compared to traditional crawlers and explore the reasons behind it. We utilize the DCLM fastText classifier in the experiments presented in this subsection.

Crawling Efficiency. We evaluate the efficiency of CRAW4LLM by comparing the number of documents it crawls or visits against baseline crawlers. As shown in Figure 3(a), even when the baselines crawl 4× the required pretraining data for selection, they still underperform compared to CRAW4LLM.

Figure 4: Precision (left) and recall (right) of the oracle documents among the documents crawled by CRAW4LLM, indegree, and random crawler. The upper bound represents always crawling the oracle documents.

Extrapolation suggests that the indegree-based crawler would need to crawl 4.8× documents (96M) to match CRAW4LLM’s performance. Figure 3(b) further illustrates that CRAW4LLM achieves the same performance while crawling only 21% of the documents required by the indegree-based crawler, or 48% when considering all visited documents. These results highlight the efficiency of CRAW4LLM, demonstrating its potential to reduce website burdens and mitigate over-crawling.

Document Coverage. In this experiment, we plot the precision and recall of the oracle-selected documents among those crawled by CRAW4LLM and baseline crawlers throughout the crawling process. As shown in Figure 4, the precision quickly reaches 1.0, while the recall increases linearly, aligning with the theoretical upper bound. The saturated performance remains until 13 million documents have been crawled, after which the performance starts to decline, likely due to the lack of connectivity of the ClueWeb22 subgraph. In contrast, baseline crawlers exhibit minimal overlap with oracle-selected data, verifying that most of their crawled

Figure 5: Correlations between the pretraining influence scores of the documents themselves and the average scores of their 1- and 2-hop outlink documents. Spearman correlation coefficients are reported in parentheses.
content is misaligned with pretraining needs and should be filtered (Li et al., 2024; Penedo et al., 2024). These results emphasize the importance of targeted crawling strategies for pretraining.

Score Correlations Across Links. CRAW4LLM tracks the outlinks of the highest-scored documents in the current iteration to enrich the queue for future crawls. As shown in Figure 5, we plot the correlations between the pretraining influence scores of current documents and their 1- and 2-hop outlinks. The results indicate a correlation in influence scores across link hops, suggesting that highly-rated documents are interconnected and can be discovered through previously crawled documents.

5 Conclusion

This paper presents CRAW4LLM, a step toward more efficient and responsible web crawling for LLM pretraining. By prioritizing documents based on the pretraining needs, our method improves crawling efficiency and reduces unnecessary crawling, easing the burden on web hosts. While fair use of web data remains a critical challenge, we hope that CRAW4LLM can help mitigate these concerns and promote more compliant and sustainable practices in obtaining pretraining data for LLMs.

Limitations

Web crawling raises important concerns regarding copyright and the fair use of web data (Longpre et al., 2024), necessitating a better solution from the entire LLM community, such as sharing benefits with website owners. In this paper, we propose a more efficient crawling method that mitigates these challenges by reducing crawling, though it does not fully resolve them. Our experiments are conducted on a web graph dataset ClueWeb22 (Overwijk et al., 2022), thereby avoiding issues associated with actual web crawling. We hope that future advancements in web crawling will better align with ethical and legal standards.

While our crawling simulation is a sufficient research setup, further validation is required to assess the effectiveness of CRAW4LLM in real-world crawling scenarios. Our CRAW4LLM and baseline crawlers implement only the selection policy (Cho et al., 1998) of a crawler, which determines which pages to crawl. Although we try to mimic real-world crawling procedures used in systems like Apache Nutch1, we do not implement other web crawling policies in industrial-level crawlers, such as the re-visit policy (Cho and Garcia-Molina, 2003a), politeness policy (Cho and Garcia-Molina, 2003b), and parallelization policy (Cho and Garcia-Molina, 2002). We leave the integration of CRAW4LLM into real-world crawling engines like Nutch and a comprehensive comparison between CRAW4LLM and traditional crawling methods in real-world crawling scenarios for future work.

References

Stefan Baack. 2024. A critical analysis of the largest source for generative AI training data: Common crawl. In FAccT, pages 2199–2208. ACM.

Paolo Boldi and Sebastiano Vigna. 2014. Axioms for centrality. Internet Math., 10(3-4):222–262.

Junghoo Cho and Hector Garcia-Molina. 2002. Parallel crawlers. In WWW 2002.

Junghoo Cho and Hector Garcia-Molina. 2003a. Effective page refresh policies for web crawlers. ACM Trans. Database Syst., 28(4):390–426.

Junghoo Cho and Hector Garcia-Molina. 2003b. Estimating frequency of change. ACM Trans. Internet Techn., 3(3):256–290.

Junghoo Cho, Hector Garcia-Molina, and Lawrence Page. 1998. Efficient crawling through URL ordering. Comput. Networks, 30(1-7):161–172.

CommonCrawl. 2007. Common crawl.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Santo Fortunato, Marián Boguñá, Alessandro Flammini, and Filippo Menczer. 2008. Approximating pagerank from in-degree. In WAW 2006.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. In NeurIPS 2022.

Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In EACL 2017.

Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora,

1https://nutch.apache.org/
Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee F. Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah M. Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Raghavi Chandu, Thao Nguyen, Igor Vasiljevic, Sham M. Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghi, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alex Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishal Shankar. 2024. Datacomp-Im: In search of the next generation of training sets for language models. In NeurIPS 2024.

Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, Kevin Klyman, Christopher Klamm, Hailey Schoelkopf, Nikhil Singh, Manuel Cherep, Ahmad Anis, An Dinh, Caroline Shamiso Chitongo, Da Yin, Damien Sileo, Deividas Matacinas, Diganta Misra, Emad A. Alghamdi, Enrico Shipple, Jianguo Zhang, Joanna Materzynska, Kun Qian, Kushagra Tiwary, Lester James V. Miranda, Manan Dey, Minnie Liang, Mohammed Hamdy, Niklas Muennighoff, Seonghyeon Ye, Seungwon Kim, Shrestha Mohanty, Vipul Gupta, Vivek Sharma, Minh Chien Vu, Xuhui Zhou, Yizhi Li, Caiming Xiong, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito, Sara Hooker, Jad Kabbara, and Alex Pentland. 2024. Consent in crisis: The rapid decline of the AI data commons. In NeurIPS 2024.

New York Times. 2023. Complaint, the new york times company v. microsoft corporation, openai, inc., openai lp, openai gp, llc, openai llc, openai opc o llc, openai global llc, oai corporation, llc, and openai holdings, llc. Case 1:23-cv-11195, United States District Court, Southern District of New York.

Arnold Overwijk, Chenyan Xiong, Xiao Liu, Cameron VandenBerg, and Jamie Callan. 2022. Clueweb22: 10 billion web documents with visual and semantic information. arXiv preprint arXiv:2211.15848.

Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation ranking: Bringing order to the web. Technical report, Stanford infolab.

Guilherme Penedo, Hynek Kydlícek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin A. Raffel, Leandro von Werra, and Thomas Wolf. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. In NeurIPS 2024.

Liping Tang, Nikhil Ranjan, Omkar Pangarkar, Xuezhi Liang, Zhen Wang, Li An, Bhaskar Rao, Linghao Jin, Huijuan Wang, Zhoujun Cheng, Suqi Sun, Cun Mu, Victor Miller, Xuezhe Ma, Yue Peng, Zhengzhong Liu, and Eric P. Xing. 2024. Txt360: A top-quality llm pre-training dataset requires the perfect blend.

Teknium. 2023. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants.

Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozehe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. 2024. Red-pajama: an open dataset for training large language models. In NeurIPS 2024.

A Details on Crawling

Our implementation of the indegree-based crawler employs a static URL scoring function, which directly returns the indegree of a given URL based on the full ClueWeb22 graph (Sec. 3). For real-world crawlers, as the true indegree value of a URL cannot be known in advance, a local graph must be maintained to track the known inlinks of discovered URLs. This local graph is updated iteratively as the discovered portion of the web expands during the crawling process (Cho et al., 1998).

Maintaining such a local graph during crawling introduces significant computational overhead. For simplicity, we instead implement the static simulation, where we directly return the global indegree of each URL. We believe that this simplified implementation does not underperform compared to real-world implementations, as our approach leverages global information from the entire graph, which should be better than the partial information from the local graph.

We run our simulated crawlers on a Linux server equipped with two Intel(R) Xeon(R) E5-2630 v3 CPUs (8 cores per socket, 16 cores in total, 1 thread per core), 125GiB of memory, and an SSD. The crawling process of CRAW4LLM takes about 21 hours to finish. In comparison, the random and indegree-based crawlers take around 10.5 and 12.5 hours, respectively. Note that since these are simulated crawls on the snapshot, the reported times do not reflect real-world crawling performance.

B Details on LLM Training and Evaluation

We pretrain a 411M-parameter\(^2\) decoder-only Transformer model using the DCLM training

\(^2\)Sometimes referred to as 400M in the DCLM paper (Li et al., 2024).
<table>
  <tr>
    <th>Hyper-parameter</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>\( n_{\text{layers}} \)</td>
    <td>24</td>
  </tr>
  <tr>
    <td>\( n_{\text{heads}} \)</td>
    <td>8</td>
  </tr>
  <tr>
    <td>\( d_{\text{model}} \)</td>
    <td>1,024</td>
  </tr>
  <tr>
    <td>\( d_{\text{head}} \)</td>
    <td>128</td>
  </tr>
  <tr>
    <td>Warmup</td>
    <td>2,000</td>
  </tr>
  <tr>
    <td>Learning Rate</td>
    <td>3e-3</td>
  </tr>
  <tr>
    <td>Weight Decay</td>
    <td>0.033</td>
  </tr>
  <tr>
    <td>z-loss</td>
    <td>1e-4</td>
  </tr>
  <tr>
    <td>Global Batch Size</td>
    <td>512</td>
  </tr>
  <tr>
    <td>Sequence Length</td>
    <td>2048</td>
  </tr>
</table>

Table 2: Model and training hyper-parameters. \( n_{\text{layers}} \), \( n_{\text{layers}} \), \( d_{\text{model}} \), and \( d_{\text{head}} \) denote the number of layers, attention heads, width, and width per attention head, respectively.

recipe (Li et al., 2024)\footnote{https://github.com/mlfoundations/dclm}. The hyper-parameters are presented in Table 2. To enhance training stability, we extend the original 411M-1x setting to 411M-4x, meaning the model is trained on 4 times the Chinchilla-optimal number of tokens (Hoffmann et al., 2022), which amounts to 32.9B tokens. The training process takes 1 day and 12 hours on 8 NVIDIA L40S GPUs. For further details, please refer to the DCLM paper (Li et al., 2024). Due to computational constraints, each pretraining experiment is conducted only once.

We use the DCLM evaluation recipe (Li et al., 2024) to evaluate model performance on 23 (22 unique) *core* tasks.

C Details on the Pretraining Influence Scorers

DCLM fastText Classifier (Li et al., 2024) is a quality filter based on fastText (Joulin et al., 2017), trained to distinguish between high- and low-quality pretraining data. A dataset size of 400K examples (200K positive, 200K negative) is used for training. Negative samples are random documents from an earlier version of the RefinedWeb reproduction from the DCLM team. Positive samples come from OpenHermes 2.5 (Teknium, 2023) and high-scoring posts from the r/ExplainLikeImFive (ELI5) subreddit. The final DCLM-baseline dataset is created by selecting only the top 10% of documents ranked by the classifier.

FineWeb-Edu Classifier (Penedo et al., 2024) is a model designed to identify and filter educational content within FineWeb. Based on the Snowflake-arctic-embed-m\footnote{https://huggingface.co/Snowflake/snowflake-arctic-embed-m} text embedding model, it is trained on 450,000 annotations generated by Llama-3-70B-Instruct, which assigned an educational quality score from 0 to 5 to every document. FineWeb-Edu is built by filtering out samples from FineWeb with scores lower than 3 with the trained classifier.

D Detailed Results

The raw (uncentered) accuracy of all evaluation tasks is presented in Table 3, 4, 5, 6, and 7. Please refer to Li et al. (2024) for more details on the evaluation tasks.

E The ClueWeb22 Dataset

ClueWeb22 (Overwijk et al., 2022) is distributed under a “TREC-style” license for research purpose. The dataset can be obtained by signing a data license agreement with Carnegie Mellon University\footnote{https://lemurproject.org/clueweb22/obtain.php}. We use ClueWeb22 only for research purpose.

F Use of AI Assistants

We use GitHub Copilot\footnote{https://github.com/features/copilot} to assist with coding and ChatGPT\footnote{https://chatgpt.com/} (powered by GPT-4o) to enhance the writing of this paper.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Pretraining Influence Scorer</th>
    <th rowspan="2">Selection Pool Size</th>
    <th colspan="5">Commonsense Reasoning</th>
  </tr>
  <tr>
    <th>CommonsenseQA</th>
    <th>COPA</th>
    <th>OpenBookQA</th>
    <th>PIQA</th>
  </tr>
  <tr>
    <td>Oracle</td>
    <td>DCLM fastText</td>
    <td>45×</td>
    <td>0.2850</td>
    <td>0.7000</td>
    <td>0.3300</td>
    <td>0.6812</td>
  </tr>
  <tr>
    <td>Oracle</td>
    <td>FineWeb-Edu</td>
    <td>45×</td>
    <td>0.2342</td>
    <td>0.6200</td>
    <td>0.3620</td>
    <td>0.6638</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>–</td>
    <td>1×</td>
    <td>0.2072</td>
    <td>0.6700</td>
    <td>0.2980</td>
    <td>0.6746</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>DCLM fastText</td>
    <td>2×</td>
    <td>0.2588</td>
    <td>0.6200</td>
    <td>0.3160</td>
    <td>0.6785</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>FineWeb-Edu</td>
    <td>2×</td>
    <td>0.2301</td>
    <td>0.6200</td>
    <td>0.3140</td>
    <td>0.6779</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>DCLM fastText</td>
    <td>4×</td>
    <td>0.2326</td>
    <td>0.6400</td>
    <td>0.3380</td>
    <td>0.6757</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>–</td>
    <td>1×</td>
    <td>0.3219</td>
    <td>0.6000</td>
    <td>0.2780</td>
    <td>0.6513</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>DCLM fastText</td>
    <td>2×</td>
    <td>0.1966</td>
    <td>0.6600</td>
    <td>0.3040</td>
    <td>0.6752</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>FineWeb-Edu</td>
    <td>2×</td>
    <td>0.1974</td>
    <td>0.6400</td>
    <td>0.3160</td>
    <td>0.6616</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>DCLM fastText</td>
    <td>4×</td>
    <td>0.2088</td>
    <td>0.6400</td>
    <td>0.3400</td>
    <td>0.6817</td>
  </tr>
  <tr>
    <td><b>CRAW4LLM</b></td>
    <td>DCLM fastText</td>
    <td>1×</td>
    <td>0.2277</td>
    <td>0.6600</td>
    <td>0.3300</td>
    <td>0.6926</td>
  </tr>
  <tr>
    <td><b>CRAW4LLM</b></td>
    <td>FineWeb-Edu</td>
    <td>1×</td>
    <td>0.3219</td>
    <td>0.6200</td>
    <td>0.3500</td>
    <td>0.6616</td>
  </tr>
</table>

Table 3: Results for commonsense reasoning tasks.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Pretraining Influence Scorer</th>
    <th rowspan="2">Selection Pool Size</th>
    <th colspan="5">Language Understanding</th>
  </tr>
  <tr>
    <th>BIG-Bench Lang. Id.</th>
    <th>HellaSwag (zero-shot)</th>
    <th>HellaSwag</th>
    <th>LAMBADA</th>
    <th>Winograd</th>
    <th>Winogrande</th>
  </tr>
  <tr>
    <td>Oracle</td>
    <td>DCLM fastText</td>
    <td>45×</td>
    <td>0.2515</td>
    <td>0.3856</td>
    <td>0.3905</td>
    <td>0.4432</td>
    <td>0.6557</td>
    <td>0.5130</td>
  </tr>
  <tr>
    <td>Oracle</td>
    <td>FineWeb-Edu</td>
    <td>45×</td>
    <td>0.2580</td>
    <td>0.3831</td>
    <td>0.3818</td>
    <td>0.3643</td>
    <td>0.6227</td>
    <td>0.5185</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>–</td>
    <td>1×</td>
    <td>0.2490</td>
    <td>0.3709</td>
    <td>0.3716</td>
    <td>0.3990</td>
    <td>0.6044</td>
    <td>0.5146</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>DCLM fastText</td>
    <td>2×</td>
    <td>0.2468</td>
    <td>0.3882</td>
    <td>0.3925</td>
    <td>0.4073</td>
    <td>0.6007</td>
    <td>0.5130</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>FineWeb-Edu</td>
    <td>2×</td>
    <td>0.2485</td>
    <td>0.3815</td>
    <td>0.3800</td>
    <td>0.3804</td>
    <td>0.6007</td>
    <td>0.5146</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>DCLM fastText</td>
    <td>4×</td>
    <td>0.2521</td>
    <td>0.4011</td>
    <td>0.4019</td>
    <td>0.4390</td>
    <td>0.6154</td>
    <td>0.5130</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>–</td>
    <td>1×</td>
    <td>0.2566</td>
    <td>0.3515</td>
    <td>0.3519</td>
    <td>0.3596</td>
    <td>0.5971</td>
    <td>0.5004</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>DCLM fastText</td>
    <td>2×</td>
    <td>0.2547</td>
    <td>0.3749</td>
    <td>0.3771</td>
    <td>0.3773</td>
    <td>0.5861</td>
    <td>0.5241</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>FineWeb-Edu</td>
    <td>2×</td>
    <td>0.2528</td>
    <td>0.3636</td>
    <td>0.3658</td>
    <td>0.3672</td>
    <td>0.5678</td>
    <td>0.5193</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>DCLM fastText</td>
    <td>4×</td>
    <td>0.2562</td>
    <td>0.3994</td>
    <td>0.4008</td>
    <td>0.4159</td>
    <td>0.6190</td>
    <td>0.5178</td>
  </tr>
  <tr>
    <td><b>CRAW4LLM</b></td>
    <td>DCLM fastText</td>
    <td>1×</td>
    <td>0.2544</td>
    <td>0.4035</td>
    <td>0.4048</td>
    <td>0.4196</td>
    <td>0.6593</td>
    <td>0.5288</td>
  </tr>
  <tr>
    <td><b>CRAW4LLM</b></td>
    <td>FineWeb-Edu</td>
    <td>1×</td>
    <td>0.2521</td>
    <td>0.3726</td>
    <td>0.3717</td>
    <td>0.3478</td>
    <td>0.6264</td>
    <td>0.5083</td>
  </tr>
</table>

Table 4: Results for language understanding tasks.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">Pretraining Influence Scorer</th>
    <th rowspan="2">Selection Pool Size</th>
    <th colspan="3">Reading Comprehension</th>
  </tr>
  <tr>
    <th>BoolQ</th>
    <th>CoQA</th>
    <th>SQuAD</th>
  </tr>
  <tr>
    <td>Oracle</td>
    <td>DCLM fastText</td>
    <td>45×</td>
    <td>0.5755</td>
    <td>0.2479</td>
    <td>0.3139</td>
  </tr>
  <tr>
    <td>Oracle</td>
    <td>FineWeb-Edu</td>
    <td>45×</td>
    <td>0.5367</td>
    <td>0.2305</td>
    <td>0.3130</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>–</td>
    <td>1×</td>
    <td>0.5080</td>
    <td>0.1799</td>
    <td>0.1882</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>DCLM fastText</td>
    <td>2×</td>
    <td>0.5807</td>
    <td>0.2053</td>
    <td>0.2759</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>FineWeb-Edu</td>
    <td>2×</td>
    <td>0.5627</td>
    <td>0.1997</td>
    <td>0.2304</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>DCLM fastText</td>
    <td>4×</td>
    <td>0.5911</td>
    <td>0.2361</td>
    <td>0.2951</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>–</td>
    <td>1×</td>
    <td>0.5324</td>
    <td>0.1666</td>
    <td>0.1616</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>DCLM fastText</td>
    <td>2×</td>
    <td>0.5697</td>
    <td>0.1843</td>
    <td>0.2390</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>FineWeb-Edu</td>
    <td>2×</td>
    <td>0.5670</td>
    <td>0.1904</td>
    <td>0.2011</td>
  </tr>
  <tr>
    <td>Indegree</td>
    <td>DCLM fastText</td>
    <td>4×</td>
    <td>0.5765</td>
    <td>0.2147</td>
    <td>0.2736</td>
  </tr>
  <tr>
    <td><b>CRAW4LLM</b></td>
    <td>DCLM fastText</td>
    <td>1×</td>
    <td>0.5440</td>
    <td>0.2264</td>
    <td>0.2215</td>
  </tr>
  <tr>
    <td><b>CRAW4LLM</b></td>
    <td>FineWeb-Edu</td>
    <td>1×</td>
    <td>0.4654</td>
    <td>0.2146</td>
    <td>0.3026</td>
  </tr>
</table>

Table 5: Results for reading comprehension tasks.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Pretraining Influence Scorer</th>
    <th colspan="2">Selection Pool Size</th>
    <th colspan="6">Symbolic Problem Solving</th>
  </tr>
  <tr>
    <th>Oracle</th>
    <th>DCLM fastText</th>
    <th>45x</th>
    <th>0.2739</th>
    <th>0.4341</th>
    <th>0.2160</th>
    <th>0.2143</th>
    <th>0.0625</th>
  </tr>
  <tr>
    <td></td>
    <td>Oracle</td>
    <td>FineWeb-Edu</td>
    <td>45x</td>
    <td>0.2826</td>
    <td>0.4606</td>
    <td>0.2870</td>
    <td>0.1762</td>
    <td>0.0313</td>
  </tr>
  <tr>
    <td></td>
    <td>Random</td>
    <td>–</td>
    <td>1x</td>
    <td>0.2391</td>
    <td>0.4568</td>
    <td>0.1970</td>
    <td>0.2143</td>
    <td>0.0000</td>
  </tr>
  <tr>
    <td></td>
    <td>Random</td>
    <td>DCLM fastText</td>
    <td>2x</td>
    <td>0.2696</td>
    <td>0.4538</td>
    <td>0.2520</td>
    <td>0.1762</td>
    <td>0.0313</td>
  </tr>
  <tr>
    <td></td>
    <td>Random</td>
    <td>FineWeb-Edu</td>
    <td>2x</td>
    <td>0.2809</td>
    <td>0.4610</td>
    <td>0.2809</td>
    <td>0.1476</td>
    <td>0.0313</td>
  </tr>
  <tr>
    <td></td>
    <td>Random</td>
    <td>DCLM fastText</td>
    <td>4x</td>
    <td>0.1957</td>
    <td>0.4568</td>
    <td>0.2600</td>
    <td>0.1857</td>
    <td>0.0625</td>
  </tr>
  <tr>
    <td></td>
    <td>Indegree</td>
    <td>–</td>
    <td>1x</td>
    <td>0.2304</td>
    <td>0.4371</td>
    <td>0.1900</td>
    <td>0.1429</td>
    <td>0.0000</td>
  </tr>
  <tr>
    <td></td>
    <td>Indegree</td>
    <td>DCLM fastText</td>
    <td>2x</td>
    <td>0.2609</td>
    <td>0.4235</td>
    <td>0.2340</td>
    <td>0.2143</td>
    <td>0.0313</td>
  </tr>
  <tr>
    <td></td>
    <td>Indegree</td>
    <td>FineWeb-Edu</td>
    <td>2x</td>
    <td>0.2304</td>
    <td>0.4545</td>
    <td>0.2060</td>
    <td>0.1619</td>
    <td>0.0313</td>
  </tr>
  <tr>
    <td></td>
    <td>Indegree</td>
    <td>DCLM fastText</td>
    <td>4x</td>
    <td>0.2174</td>
    <td>0.4531</td>
    <td>0.2520</td>
    <td>0.1607</td>
    <td>0.0938</td>
  </tr>
  <tr>
    <td>CRAW4LLM</td>
    <td>DCLM fastText</td>
    <td>1x</td>
    <td>0.2696</td>
    <td>0.4371</td>
    <td>0.1620</td>
    <td>0.2095</td>
    <td>0.0938</td>
    <td></td>
  </tr>
  <tr>
    <td>CRAW4LLM</td>
    <td>FineWeb-Edu</td>
    <td>1x</td>
    <td>0.2609</td>
    <td>0.4750</td>
    <td>0.1780</td>
    <td>0.2048</td>
    <td>0.0938</td>
    <td></td>
  </tr>
</table>

Table 6: Results for symbolic problem solving tasks.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Pretraining Influence Scorer</th>
    <th colspan="2">Selection Pool Size</th>
    <th colspan="6">World Knowledge</th>
  </tr>
  <tr>
    <th>Oracle</th>
    <th>DCLM fastText</th>
    <th>45x</th>
    <th>0.5951</th>
    <th>0.3166</th>
    <th>0.4945</th>
    <th>0.1176</th>
    <th>0.2805</th>
  </tr>
  <tr>
    <td></td>
    <td>Oracle</td>
    <td>FineWeb-Edu</td>
    <td>45x</td>
    <td>0.6343</td>
    <td>0.3549</td>
    <td>0.5407</td>
    <td>0.1726</td>
    <td>0.2706</td>
  </tr>
  <tr>
    <td></td>
    <td>Random</td>
    <td>–</td>
    <td>1x</td>
    <td>0.5152</td>
    <td>0.2799</td>
    <td>0.5186</td>
    <td>0.0461</td>
    <td>0.2552</td>
  </tr>
  <tr>
    <td></td>
    <td>Random</td>
    <td>DCLM fastText</td>
    <td>2x</td>
    <td>0.5425</td>
    <td>0.2807</td>
    <td>0.5081</td>
    <td>0.0648</td>
    <td>0.2561</td>
  </tr>
  <tr>
    <td></td>
    <td>Random</td>
    <td>FineWeb-Edu</td>
    <td>2x</td>
    <td>0.5535</td>
    <td>0.2816</td>
    <td>0.5194</td>
    <td>0.0863</td>
    <td>0.2483</td>
  </tr>
  <tr>
    <td></td>
    <td>Random</td>
    <td>DCLM fastText</td>
    <td>4x</td>
    <td>0.5577</td>
    <td>0.2867</td>
    <td>0.5126</td>
    <td>0.0970</td>
    <td>0.2543</td>
  </tr>
  <tr>
    <td></td>
    <td>Indegree</td>
    <td>–</td>
    <td>1x</td>
    <td>0.4857</td>
    <td>0.2509</td>
    <td>0.4888</td>
    <td>0.0138</td>
    <td>0.2618</td>
  </tr>
  <tr>
    <td></td>
    <td>Indegree</td>
    <td>DCLM fastText</td>
    <td>2x</td>
    <td>0.5248</td>
    <td>0.2790</td>
    <td>0.5205</td>
    <td>0.0555</td>
    <td>0.2464</td>
  </tr>
  <tr>
    <td></td>
    <td>Indegree</td>
    <td>FineWeb-Edu</td>
    <td>2x</td>
    <td>0.5143</td>
    <td>0.2679</td>
    <td>0.5044</td>
    <td>0.0553</td>
    <td>0.2389</td>
  </tr>
  <tr>
    <td></td>
    <td>Indegree</td>
    <td>DCLM fastText</td>
    <td>4x</td>
    <td>0.5749</td>
    <td>0.2935</td>
    <td>0.5084</td>
    <td>0.0959</td>
    <td>0.2430</td>
  </tr>
  <tr>
    <td>CRAW4LLM</td>
    <td>DCLM fastText</td>
    <td>1x</td>
    <td>0.6103</td>
    <td>0.3208</td>
    <td>0.5143</td>
    <td>0.1323</td>
    <td>0.2661</td>
    <td></td>
  </tr>
  <tr>
    <td>CRAW4LLM</td>
    <td>FineWeb-Edu</td>
    <td>1x</td>
    <td>0.6427</td>
    <td>0.3592</td>
    <td>0.5319</td>
    <td>0.1859</td>
    <td>0.2736</td>
    <td></td>
  </tr>
</table>

Table 7: Results for world knowledge tasks.

Reviewer paper summaries: This work introduces CRAW4LLM, a method that uses a pretrained LM (in this case DCLM) to score webpages for inclusion in a LM pretraining dataset.  This scoring is done during the crawl, thus impacting the search and which pages are included.  The goal here is to improve crawl efficacy by only visiting/evaluating essential web pages.  The major claim and contribution of the paper is that they show this significantly outperforms baseline random/indegree methods (which are then scored via DCLM) at a given selection pool size (20M).
Summary of strengths: 1) Clear research contribution.

2) A good foundation on which further work can be based.  I feel like there are many ways to expand this paper, which is a good thing, however, most of these would be outside of the scope of the paper and as such do not need to be done for acceptance.   For example, a consideration of other LMs as scorers and the effect of their pretraining data has on selecting different web pages.  

3) Good limitations.  

4) The claims of the paper are narrow and thus a good use of the short paper format.
Summary of weaknesses: There are three major limitations: A), only a single LM is used as a scorer,  B), only a single LM is trained given the CRAW4LLM crawled data, and,  C), because of this, only a single data scale (selection pool size) is evaluated for CRAW4LLM. 

A) Given a quick look at Li et al. 2024, (which introduced the DCLM model), I was unsure how their work and yours relate in terms of experimental setting.  Lines 078, 100, and 165  give a very limited understanding of these prior works or how your work relates to them.  I think the main difference is that they used a collect-then-filter approach while you filter while collecting? If so, this should be emphasized harder by better explaining prior work.  Because only a single LM  (DCLM) is used as a scorer, this work feels like an evaluation of the DCLM model in a new environment plus the aforementioned change in approach.  This is OK, it's just not clear how your work relates to  Li et al. 2024.  As such I can not evaluate the novelty of your paper well.  

B) The main research question concerns the efficacy of finding webpages for pertaining LMs, however, only a single CRAW4LLM-based model is trained.  The issue here is that this means that C) happens which I feel is a major limitation to the underlying claims being made.  I understand LLM training is expensive and time-consuming but it would have been nice to at least see a 2X LLM being trained with CRAW4LLM (at minimum) and then have just one 3X baseline.  This at least provides two data points in terms of selection pool size instead of just one.  

While most results are good, the reading comprehension results are not (compared to the 2x baselines).  This is not discussed.  I expect this is just do to needing more data, which is one reason why the above-proposed experiment would be nice.

Importantly, I feel like evaluating CRAW4LLM at various selection pool sizes is within the current scope of the paper and would significantly improve the paper.  This is my only issue with soundness, but its a non trivial issue.   This also affects things like software release.  The core of this paper is that you present a method that others could use to do more efficient data gathering and promise to release code, which is great, however, with the results as is, I'm not sure if someone would automatically have confidence to just use your method over alternatives.  

Fixing this would raise my soundness score and slightly increase my overall assessment score and I believe a resubmit would improve this paper.

I have a low level of excitement for this paper as is.  As mentioned above this presents a good foundation on which further work can be based, which suggests that it could be published, but, it also has limitations that may suggest that Findings would be the most suitable place for publication if so.
Comments, suggestions and typos: Was DCLM trained on ClueWeb22?  This should be explicitly mentioned either way.  

Use extra space given to expand the explanation of Li 2024 and how your work is different compared to them to help emphasize novelty.
Overall assessment scores: 2.5