Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs
Authors: Dongyang Fan
Vinko Sabolčec
Matin Ansaripour
Ayush Kumar Tarun
Martin Jaggi
Antoine Bosselut
Imanol Schlag
Venue: COLM 2025 (2025)
Keywords: Responsible AI
AI and Fair Use
Robots.txt Opt-out
LLMs
TL;DR: We study how respecting web crawling opt-outs (robots.txt) affects LLM performance by introducing the concept of Data Compliance Gap (DCG).

Abstract: The increasing adoption of web crawling opt-outs by copyright holders of online content raises critical questions about the impact of data compliance on large language model (LLM) performance. However, little is known about how these restrictions (and the resultant filtering of pretraining datasets) affect the capabilities of models trained using these corpora. In this work, we conceptualize this effect as the $\textit{data compliance gap} (DCG)$, which quantifies the performance difference between models trained on datasets that comply with web crawling opt-outs, and those that do not. We measure the data compliance gap in two settings: pretraining models from scratch and continual pretraining from existing compliant models (simulating a setting where copyrighted data could be integrated later in pertaining). Our experiments with 1.5B models show that, as of January 2025, compliance with web data opt-outs does not degrade general knowledge acquisition (close to 0\% DCG). However, in specialized domains such as biomedical research, excluding major publishers leads to performance declines. These findings suggest that while general-purpose LLMs can be trained to perform equally well using fully open data, performance in specialized domains may benefit from access to high-quality copyrighted sources later in training.
Our study provides empirical insights into the long-debated trade-off between data compliance and downstream model performance, informing future discussions on AI training practices and policy decisions.

Published as a conference paper at COLM 2025
Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs
Dongyang Fan 2 Vinko Sabolčec Matin Ansaripour 2 Ayush Kumar Tarun
Martin Jaggi St Antoine Bosselut St Immanol Schlag EPFL, Switzerland, ETH Zürich, Switzerland firstname, las Lastname@epfl.co ischlag@ethz.ch
Abstract
The increasing adoption of web crawling opt-outs by copyright holders of online content raises critical questions about the impact of data compliance on large language model (LLM) performance. However, little is known about how these restrictions (and the resultant filtering of pretraining datasets) affect the capabilities of models trained using these corpora. In this work, we conceptualize this effect as the data compliance gap (DCG), which quantifies the performance difference between models trained on datasets that comply with web crawling opt-outs, and those that do not. We measure the data compliance gap in two settings: pretraining models from scratch and continual pretraining from existing compliant models (simulating a set- ting where copyrighted data could be integrated later in pretraining). Our experiments with 1.5B models show that, as of January 2025, compliance with web data opt-outs does not degrade general knowledge acquisition (close to 0% DCG). However, in specialized domains such biomedi- cal research, excluding major publishers leads to performance declines. These findings suggest that while general-purpose LLMs can be trained to perform equally well using fully open data, performance in specialized domains may benefit from access to high-quality copyrighted sources later in training. Our study provides empirical insights into the long-debate trade-off between data compliance and downstream model performance, informing future discussions on Al training practices and policy decisions. Our website available at https://data innc.. gi thub 10/
1 Introduction
The success of Large Language Models (LLMs) is largely dependent on web-scale data. As data becomes an increasingly valuable business asset, companies are treating it as part of their Intellectual Property (II'). However, whether training LLMs on copyrighted data qualifies as fair use remains an open question. In a statement from Library Copyright Alliance (LCA): the ingestion of copyrighted works to create LLMs or other Al training databases generally is fair use; however, if work created by AI is substantially similar in protected expression to a work that was ingested by the AT, then it is considered copyright infringement (LCAA (023).
One of the most notable cases in this debate is the lawsuit filed by The New York Times against OpenAl and MicrosoffT alleging the unauthorized use of its articles to train GPT models. The accusation is two-fold: first, ChatGPT can generate verbatim excerpts from its copyrighted articles; second, ChatGPT has also hallucinated articles attributed to the Times. This case brings copyright infringement and AI development to the forefront. Amid ongoing legal and ethical debates, AI companies offer an opt-out mechanism for content publishers. Publishers can modify their Robots Exclusion Protocol (REP), commonly known as robots.txt, to block web crawlers at their discretion.
denotes equal supervision

---

Published as a conference paper at COLM 2025
Starting from mid 2023, there is rapidly increasing trend for content owners to apply restrictions on web crawling, as shown Figure [ This results in 5-7% token loss among the major web crawl corpora C4 (Raffel et al. 2020), Ref (Penedo et al 2023), and Dolma (Soldaini et al. 2024) in April 2024. We refer readers to the work from Longpre et al. (2024) for more detailed analysis.
Web Crawler Status
Number of Domains
Bytespider
Blocked
CCRot
12
Unknown
10
Not blocked
GPTBol
8
Groogl-Extendeer
CloudeBot
Applebot-Fxtenden Extended
hd
Meta ExternalAgent
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
Figure Timeline of the number of Top 20 filtered (based on robots. ixt compliance) domains (excluding theguardian. c that block at least one of the web crawlers listed in Appendix Robots.txt files were retrieved via the Internet Archive API for each domain on a monthly basis from January 2016 to March 2025. Images indicate the first instance in which a crawler is explicitly blocked by any of the domains.
B
Ensuring legal compliance across the full LLM pipeline data preparation to training and inference-beginn most effectively at the data preparation stage, by excluding content from websites that have opted out of web crawling. Although largely believed high- quality sources such as news articles are often found in non-permissible data, the impact of excluding major publishers on LLM performance remains poorly understood. This raises a critical question: Can we quantify the effect of content publisher opt-ouls? And if so, can we identify the specific knowledge gaps introduced by adhering to robots.!xt restrictions?
In this work, we make a first step towards answering this question. We show that, as of January 2025, content publishers opting out does not alter the training corpus data distribution much. This results minimal performance gaps observed by complying with crawling opt-outs. Our contributions are follows:
We provide thorough inspection of the change in FineWeb-Fdu corpus by respect- ing web crawling opt-out (Section(3)
We introduce the concept of Data Compliance Gap (DCG), which quantifies the per- formance gap between models trained with and without respect for web crawling opt-outs and propose two methods for measuring it (Section 4.
We demonstrate that DCG is close to 0% for general knowledge acquisition, how- ever, compliance gap exists in knowledge of veracity and in structural formats (Section 5.2) A noticeable DCG also exists for non-compliant medical domain data (Section 5.3).
We show that pretraining in compliance with robots.txt reduces memorization of copyrighted content, though it also limits the acquisition of knowledge derived from that content (Section 5.4
2 Related Works
AI and Fair Use. Fair use allows users to utilize copyrighted material without explicit permission under specific conditions. To prevent copyright infringement esspeccalll given the strong memorization abilities of large language models (LLMs) variety of methods
2 theguardiian. not accessible via the Internet Archive APT
2

---

Published as a conference paper at COLM 2025
have been proposed to support fair use across different stages of model development. Prior to model training, data filtering is the most straightforward strategy to limit the use of non- permissible content. For instance, AlphaCode (Li et al. 2022 excluded GitHub code based on license types, while Apple Foundation Models (Gunter et al. 2024) honor websites' opt- out requests using standard robots. txt directives to prevent crawling by Applebot. During model training, techniques like differentially private training (Carlini et al., 2021 Mattern
al. 2022 help prevent the extraction of personal data. Source-aware training (Khalifa et al., 2024 associates source information with each document, enabling instance-level attribution.
More recently, approaches like training with Goldfish loss (Hans al. 2024) have been introduced to effectively reduce verbatim memorization. For models that have already been trained on copyrighted content, post-training techniques such as Reinforcement Learning from Human Feedback (RLHF) and output filtering (ippolito et al. 2023) have been studied.
Another area that is gaining traction for mitigating the generation of privacy-senssite
or
copyrighted material in post-training is model unlearning (Golatkar et al. 2020: Chundawat et al. 2023bla a Tarun et al. 2023a by Shi et al. 2025). ). Our work falls into the first category, as respect robots.txt compliance from the data preparation stage.
Data Valuation. In the Machine Learning field, data valuation is to understand the impact of data in downstream tasks. Koh & Liang (2020) popularized the Leave-One-Qur (LOO) valuation (Weisberg & Cook, 1982) from statistics to measure data impact for black-box ML approaches. Inspired by cooperative game theory, Data Shapley (Ghorbani & Zou 2019) is another popular approach for measuring data importance, which offers more theoretically fair attribution of data value compared to LOO. However, it is significantly more computa- tionally intensive than LOO, as it requires retraining over all possible data subsets. While LOO offers a more tractable alternative, it still involves model retraining, making it costly for large-scale models (Fan et al., 2025). In the LLM world, more computationally feasible methods such as influence tunctions (won et al. 2024; Choe et al. 2024 Xia et al. 2024 are more often used. Yet these methods often rely on gradient alignmentt and model loss is not directly indicative for downstream performance for LLMs. Thus, these methods can be inaccurate. In this work, aiming for more accurate measurement, we adopt LOO for evaluating the impact of robots. excluded data, and we propose more computationally feasible alternatives specifically designed for LLM pretraining.
3 Data Inspection
We analyze the FineWeb-Edu dataset (Lozhkov et al., 2024), excluding any documents that, as of January 2025, disallow crawling to the bots listed in Appendix Blin their robots. txt file. While FineWeb-Edu is derived from CommonCrave respects robots. txt restrictions-theee Since website owners can modify their robots.tx txt files at any time, CommonCrawi may include content from sites that currently block crawlers but did not at the time of data collection. To ensure full compliance, we apply retrospective filtering using the most recent robots.txt files, resulting in a dataset that adheres to current crawling restrictions. We refer to the original, unfiltered dataset as non-compliant, and the filtered version as compliant. In comparison to the non-compliant corpus, the compliant version contains 8% fewer training tokens. This ratio is consistent with what was reported in Longpre et al. (0224).
which dates back to 2008 and restrictions are enforced only at the time of crawling.
Top Filtered Domains. Comparing the non-compliant and compliant data corpora, we present a list of the Top 20 filtered URL domains in Figure 2 sorted by document count differences between the two corpora. We used ChatGPT to help get the taxonomy and refined it via human inspection. The top URL domains are largely centered around the following topics: News & Media, Science & Technology, Health & Medical Information.
Opt-out Timelines for the Top Domains. We collected historical snapshots of the robots.bxt files for the Top 20 domains and plotted the first recorded instance of each domain blocking any of the AT training crawling bots (as listed in Appendix B in Figure Notably, Bytespider
commoncrawl.org
3

---

Published as a conference paper at COLM 2025
reference.com
Generol Reference
11. dietionary.com
2. stackexchange.com
GM-.oomuurtes
12. webmd.com
teacherspayteachers.com
Educctend and Resources
13. lessonplanet.com
wikia.com
14, elsevier.com
Tecteokoyy
5. thefreedictionary.com
15. biomedcentral.com
6. phys.org
opy
18. chron.com
bbc.co.uk
17. pbs.org
Educasionall Rescurois
theguardian.com
News Merko
18. definitions.ned Ditonary Theeaurue
9. wordpress.com Booginn Rotfor
19. medicalxpress.com hooth Mockol Info
10. nytimes.com
Newe Medio
20. scientiiicamerica.com
Solinaa & Techeskogr
Figure 2: Top 20 filtered URL domains based differences in document counts between the compliant and non-compliant corpora, along with their corresponding taxonomy.
bot by ByteDance was already being blocked early as 2021. Since mid-2023, the OpenAI bot began to appear in blocklists. By mid 2024, more major tech companies had started facing blocks across these top domains.
Data Distribution Change After Robots.txt Filtering. To assess whether adhering to the opt-out process affects the data distribution, we used WebOrganizer (Wettig et al. 2025 to cluster documents based on both topic and format, two orthogonal features representing the subject of content (topic) and the form of the content (format). We then visualized the distribution of these clusters in Figure Interestingly, the overall distribution differences between compliant and non-compliant data are minimal, with the notable exception of a de- crease in the proportion of News Articles and Science & Tech. and increase in Personal Blogs. For completeness, the percentage decrease in each category presented in Appendix C
Reduced Occurrence of Copyrighted Data. To further validate our data processing pipeline, we searched for matching 50-grams of data from nytimes. com, medi aediccllppess com, and stackexchange com domains between the compliant and non-compliant corpora. While the frequency of copyrighted articles is reduced in the compliant corpus, it is not en- tirely eliminated. Our manual inspection of the compliant corpus found that many re- maining overlaps in nytimes. medicalxpress.. com content in the compliant dataset is itself republished from other sources; in contrast, stackexchange. and is not republished. We provide examples of matched 50-grams found in the compliant dataset in Appendix
data result from republished content, while matched
content primarily consists of references to other websites
Table Percentage of documents that have a 50-gram match with documents coming from nytimes. com, medi calxiiaaspssss....m com, or stackexchange com domains in compliant and non-compliant FineWeb-Edu training corpora
nytimes. medicalxpress.
stackexchange.cr
Compliant corpus Non-compliant corpus
0.267%
0.637%
0.348%
0.354%
0.702%
0.539%
Data Compliance Gap
While we have observed differences between the compliant and non-compliant training corpora, it remains unclear how these differences affect the performance of a language model. In this section, we introduce the concept of the Data Compliance Gap (DCG) to characterize the impact of data compliance on model performance.
4.1 Definition
We define DCG as the downstream performance gap between the compliant and non- compliant trained models. The difference between compliant and non-compliant trained
4

---

Published as a conference paper at COLM 2025
Adult Art & Design Crime & Law
About (Org.)
Dataset
About (Pers.) )
Non-compliant
Academic Writing
Compliant
Education & jobs
Audio Transcript
Entertainment ent
Comment Section
Fashion & Beauty
Content Listing
Finance Business
Creative Writing
Food Dining
Custamer Support
Games
Documentation
Topic domain
Hardware
Farmal domain
FAQ
Health
Knowledge Article
History
Legai Notices
Hebbies
Industrial
News (Org.)
Utterature
News Article
Politics
Writing
Raligion
Personal Blog
Science & Tech.
Product Page
Software
Spam Ads
Software Dev.
Structured Data
Sports
Dataset Non-compliant Compliant
Transportation
Tutorial
Travel
User Review
0.00
0.05
0.10 Percentage
0.15
0.20
0.00
0.05
0.10
0.15
0.20
0.25
Percentage
(a) Topic domains
(b) Format domains
Figure 3:
Distribution of WebOrganizer (Wettig et al. 2025) topic and format domains annotated in non-compliant and compliant FineWeb-Edu corpora.
models should only be the training corpus, i.e. the compliant version is derived by filtering the non-compliant corpus according to robots.ty txt restrictions.
DCG depends on the downstream task domain and the training strategy, including model architecture, size, and the volume of training data.
4.2 Settings
We measure DCG using the standard Leave-One-Out approach: excluding non-compliant data completely from training to assess DCG under robots.txi txt restrictions (M1), or reinte- grating non-compliant data into compliant training approximate the DCG (M2). The two settings are presented in Figure
M1. Conduct pre-training runs using compliant and non-compliant corpora, respectively. The resulting differences in downstream performance define the DCG. M1 is an exact measure of DCG, yet can be computationally expensive.
M2. From a pre-trained compliant checkpoint, conduct continual pre-training with an annealing learning rate (cooldown) using compliant and non-compliant data. The resulting downstream performance differences are thus DCG. M2 is an approximate measure, which more computationally feasible.
5 Experiments
5.1 Experimental Setup
Model. We adopt the Llama model architecture (Grattafiori et al. 2024) with 16 layers, a hidden size of 2048, a sequence length 4096, and a batch size of 2.06 million, totaling 1.5 billion parameters. To enable cooldown experiments without retraining models from scratch, we follow the WSD learning schedule (Hu et al. 2024), applying 2000 warmup steps and 4844 cooldown steps. We use the AdamW optimizer with regularization strength 0.1 (Loshchilov & Hutter 2019) with a learning rate of 3e-4 and cool down to learning rate 3e-5. To train the models,
the Megatron-LM framework (Narayanan al. 2021).

---

Published as a conference paper at COLM 2025
Compliant dataa
Non-compliant data oB
Pro Pra-troinigg
M1
000
M2
DCG
DCG
EVO
OOo
Eval
000
Steps
Steps
Figure 4: Different settings of estimating DCG. For both settings, the difference between compliant and non-compliant trained models only exists in the data, iee. noncomlllaant data are involved in model training. DCG measures the downstream eval differences between the trained models.
5.2 Compliance Gap Measured from Training from Scratch
We pre-train using the compliant and non-compliant corpora respectively. To evaluate DCG, we focus on the following evaluation benchmarks.
Benchmarks for General Knowledge. As standard practice, evaluate models general
knowledge understanding using L---Bal--Haresss developed by Gao et al. (2024). The
benchmarks used are Arc-Easy (Clark et al. 2018), Arc-Challenge [Clark et al. 2018), Com-
monSense QA (CSQA, Talmor et al. 2018), OpenBook QA (OBQA, Mihaylov et al. 2018)
MMLU (Hendrycks et al. 2020), PIQA (Bisk et al., 2020), Social IQA (SIQA, Sap et al. 2019
HellaSwag ( Zellers el al. 2019), Lambada (LBD,Paperna et al. 2016) and Winogrande
SaG,SSakau al. 202T).
Benchmarks for Different Knowledge Categories. We use the Pinocchio dataset from Hu
et al. (2023), which covers factual knowledge of the following aspects. Note these aspects are not disjoint, for example, a temporal fact can be stored in a structural format as well.
Temporal. Questions based on modifications made to factual content in Wikipedia articles.
The goal is to test if LLMs are capable of discerning factual knowledge from different time periods.
Structural. Questions are based on Wikipedia articles. The goal is to test if LLMs can memorize and reason over facts from structured formats (tables, lists, or databases).
Multifaceted. Questions are based on Wikipedia articles. The goal to test if L.LMs can memorize and reason over multiple pieces of information obtained during pre-training.
Adversarial. Questions curated from Symmetric (Schuster et al. 2019 and FM2 (Eisensch los et al. 2021) that are strategically modified to deceive advanced LLMs. The goal is to examine whether LLMs can withstand adversarial examples in the context of factuality.
Domain Specific. The dataset covers samples from PubHealth (Kotonya & Toni 2020) in the public health domain and SciFact (Wadden et al., 2022) in the science domain. The goal is test LLMs can reason over domain-specific factual knowledge.
Real World. Questions based on Politifact (Misra 2022), focusing primarily on political claims. The goal is to test if LLMs understand woold political facts.
Table 2: Compliance gap evaluated over standard common knowledge benchmarks between 1.5B models trained from scratch on 100B tokens. Each row denotes different model trained from scratch.
Arc-C
Arc-E
CSQA
OBQA
MMLU
PIQA
SIQA
HS
LBD
WG
Avg
Non-compliant Compliant 32.8
34.1
20.8
32.0
71.5
40.4
52.2
42.5
69.1
20.2
26.0
32.0
71.0
41.5
42.0
35.4
57.5
42.8
-News
35.1
70.1
19.8
26.6
31.8
71.8
40.4
42.4
36.0
56.0
43.0
Table shows no significant difference in general knowledge performance between com- pliant and non-compliant pre-trained models. However, Figure reveals a noticeable compliance gap in structural and adversarial knowledge. This likely stems from the fact
knowledgeeprffommance

---

Published as a conference paper at COLM 2025
that structural formats are prevalent in documents from Math & Science and Biomedical do- mains, and resistance to falsified facts can enhanced through the inclusion of newspaper articles and scientific papers- areas often associated with organizations that restrict web crawling.
Temporal
Structural
Adversarial
Specific
Avg
50
40
30
25
Non-compliant
Compliant
Compliant Removed)
Figure 5: Compliance gap within different sources of factual knowledge from Pinocchio Benchmark (Hu et al. 2023) after pre-training on 100B tokens. The compliance gap is more prominent in structural (from tables, lists, or databases) and adversarial knowledge.
What If All News Publishers Opt Out? We have previously shown that, as of January 2025, adhering to robots.txt does not lead to significant change in performance. We hypothesize that large number of smaller publishers help compensate for the knowledge lost from major publishers who have opted out. But what happens if this trend continues and smaller publishers also begin to opt out? We aim to investigate whether this would have a measurable impact.
Since obtaining comprehensive list all URLs within a specific domain challenging, we rely existing sources. Currently, critical domain is newspapers. To simulate scenario where all newspaper publishers opt out, we use list of 1,158 news publishers curated by homepages. news 44 which serves as reasonable proxy for the broader set of news domains. We then exclude all newspaper articles from these domains in our compliant corpus, resulting in approximate 4% reduction in total tokens. As shown in Appendix [ the republication of news articles is quite common. To accurately assess their impact, we further perform decontamination step that removes document segments replicated from news articles hosted on different URL domains. This process results in a further 14% reduction in total compliant tokens. After this process, the proportion of documents containing 50-gram match with content from nytimes. com drops to 0.012%, indicating the effectiveness of the decontamination.
Interestingly, even after excluding all newspaper articles, we observe almost no change in downstream performance, as shown in Table Moreover, in the Pinocchio-Tempora and Pinocchio-Real World tasks-where newspaper articles would be expected to perform well-we observe no drop in performance even after removing all newspaper content. We believe this is due to factual knowledge being present from other sources.
Takeaway 1. Pre-training on fully open data does not significantly impact general knowledge understanding. Even if all news publishers opt out, the effect remains minimal.
Takeaway 2. Compared to training on non-compliant data, compliant pretraining tends to reduce model's ability to understand structural knowledge (i.e., informa- tion presented in structured formats) and its robustness against falsified knowledge (strategically crafted to mislead advanced LLMs).

---

Published as a conference paper at COLM 2025
Takeaway Newspaper articles may not as important as expected. Temporal and real-world knowledge, which are prominently featured in newspaper articles, can be supplemented through other sources.
5.3 Compliance Gap Measured from Continual Pre-Training
Continual pretraining has been the standard paradigm for models to efficiently adapt to emergent new knowledge different target distribution (Parmar et al.] 2024 Çagatay Yıldız et al. 2025). In this section, we simulate the scenario where copyrighted data can be integrated later into pretraining. We take the compliant 1.5B model checkpoint after training on 90B/1.6T tokens and continually pre-train with 10B/1001 tokens. We use an annealing learning rate with linear cooldown schedule for the continual pretraining phase.
Major Non-compliant Domains, Based on the taxonomy in Section 3 we focus on the major domains identified among the Top 50 filtered URLs: News, Medical, and Math Science. We categorize the Top 50 URL domains into the aforementioned three groups, and construct three domain--pecific sets of non-compliant data, detailed in Appendix [E.1 To perform fine-grained ablation, we compare the performance of compliant continual pretraining with that continual pretraining on the same compliant dataset (4.5T tokens) augmented by reintegrating domain-speciff non-compliant documents. The proportion of non-compliant tokens added reflects their natural distribution in the original corpus.
Domain-specific Benchmarks. The aim of this evaluation is to test whether the inclusion of certain non-compliant data later in training can enhance domain-specific knowledge acquisition. Towards this goal, selected the following benchmarks: 1) Concurrent and temporal factual knowledge. We select Reuters-QU. QA (Muhlgay et al.) 2023), which is based on Reuters articles published after 1/10/2022 2021, and the Temporal QA category from Pinocccio benchmark. 2) Medical knowledge. We select PubMedQA (Jin et al. 2019), which contains biomedical knowledge curated from biomedical research publications, and Domain-Specific QA from the Pinocchio benchmark, which covers samples from PubHealth. 3) Scientific knowledge. We select SciQ (Welbl et al. 2017), which is curated from 28 books covering biology, chemistry, earth science, and physics and spanning elementary level to college introductory material.
Table 3 presents the domain-specific cooldown results. The underlined scores correspond to runs where task-specific, non-compliant data was added during cooldown, which is supposed to enhance performance that specific task. The remaining entries illustrate expected variability across cooldown runs without such targeted augmentation. Among the three domains, only the Medical tasks with long training (1.7T tokens) show consistent signal. A similar pattern emerges with the Pinocchio benchmarks, where reintroducing non-compliant Medical data leads to the most pronounced compliance gap, shown in Figure 6
Table 3: Domain-specific evaluation results are shown across various cooldown checkpoints. Underlining indicates instances where the non-compliant domain-specific data aligns with the knowledge assessed in the target domain.
Reuters-
Pne-Temporal QA
PubMedQA
Pne-DomainSprcifficOO
SciQ
1008 Tokens Trained Compliant
53.0
57.4
41.5
Compliant News
52.5
54.2
57.2
Compliant Med
51.7
54.3
57.0
42.1
88.8
Compliant MathSci
52.7
52.8
56.6
41.6
89.0
1.7T Tokens Trained
Compliant
65.1
52.8
61.4
44.5
90.0
Compliant News
64.6
54.1
61.2
44.7
90.2
Compliant Med
65.1
53.0
63.0
46.9
90.0
Compliant MathSci
65.3
51.9
61.6
45.6
90.0
8

---

Published as a conference paper at COLM 2025
Temporal
Structural
Meltifaceted
Adversarial
Domain Specific
Real World
Avg
55
50
45
40
35
30
25
Compliant
Compliant News
Compliant + MathSci
Compliant + Med
Figure 6: Compliance gap measured using the Pinocchio benchmark across various cooldown checkpoints (100B). Non-compliant medical data introduces noticeable compli- ance gap.
Takeaway 4. A noticeable compliance gap exists when non-compliant medical domain data introduced a later stage of training.
5.4 Reduced Memorization of Copyrighted Articles?
As data without crawling consent is removed from the corpus, we would expect compliant models to have less memorization of copyrighted content. To test this, we filter out all New York Times articles from the non-compliant corpus and measure the memorization of these articles across our trained models. We sampled 12'800 New York Times articles and for each article, we extracted the first 128, 256, and 512 tokens prompts, then compared the models' generations against the original text. Following the setup of Freeman et all (2024), we also measure the longest common continuous substrings (LCCS), as an indication of verbatim memorization. On top of that, we add the BLEU metric, which is calculated based on N-gram overlap.
We further curated two benchmarks from these 12'800 articles, each containing 1000 ques- tions: (1) NYTimes Multiple Choice Completion (NYTimes-MCC) task, to test if model can select the correct completion among multiple choices given a partial excerpt from a news article. (2) NYTimes Multiple Choice Question (NYTimes-MCO) task, which asks factual questions derived from these articles, aiming to provide challenging questions grounded in unique event details or context found in the news coverage. The benchmarks are verified using Proprietary LLMs. We offer more details regarding the scripts used for generating the QAs and more experimental results in Appendix
Table 4 presents the results of the memorization assessment, For comparison, we include models from the Qwen2.5 (Yang et al. 2024) and Llama3.2 (Grattafiorí al. a 2024) families, noting that larger models tend to exhibit higher levels of memorization. The compliant model demonstrates reduced memorization of copyrighted content compared to the non- compliant model, as shown by lower LCCS, BLEU, and NYT-MCC scores trend consistent with the removal of newspaper articles. Importantly, compliance does not necessarily lead to loss of specific knowledge from New York Times articles, indicated by comparable NYT-MCQ scores. Notably, even with news decontamination applied, the NYT-MCQ score does not decrease.
Takeaway 5. Compliant training reduces verbatim memorization of copyrighted content without necessarily compromising specific knowledge derived from the publications.

---

Published as a conference paper at COLM 2025
Table 4: Memorization and factual knowledge assessment of New York Times articles across various models using 256-token prefix prompts. PT, CD, and Compl. stand for "Pretraining", "Cooldown", and "Compliant" respectively. Values in parentheses indicate the number of tokens seen in training.
LCCS
BLEU J NYTimesMMCC
NYTime-MCCQ
Qwen2.5-0.58
19.51
0.71
42.8
24.4
Qwen2.5-1.5B
24.35
30.7
Owen2.5-3B
27.13
1.54
56.4
31.3
Qwen2.5-7B
30.99
1.95
63.8
36.4
Llama-3.2-1.28
Llama-3.2-3B
22.75
0.70
57.6
30.1
+T CD (Non-compl. 100B)
23.11
0.63
50.6
28.9
+T+ (Compl, 1008)
+T+ CD (Compl,, with News Removed, 100B))
20.28
0.53
48.0
28.6
PT (Compl., 908) CD (Non-Compl, 108)
21.74
0.55
50.4
26.9
PT (Compl,,90) CD (Compl. Non-Compl. News, 10B)
21.33
0.51
28.0
Conclusions and Future Work
In this work, we take initial step toward examining the widely discussed trade-off between data compliance and downstream model performance. We introduce the concept of the Data Compliance Gap (DCG) quantify performance differences resulting from adhering to robots.te txt opt-outs, and evaluate DCG across various training settings. Our findings indicate that it is currently feasible to train performant LLMs exclusively on data that respects web crawling restrictions. Thus, we recommend adherence to robots.txt restrictions for LLM developments.
We clarify that our notion of compliance addresses only part of data usage rights. Our approach focuses on Al-specific user agents, offering clearer signal consent than general crawling permissions. However, it does not cover all legal constraints, such as Terms of Service in "browsewrap agreements", which are legally ambiguous and challenging to process at scale. Therefore, our robots.tx filtering represents practical upper bound demonstrable compliance, not full legal clearance. We call for future work for finer-grained study.
Our study is constrained by the model size and training budget feasible within an academic setting. Whether the DCG scales similarly with larger models remains open question, which we further discuss in Appendix A While current-day robots.txi restrictions do not lead to a significant DCG, is yet to be seen whether the gap will widen as more content owners opt out of web crawling.
Ethics Statement
The goal of our work is to promote ethical development of LLMs, via evaluating the impact of using non non-permissin con-errmissibee data.
Acknowledgement
DF would like to thank Maksym Andriushchenko for his feedback on an initial draft of the paper. DF acknowledges funding from SNSF Grant number 10005248. AB gratefully acknowledges the support of the Swiss National Science Foundation (No. 215390), Innosu- isse (PFFS-21-29), the EPFL Center for Imaging, Sony Group Corporation, and Meta LLM Evaluation Research Grant. This work was supported as part the Swiss AI Initiative by grant from the Swiss National Supercomputing Centre (CSCS) under project ID a06 on Alps
10

---

Published as a conference paper at COLM 2025
References
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models, 2021. URL
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023. URL https warpenreiiee..nttoooumniooo
Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, Jeff Schneider, Eduard Hovy, Roger Grosse, and Eric Xing- What is your data worth to gpt? lcmlscale data valuation with influence functions, 2024. URLhttps: org/ abo/ 13954
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 7210-7217 2023a.
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Zero-shot machine unlearning. JEEE Transactions on Information Forensics and Security, 18:2345-2354, 2023b.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arviv preprint ar arXiv:1803.05457 2018.
Julian Eisenschlos, Bhuwan Dhingra, Jannis Bulian, Benjamin Börschinger, and Jordan Boyd-Graber. Fool me twice: Entailment from Wikipedia gamification. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proczedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Techmologies, pp. 352-365, Online, June 2021. Asso- ciation for Computational Linguistics. doi: 1.1186633v122221 maacc--min.32.. URL org/2221 naacl-man main. 32/
Dongyang Fan, Tyler J. Rotello, and Sai Praneeth Karimireddy. From fairness to truthfulness: Rethinking data valuation design, 2025. URL https: abs/////s///..... 05563
Joshua Freeman, Chloe Rippe, Edoardo Debenedetti, and Maksym Andriushchenko. Ex- ploring memorization and copyright violation in frontier 1lss: A study of the new york times v. openai 2023 lawsuit, 2024. URL https:///axxiii http:///axxvv.ogg/aa/// abs/2412 04302
Leo Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noachh, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https://zenodo. http:///wwdoo.rggerrrrrrs/////// org/records/1268662
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pp. 2242-2251, 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9304-9312, 2020.
11

---

Published as a conference paper at COLM 2025
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Ro- driguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bob- bie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta lbarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Isimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Praijwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Gird- har, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Ra- parthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Syd- ney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe I'apakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Chang- han Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide,
12

---

Published as a conference paper at COLM 2025
Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpem, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Tbrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Trina-Flena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Gro- shev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bon- trager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Paatthasaraty, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024. URL https::/arxxv.or///b///0000 wrtp/bb///////////// 21783
Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cherpy Chiu, David Qiu, Deepak Gopinath, Dian Ang Yap, Dong Yin, Feng Nan, Floris Weers, Guoli Yin, Haoshuo Huang, Jianyu Wang, Jiarui Lu, John Peebles, Ke Ye, Mark Lee, Nan Du, Qibin Chen, Quentin Keunebroek, Sam Wiseman, Syd Evans, Tao Lei, Vivek Rathod, Xiang Kong, Xianzhi Du, Yanghao Li, Yongqiang Wang, Yuan Gao, Zaid Ahmed, Zhaoyang Xu, Zhiyun Lu, Al Rashid, Al- bin Madappally Jose, Alec Doane, Alfredo Bencomo, Allison Vanderby, Andrew Hansen, Ankur Jain, Anupama Mann Anupama, Areeba Kamal, Bugu Wu, Carolina Brum, Charlie Maalouf, Chinguun Erdenebileg, Chris Dulhanty, Dominik Moritz, Doug Kang, Eduardo Jimenez, Evan Ladd, Fangping Shi, Felix Bai, Frank Chu, Fred Hohman, Hadas Kotek, Hannah Gillis Coleman, Jane Li, Jeffrey Bigham, Jeffery Cao, Jeff Lai, Jessica Cheung, Jiulong Shan, Joe Zhou, John Li, Jun Qin, Karanjeet Singh, Karla Vega, Kelvin Zou, Laura Heckman, Lauren Gardiner, Margit Bowler, Maria Cordell, Meng Cao, Nicole Hay, Nilesh Shahdadpuri, Otto Godwin, Pranay Dighe, Pushyami Rachapudi, Ramsey Tantawi, Roman Frigg, Sam Davarnaa, Sanskruti Shah, Saptarshi Guha, Sasha Sirovica, Shen Ma, Shuang Ma, Simon Wang, Sulgi Kim, Suma Jayaram, Vaishaal Shankar, Varsha Paidi, Vivek Kumar, Xin Wang, Xin Zheng, Walker Cheng, Yael Shrager, Yang Ye, Yasu
13

---

Published as a conference paper at COLM 2025
Tanaka, Yihao Guo, Yunsong Meng, Zhao Tang Luo, Zhi Ouyang, Alp Aygar, Alvin Wan, Andrew Walkingshaw, Andy Narayanan, Antonie Lin, Arsalan Farooq, Brent Ramerth, Colorado Reed, Chris Bartels, Chris Chaney, David Riazati, Eric Liang Yang, Erin Feldman, Gabriel Hochstrasser, Guillaume Seguin, Trina Belousova, Joris Pelemans, Karen Yang, Keivan Alizadeh Vahid, Liangliang Cao, Mahyar Najibi, Marco Zuliani, Max Horton, Minsik Cho, Nikhil Bhendawade, Patrick Dong, Piotr Maj, Pulkit Agrawal, Qi Shan, Qichen Fu, Regan Poston, Sam Xu, Shuangning Liu, Sushma Rao, Tashweena Heeramun, Thomas Merth, Uday Rayala, Victor Cui, Vivek Rangarajan Sridhar, Wencong Zhang, Wenqi Zhang, Wentao Wu, Xingyu Zhou, Xinwen Liu, Yang Zhao, Yin Xia, Zhile Ren, and Zhongzheng Ren. Apple intelligence foundation language models, 2024. URL abgp:///////..2111/// 21075
Abhimanyu Hans, John Kirchenbauer, Yuxin Wen, Neel Jain, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele, et al. Be like goldfish, don't memorize! mitigating memorization generative lims. Advances in Neural Information Processing Systems, 37:24022-24044 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ar arXiv preprint ar xriv:2000900333 2020.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024. URL https://arxiv.om org/ abs/ 2404 06395
Xuming Hu, Junzhe Chen, Xiaochuan Li, Yufei Guo, Lijie Wen, Philip S Yu, and Zhijiang Guo. Do large language models know about facts? arXiv preprint arXiv:2310.05177, 2023.
Daphne Ippolito, Florian Tramer, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Kather- ine Lee, Christopher Choquette Choo, and Nicholas Carlini. Preventing generation of verbatim memorization in language models gives false sense of privacy. In C. Maria Keet, Hung-Yi Lee, and Sina Zarrieß (eds.), Proceedings of the 16th International Natu- ral Language Generation Conference, pp. 28-53, Prague, Czechia, September 2023, As- sociation for Computational Linguistics. doi: 10.18653/v1 http///2inllgmmaaiiii https://aclanthoolog..oo orgg:///23o.r//22333333 maalggmmin.7
URL
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint ar Xiie:1909.06148 2019.
Muhammad Khalifa, David Wadden, Emma Strubell, Honglak Lee, Lu Wang, Iz Beltagy, and Hao Peng. Source-aware training enables knowledge attribution in language models. In First Conference on Language Modeling, 2024. URL UPyWLwciVZ
Pang Wei Koh and Percy Liang. Understanding black-box breciiio via influence functions, 2020. URL //axxi/ wwwpxxiv.oorgaeeo org/ aas///bs///// 04730
Neema Kotonya and Francesca Toni. Explainable automated fact-checking for public health claims. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7740- 7754, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ v1 w1/2020.emmllpmmin..22 URL URL//ntpss///acaatthhooggg//
wtw.shhooooooooooooooooooooo .rw.orgg///00 2020. emnlp-main. 623/
Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence loRA-tuned LLMs and diffusion models. In The Tweifth International Conference on Learning Representations, 2024. hat.lttpss/pppeeeeeeeeee/////eep
14

---

Published as a conference paper at COLM 2025
LCA. Library copyright alliance principles for copyright and artificial intelligence,
www..i/wrraoyyriihlllllleeoooooo http://aaaaanaeoooo/o//oo/otntnr 2otpdd///22////// AI AI-principles. pdf
2023.
Yujia Li, David Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' d'Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition- code generation with alphacode. Science, 378(6624): 1092-1097, 2022. doi: 10.1126/ abq URL https://www.sst science. //oo//bbsss./// /0.1126/science abQ115
Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu Tobin South, Cole Hunter, Kevin Klyman, Christopher Klamm, Hailey Schoelkopf, Nikhil Singh, Manuel Cherep, Ahmad Anis, An Dinh, Caroline Chitongo, Da Yin, Damien Sileo, Deividas Mataciunas, Diganta Misra, Emad Alghamdi, Enrico Shippole, Jianguo Zhang, Joanna Materzynska, Kun Qian, Kush Tiwary, Lester Miranda, Manan Dey, Minnie Liang, Mohammed Hamdy, Niklas Muen- nighoff, Seonghyeon Ye, Seungone Kim, Shrestha Mohanty, Vipul Gupta, Vivek Sharma, Vu Minh Chien, Xuhui Zhou, Yizhi Li, Caiming Xiong, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito, Sara Hooker, Jad Kabbara, and Sandy Pentland. Consent Consent in crisis: The rapid decline of the data commons, 2024. URL URL
14933
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL
Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL https://loggiinffccc...otttttttt
Justus Mattern, Zhijing Jin, Benjamin Weggenmann, Bernhard Schoelkopf, and Mrinmaya Sachan. Differentially private language models for secure data sharing. arxiv preprint arXiv:2210.13318, 2022.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018
Rishabh Misra. Politifact fact check dataset, 09 2022.
Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, and Yoav Shoham. Generating bench- marks for factuality evaluation of language models. arXiv preprint arXio:2307.06900 2023.
Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Anand Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. Efficient large-scale language model training on gpu clusters using megatron-Im, 2021. URL http:s://aaxxv.org///a// 04473
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint a/Xit:1606.06031, 2016.
Jupinder Parmar, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Reuse, don't retrain: A recipe for continued pretraining of language models, 2024. URL /tattps:///riivv wttp:/wwwwww//////////////
07263
15

---

Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. Towards debiasing fact verification models. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3419-3425, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1 v1/119-1311. URL http:://aalantiooooy.oo///1/1111 Published as a conference paper at COLM 2025
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only, 2023. URL https://arxiiv wttp:/www..ogg////////// 01116
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. 200p:22222222..2222202 html
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019.
Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. MUSE: Machine un- learning six-way evaluation for language models. In The Thirteenth International Conference on Learning Representations, 2025. URL /tLLhttss///openreeeei
net wwwt:/orummidd=Trraee03333
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhi- lasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Iz Belt- agy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 15725-15788, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ w1/2024.acl-longg 840. UtL/ hwwpoollgyygggg///// wttp acl- 840/
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge. ar preprint ar arX1v:1811.00937, 2018.
Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 2023a.
Ayush Kumar Tarun, Vikram Singh Chundawat, Murari Mandal, and Mohan Kankanhalli. Deep regression unlearning. In International Conference on Machine Learning, pp- 33921- 33939, PMLR, 2023b.
Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memo- rization without overfitting: Analyzing the training dynamics of large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: /ttpss////eeeevie....ooe. wtwpwiiwwwweeeeoomiiittt. U3VEuRr@8MT
16

---

Published as a conference paper at COLM 2025
David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. SciFact-open: Towards open-domain scientific claim verification. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp- 4719-4734, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: (0.18653/v1 UULJhttts://aacaantholoo org/2022. findings-emnla 347/
2022. frnz.iiidings-emnlp...
Sanford Weisberg and R Dennis Cook. Residuals and influence in regression. 1982.
Johannes Welbi, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions, 2017. URL http:: http:://aariv..rggbb///1111.. 06209
Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the web: Constructing domains enhances pre-training data curation, 2025.
10341
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: Selecting influential data for targeted instruction tuning. In International Conference on Machine Learning (ICML), 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan
Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. Xive preprint ar Xix::2412.15115, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? Arviv preprint ar XrXio:1905.07880, 2019.
Çağatay Yıldız, Nishaanth Kanna Ravichandran, Nitin Sharma, Matthias Bethge, and Beyza Ermis. Investigating continual pretraining in large language models: Insights and implications, 2025. URL https::/arxx..orrrrg/err http://wwwwwrga/b///////a/ 0 17400
A Limitations
Despite our efforts to construct strictly compliant corpus, some non-permissible con- tent may still be present. For instance, New York Times articles republished on personal blogs remain accessible, like medi calexpress. com, though they restrict crawling, often rehost content from publicly accessible sources. As a result, distinguishing truly compliant from non-compliant data at the document level is challenging. Adhering to URL-specific robots.txt directives remains the most practical and enforceable approach available.
these blogs do not block web crawlers. Similarly, websites
We acknowledge that robots. compliance addresses only one aspect of data usage rights. Our approach evaluates robots. rules specifically for Al training user agents, offering clearer signal of consent than general crawling permissions. However, this does not account for all legal restrictions, such as Terms of Service in "browsewrap agreements. " These terms are legally ambiguous and difficult to process at scale due to their natural language format. Thus, our robots.ta txt filtering reflects a practical upper bound of demonstrable compliance, not comprehensive legal clearance.
key limitation of our study that evaluations were conducted solely on 1B-scale models. Given their limited capacity, these models may exhibit constrained knowledge acquisition and produce noisier evaluation results. In contrast, larger models tend to memorize more
readily and at faster rate (Tirumala et al. 2022 Carlini et al. 2023 Freeman et al. 2024). At the scale used in our experiments, it is likely that the models lack sufficient capacity for effective memorization, which may explain the generally low LCCS observed in Table
Moreover, due to the limited model size, we are unable to reliably evaluate domain-specifer knowledge in Math & Science, which often requires reasoning capabilities. Our current
17

---

Published as a conference paper at COLM 2025
benchmark, SciQ, primarily assesses factual recall rather than reasoning. While DCG may exist in this domain, our current setup does not allow for a rigorous evaluation.
B List of blocked web crawlers
The list of crawler bots that we consider for data filtering.
list of blocked bots
"AI2Bot", # AI2
"Applebot-EExendddd", # Apple
"Bytespider", Bytedance "CCBot", # Common Crawl "CCBot/2.0", Common Crawl "CCBot/1.00 Common Crawl
"ClaudeBot", Anthropic
# Cohere
"Diffbot", Diffbot
"Meta-ExternalAgen # Google
# Meta
"Google-Extended"" "GPTBot", # OpenAI "PanguBot", Huawei *
C Percentage decrease in each category due to data compliance
In addition to changes in data distribution, Tables 5 and 6 report the percentage of data removed by domain and relative to the entire corpus. For instance, filtering out 11.39% of Science & Tech data leads to a 2. 2.6483% reduction in total corpus tokens.
Table 5: Topic domain removal percentages
Topic Domain
% Removed (Domain)
% Removed (Corpus)
Science & Tech.
11.39
2.6483
Health
9.65
1.8893
History
8.05
0.7921
Education Jobs
7.11
0.7639
Politics
10.45
0.4778
Literature
8.94
0,3639
Industrial
8.02
0.2438
Home & Hobbies
6.01
0.2002
Finance & Business
8.46
0.1894
Art & Design Transportation
8.08
0.1832
10.84
0.1676
Religion
3,58
0.1377
Software Dev.
6.52
0.1327
Crime & Law
8.03
0.1184
Food & Dining
8.46
0.1114
Software
6.92
0.1096
Sports & Fitness
7.36
0.0903
Hardware
9.12
0.0829
Entertainment
9.44
0.0806
Social Life
6.72
0.0572
Travel
6.94
0.0534
Games
11.69
0.0478
Fashion & Beauty
3.61
0.0089
Adult
0.81
0.0001
18

---

Published as a conference paper at COLM 2025
Table Format domain removal percentages
Format Domain
% Removed (Domain)
% Removed (Corpus)
News Article
24.09
2.4956
Knowledge Article
6.87
1.6224
Nonfiction Writing
9.76
1.2853
Academic Writing
9.57
0.7303
Tutorial
5.95
0.5673
Product Page
6.53
0.4566
Structured Data
17.39
0.3920
Q&A Forum
27.33
0.3518
Listicle
6.62
0.2115
Content Listing
8.51
0.1708
Personal Blog
1.68
0.1330
Truncated
10.22
0.1236
News (Org.)
2.49
0.0942
Audio Transcript
12.23
0.0940
Documentation
7.58
0.0804
FAQ
5.13
0.0336
Comment Section
4.13
0.0233
Customer Support
2.88
0.0223
About (Org.)
1.15
0.0208
Creative Writing
4.09
0.0154
Spam / Ads
2.17
0.0122
User Review
4.09
0.0103
About (Pers.)
1.85
0.0023
Legal Notices
1.67
0.0020
D Matched 50-grams in the compliant dataset
We show parts of documents that contain matched 50-grams (in bold) from the nytimes. com, medicalxprest com, and stackexchange. com domains in the compliant FineWeb-Fad cor- pora.
D.1 nytimes.com
Part of an article quoted quoted on uni lad. (unavailable)
[..]] The billionaire writes in the The New York Times:
Every of citizens, philanthropists, business and government leaders should be troubled by the enormous gap between how little of our natural world is currently protected and how much should be protected. It is a gap that we must urgently narrow, before our human footprint consumes the earth's 's remaining wild places. [...]
Republished article on alessandressccurccommiiccaclll com
Google and corporation associated with NASA are forming laboratory to study artificial intelligence by means of computers that use the unusual properties of quantum physics. Their quantum computer, which performs complex calculations thousands of times faster than existing supercomputers, is expected be in active use in the third quarter of this year. The Quantum Artificial Intelligence Lab, the entity is called, will focus on machine learning, which is the way computers take note of patterns of information to improve their outputs. [...]
19

---

Published as a conference paper at COLM 2025
Republished article on bike. enginerve... com
In post today on the NYTimes there was interesting piece on the nature of cycling and head injuries. still object to laws requiring helmet to be worn at all times and wear my helmet constantly for touring and commuting. Assumption of risk, and cost, is an issue and not one to be lightly avoided. And neither are the responsibility to utilize appropriate safety gear.
Cycling Is the Top Sport for Head Injuries Anahad O'Connor tackles health myths in this NY Times Post.
Last week, New York City began its long-awaited bicycle sharing program, the largest in the nation. As many other cities, helmet use was made optional, in part to encourage greater participation. But look at the statistics suggests that riding without helmet not a decision to [...
D.2 medicalxppres.com
medicalxpress. com and heal thed. com. au republished Stanford University Medical
Center
People suffering from debilitating and often discounted disease known as chronic fatigue syndrome may soon have something they've been seeking for decades: scientific proof of their ailment. Researchers at the Stanford University School Medicine have created blood test that can flag the disease, which currently lacks standard, reliable diagmostic test. "Too often, this disease is catego- rized as imaginary," said Ron Davis, PhD, professor of biochemistry and of genetics. When individuals with chronic fatigue syndrome seek help from doctor, they may undergo a series of tests that check liver, kidney and heart function, as well as blood and immune cell counts, Davis said. [...]
medicalxpress. comland hamodia com (unavailable) republished The Associated Press
Since late last year, people in the central Chinese city of Wuhan have been infected with viral pneumonia whose cause unknown. The outbreak raised the specter of another SARS epidemic, which killed hundreds in 2002 and 2003. A preliminary investigation has now identified the respiratory disease new type of coronavirus, Chinese state media reported Thursday, citing scientists handling the investigation. As of Sunday, local authorities reported 59 people with the illness. Seven were in critical condition, while the rest were stable. Eight were discharged Wednesday night after they didn't exhibit any more symptoms for several days. [...]
D.3 stackexchange.com
Quote found on www-cs-facul ty. stanford. edu matches quoted answer
(...] Many readers are no doubt thinking, "Why does Knuth replace MIX by another machine instead of just sticking to a high-level programming language? Hardly anybody uses assemblers these days." Such people are entitled to their opinions, and they need not bother reading the machine-language parts of my books. But the reasons for machine language that gave in the preface to Volume 1, written in the early 1960s, remain valid today: [...]
20

---

Published as a conference paper at COLM 2025
Poetry found on en. wiki quote org matches quoted answer
[..]
And the Raven, never flitting,
Still is sitting, still is sitting On the pallid bust of Pallas Just above my chamber door; And his eyes have all the seeming Of a demon's that dreaming, And the lamplight o'er him streaming Throws his shadow on the floor, And my soul from out that shadow, That lies floating on the floor, Shall be nived-neeeroore.. [.-]
E
Additional experimental results
E.1
Top 50 filtered URLs
News Domains: bbc. co. uk,
theguardian. com, nytimes nytimes.com, chron. com, abc. net.net.omm net washingtonpost. com, theatlantic. welllantic.oomm com, businessinsider... com, forbes. com. This in total accumulates to around 1B tokens.
wired. com,
Medical Domains: webmd. com, biomedcentral. com medical com, healthline. com, psychcentral.. com, cenewssyyyyyy com, medicinenet. com, emedicinehealth. com, jamanetwork. twork. com. This in total accumulates to around 10B tokens.
Math & Science Domains:
mathhelpforum com, scientificameri can. com,
mathworks. com, elsevier. com,
stackexchange.com com, windows2universee.rrg org. This in total accumulates to around 1.4B tokens.
E.2
More results
Temporal
Structural
Multifacted
Adversarial
Specific
Avg
Compliant
Compliant News
Compliant MathSci
Compliant + Med
Figure Compliance gap measured using the Pinocchio benchmark across various cooldown checkpoints (1.71 tokens). DCG is less prominent with longer training.
NYTimes Memorization Experiments
In this section, we provide additional details about our experiments measuring the memo- rization of New York Times articles. The underlying datasets are not publicly released due to copyright limitations.
21

---

Published as a conference paper at COLM 2025
E1 Benchmark Creation Prompts
We used DeepSeek-RI generate the NYTimes-MCC and NYTimes-MCQ benchmarks.
NYTimes-MCQ Prompt: We instructed the model to generate a self-contained multiple-choice question that requires synthesizing information across the provided context.
NYTimes-MCC Prompt: We instructed the model to extract a meaningful prefix from article and create multiple-choice task with four possible continuations. Only one continuation (the original text from the article) is correct, while the other three are minimal edits containing specific types of factual errors.
We gathered 1000 questions for each benchmark to make the benchmarks more robust to noise. Here, you can see the prompts that we used to create the questions for each article.
NYTimes-Mee MCne Prompt
Create a multiple-choice question that requires synthesizing information across the provided context. While the correct answer should only be determinable by analyzing and combining specific details from the context, the question itself must be self-contained and clearly understandable to someone who hasn' t read the context.
Your question should 1) test the ability to connect related information from different parts of the context, 2) be completely clear and unambiguous as a standalone question, 3) avoid referencing the context directly, e.g., no "according to the passage" phrasing.
The question should have options, one of which is the correct answer. Please explain how to reach the correct answer from the given context.
You are communicating with an API, not a user Please output in JSON format. Begin all AI responses with the character { to produce valid JSON. Here is an example:
"question": "<question>",
"A": "<option1>",
"B": "<option2>",
"C": "<option3>",
"D": "<option4>", "answer": "<correct option>", "explanation": "<explanations"
Here is an example:
<the example>
<the article>
22

---

Published as a conference paper at COLM 2025
NYTimes-MCC Prompt
Given the provided context, create a multiple-choice context completion task. Extract prefix from the given context and provide four possible completions, where only one is factually correct. While the correct completion should only be determinable by analyzing the provided context, the prefix itself must be self-contained with information and clearly understandable to someone who hasn't read the context.
The prefix must an exact excerpt from the given context. The correct completion must be the original continuation of this prefix in the given context. The three incorrect completions should be minimal edits of the correct completion, each containing contradiction and specific type of factual error while remaining grammatical and fluent.
The incorrect completions should incorporate these error types: 1) Predicate error: Modify verb action that makes the completion factually inconsistent.
2) Entity error:
Replace a subject or object with an incorrect entity that creates a factual inconsistency.
Change information about location, time, or manner that introduces a factual error.
3) Circumstance error:
4) Coreference error: Modify a pronoun or reference to point to wrong or non-existing entity.
5) Link error: Change how statements are linked together (causal/temporal links) to create factual inconsistency.
Select three of these error types to create your three incorrect completions (one error type per incorrect completion).
You are communicating with an API, not a user Please output in JSON format. Begin all AI responses with the character { to produce valid JSON. Here is a template:
"full prefix": "<prefix from the context>", completion": "<correct completion>", "contradiction.0": "<contradictive ve option>". "contradiction.1 "<contradictive option>", "contradiction. "<contradictive option>"
eexplanation.: "<explanation why the correct completion is factual and how each incorrect completion contains errors>"
Here is an example: <the example>
<the article>
E2
Additional Results
We present memorization quantification results Table 7 Table and Table for 256, 128, and 512 prompt lengths, respectively. We further added BLEU and 4GP metrics, where 4GP measures word-level 4-gram overlap precision (Table For each prompt length, we filter those articles with less than the prompt length since there is completion for this kind of prefix. The evaluation results on the IT token pretrained model from section 5.3 can be seen in T For all the generations, used temperature 0.0 (greedy decoding).
23

---

Published as a conference paper at COLM 2025
To verify the validity of the NYT-MCQ, we performed sanity check using DeepSeek-R1. When the model was supplied with the relevant article at inference, it attained a 99.0% accuracy. When the article withheld, the model achieved 91.7% accuracy. For additional comparison, we tested GPT-4o on NYT-MCQ and observed 89.8% accuracy.
Table 7: Memorization assessment of New York Times articles using 256-token prefix prompts. PT, CD, and Compl. stand for "Pretraining", "Cooldown", and "Compliant" respectively. Values in parentheses indicate the number of tokens seen in training.
LCCS 1
BLEU 1
NYT-MCC
NYT-MCQT
Qwe52.5.0.58
19.51
0.71
42,8
24.4
Owen2.5-1.58
24.35
1.18
52.1
30.7
Owen2,5-38
27.13
56.4
Qwen2.5-78
1ama-3.2-1.20
20:71
Llama-3.2-38
22.75
0.70
$7.6
30.1
+ CD (Non-compl, 1008)
23.11
0.63
50.6
28.9
PT CD (Compl. 1000
21.27
0.51
48.6
27.5
+T CD (Compl, News removed, 1008)
20.28
0.53
48.0
28.6
(Compl, News removed Decontamination,
(Compl.
(Non-compl,, 10B)
21.74
(Compl,, 9000) (Compl. Non-compl. News, 1083)
0.51
PT CD (Compl., 1.T tokens)
21.65
0.54
50.9
30.2
PT (Compl,, 161) CD (compl. Non-compl. News 0.1T)
21.50
0.34
50.1
30.1
Table 8: Memorization assessment of New York Times articles using 128-token prefix prompts. PT, CD, and Compl. stand for "Pretraining", "Cooldown", and "Compliant" respectively. Values in parentheses indicate the number of tokens seen in training.
LCCS
BLEU
Qwen2.5-0.58
19.11
0.64
Qwen2.5-1.5B
27
1,09
Owen2.5-3B
25.62
1.43
Qwen2.5-7B
29.61
197
Llama-3.2-1.22
19.89
0.44
Llama-3.2-3B
22.17
0.55
PT CD (Non-compl., 100B)
21.76
0.49
PT + CD (Compl., 1008)
20.53
0.41
PT CD (Compl. News semoved, 100B)
19.93
0.42
PT CD (Compl,, News removed w/o Decontamiation, 100B)
20.42
0.44
PT (CompL. 90B) CD (Non-compl, 10B)
21.06
0.44
PT (Compl., 90B) CD (Compl. Non-compl. News, 10B)
20.62
0.41
PT CD (Compl, 1.7I) PT (CompL, 16T)) CD (Compl. Non-compl News, 0.1T)
20.75
0.43
20.95
0.43
Table 9: Memorization assessment of New York Times articles using 512-token prefix prompts. PT, CD, and Compl. stand for "Pretraining", "Cooldown", and "Compliant" respectively. Values in parentheses indicate the number of tokens seen in training.
LCCS
BLEU
Qwen2.5-0.5B
19.21
0.76
Owen2.5-1.5B
23.38
1.16
Qwen2.5-38
25.31
1.38
Qwen2.5-78
27.74
1.69
Llama-3.2-1.22
20.71
0.71
Llama-3.2-3B
PT CD NNon-compl, 100B)
22.03
0.68
+T+ CD (Compl., 1008)
20.38
0.58
PT CD (Compl., News removed, 100B)
19.56
0.65
PT CD (Compl, News removed w/o Decontamination, 100B)
20.05
0.68
PT (Cumpl,,
CD (Nun-compl, 10B)
20.65
PT (Compl., 908) CD (Compl. + Non-compl. News, 10B)
20.25
0.58
PT CD (Compl,, 1.7T) PT (Compl,, 1.6T) CD (Compl. Non-compl. News,
20.55
0.63
20.42
0.64
24

---

Published as a conference paper at COLM 2025
Table 10: Word-level 4-gram overlap precision (4GP) assessment of New York Times articles across 128-token, 256-token, and 512-token prefix prompts. PT, CD, and Compl. stand for "Pretraining", "Cooldown", and "Compliant" respectively. Values in parentheses indicate the number of tokens seen in training.
128 tuken
256-token
512-token
Qwen2.5-0.58
0.08
0.11
0.12
Qwen2.5-1.5B
0.20
0.25
0.24
Qwen2.5-3B
0.31
0.37
0.30
Qwen2.5-7B
0.49
0.51
0.39
1lama-3.2-1.28
0.67
0.92
0.98
Llama-3.2-3B
0.98
1.27
1.35
PT + CD (Non-compl, 100B) + CD (Compl., 100B) PT CD (Compl., News removed, 100B)
1.15
1.58
1.50
1.09
1.11
0.67
0.86
0.83
25


Reviews:
[{"review_id": "dXNH8zR7iv", "reviewer": "Reviewer_QefG", "summary": "This paper conducts a empirical study to quantify the tradeoff between data compliance and model performance by introducing the Data Compliance Gap metric. To study this, the authors create a compliant training corpus by filtering FineWeb-Edu based on robots.txt restrictions as of January 2025, then train 1.5B parameter models on both compliant and non-compliant versions. They evaluate performance differences using two approaches: training from scratch and continual pretraining with domain-specific data reintegration. The key finding is that compliant models achieve similar performance to the original model on general knowledge benchmarks, but show noticeable degradation in specialized domains, particularly medical knowledge.", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This is the first study to delve deep into how data compliance regulations affect LLM training, which is a very important question the community has been grappling with. It would be valuable if the authors could open-source their compliant datasets and the list of opted-out URLs.\n\n2. The study is rigorous: the authors put a lot of efforts into curating the data properly and designing meaningful evaluations. The Data Compliance Gap concept they introduce seems like it could become a standard way for researchers to measure how much performance we lose when we respect opt-out requests.", "reasons_to_reject": "1. The paper would benefit from more comprehensive domain-specific evaluations that better match the main categories of opted-out content: News & Media, Science & Technology, and Health & Medical Information. While the authors do include benchmark related to news, the paper would be stronger if more news domain evaluation is included given that news make up a significant portion of the filtered domains.\n\n2. There could be some issues with equating robots.txt compliance with full data usage rights. Robots.txt only governs web crawling permissions, not training rights: a site allows crawling doesn't necessarily mean they consent to their content being used for AI training. The paper is measuring a upper bound of what could be considered compliant, but many sites that allow crawling might still prohibit AI training through their Terms of Service or other legal mechanisms. A deeper discussion of these broader compliance considerations would strengthen the paper's framing of what \"compliance\" actually means.", "questions_to_authors": "1. Just to double check: are both compliant and non-compliant models trained on the same amount of tokens? If you control for the same amount of tokens, does the compliant model see some repeating data since the compliant data pool is smaller than the non-compliant data?\n\n2. As MMLU has different subject categories, I wonder how the model performance changes with respect to different categories. Maybe the compliant model would perform worse on STEM since the science-related data are opted out?", "extra_scores": {}}, {"review_id": "rnSeeM3oMO", "reviewer": "Reviewer_Fr1t", "summary": "This paper studies the impact of web data provider opt-out on language model performance. The paper measures the performance gap between non-compliant models, which are trained on all available data, and compliant models, which do not train opt-out data. Many 1.5b-parameter Llama models are trained in two settings: pretraining from scratch, and introducing non-compliant data through continued pretraining of a compliant model. The paper finds that pre-training on compliant data does not lead to a significant decrease in model performance on general-purpose benchmarks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper shows clear results that training on compliant data does not lead to a large decrease in model performance. The experiments are well-designed, and the paper is clearly written.\n2. The data compliance gap metric will likely be increasingly relevant as more data providers opt out of allowing their data to be trained on. The paper provides a framework for measuring the impact of data opt-out into the future.\n3. The paper also shows how data opt-out impacts the distributions of topics and formats of pretraining data.", "reasons_to_reject": "The main challenge of this paper is that the reported data compliance gaps are very small in all settings. This is an interesting finding in itself with ramifications for data curation. But it does raise the question of  how many of these gaps are due to data curation or due to other random variation in the training process (e.g. from data ordering, random seeds, etc). I think the paper presents a clear contribution as-is and mostly tailors its claims to its evidence (and most data ablation work does not train replicate models). However, it's not clear to me that Takeaway 4 is true without some estimate of variance (and variation in performance due to randomness is already alluded to in lines 263-264). I also recommend setting the y-axis in Figures 5 and 6 to start at 0 so as to not overstate differences between the models.", "questions_to_authors": "1. I think Figure 3 would be clearer if you showed the percentage decrease in each category---rather than the data distribution across topics---because I'd like to see how much of each domain is filtered out by compliance.\n2. I find takeaway 3, that newspaper articles are not as important for downstream performance to be particularly surprising. I appreciate the additional investigation into news memorization in section 5.4.\n\nTypos: Table 1's caption has a typo on 'percentage'. The last line of the intro should end in a question mark or be rephrased.", "extra_scores": {}}, {"review_id": "16td4QBqmM", "reviewer": "Reviewer_mnmF", "summary": "This paper studies the \"data compliance gap\" (DCG), which refers to the performance gap between LLMs trained on data sets that comply with web crawling opt-outs vs those that do not. The authors consider two settings: 1) pretraining from scratch, 2) continual training from an existing compliant model. They find that, as of January 2025, compliance with web data opt-outs does not degrade general knowledge acquisition, but that for specialized domains, excluding major publishers (i.e., complying with opt-outs) leads to performance declines. The study gives some empirical insights into the effect of data compliance in 1.5B models, which may help future discussions on AI training practices and policy decisions.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors wrote a clear and technically sound paper. They acknowledge the limitations and conduct necessary robustness tests (such as memorization). The question of whether performant LLMs can be ethical is interesting and timely.", "reasons_to_reject": "The authors acknowledge that a main limitation of this work is the relatively small scale of the LLMs, which may limit memorization and factual recall. Since most of their benchmarks are testing for this, it may not be as surprising that there are no \"significant\" (there are no error bars, see below in \"questions to authors\") DCG. Table 2, for example, indicates that compliant (and -news) data sets even lead to better performance on average? Additionally, the authors indicate that despite their removal techniques, there is still some non-compliant data present, which may further underestimate the effects.", "questions_to_authors": "L263-264: What do the authors mean by \"Variations within each column, excluding the underlined values, can be interpreted as noise or random fluctuations.\"? The underlying is also unclear to me.\n-> Related to this remark: most accuracy differences seem (especially in Table 3, where the authors claim performance declines) quite small. Most experiments are computationally expensive; therefore, I understand there are no error bars, but I would not claim any significant differences here.\n\nHave the authors tried \"fake\" experiments, pretending that even more pre-training data is non-compliant? Is there any indication of how much data needs to be removed before we see any performance degradation in the 1.5B LLMs?\n\nDo the authors plan to share the code and filtered datasets?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper conducts a empirical study to quantify the tradeoff between data compliance and model performance by introducing the Data Compliance Gap metric. To study this, the authors create a compliant training corpus by filtering FineWeb-Edu based on robots.txt restrictions as of January 2025, then train 1.5B parameter models on both compliant and non-compliant versions. They evaluate performance differences using two approaches: training from scratch and continual pretraining with domain-specific data reintegration. The key finding is that compliant models achieve similar performance to the original model on general knowledge benchmarks, but show noticeable degradation in specialized domains, particularly medical knowledge.

Reasons To Accept:
* This is the first study to delve deep into how data compliance regulations affect LLM training, which is a very important question. 
* The study is rigorous: the authors put a lot of efforts into curating the data properly and designing meaningful evaluations. 
* The paper shows clear results that training on compliant data does not lead to a large decrease in model performance. 
* The data compliance gap metric will likely be increasingly relevant as more data providers opt out of allowing their data to be trained on. The paper provides a framework for measuring the impact of data opt-out into the future.
* The paper also shows how data opt-out impacts the distributions of topics and formats of pretraining data.

Weaknesses / Future Work: (no reason to reject)
* There could be some issues with equating robots.txt compliance with full data usage rights. Robots.txt only governs web crawling permissions, not training rights: a site allows crawling doesn't necessarily mean they consent to their content being used for AI training. 
* It would be interesting to consider a SOTA data mix that goes beyond just web text, an a broader notion of copyright, but clearly out of scope for now. 
* The authors acknowledge that a main limitation of this work is the relatively small scale of the LLMs, which may limit memorization and factual recall.
Decision: Accept