<figure><img src="image_1.png" /></figure>

HAL
open science

# Web Crawler Restrictions, AI Training Datasets & Political Biases

Paul Bouchaud, Pedro Ramaciotti

<figure><img src="image_2.png" /></figure>

## To cite this version:

Paul Bouchaud, Pedro Ramaciotti. Web Crawler Restrictions, AI Training Datasets & Political Biases. 2025. hal-05302425

HAL Id: hal-05302425

https://hal.science/hal-05302425v1

Preprint submitted on 9 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Web Crawler Restrictions, AI Training Datasets
& Political Biases

Paul Bouchaud¹,², Pedro Ramaciotti¹,²,³

¹Complex Systems Institute of Paris Ile-de-France CNRS, Paris, France.

²médialab, Sciences Po, Paris, France.

³Learning Planet Institute, CY Cergy Paris University, Paris, France.

**Abstract**

Large language models rely on web-scraped text for training; con-
currently, content creators are increasingly blocking AI crawlers to
retain control over their data. We analyze crawler restrictions across
the top one million most-visited websites since 2023 and examine
their potential downstream effects on training data composition.
Our analysis reveals growing restrictions, with blocking patterns
varying by website popularity and content type. A quarter of the
top thousand websites restrict AI crawlers, decreasing to one-tenth
across the broader top million. Content type matters significantly:
34.2% of news outlets disallow OpenAI's GPTBot, rising to 55%
for outlets with high factual reporting. Additionally, outlets with
neutral political positions impose the strongest restrictions (58%),
whereas hyperpartisan websites and those with low factual report-
ing impose fewer restrictions —only 4.1% of right-leaning outlets
block access to OpenAI. Our findings suggest that heterogeneous
blocking patterns may skew training datasets toward low-quality
or polarized content, potentially affecting the capabilities of models
served by prominent AI-as-a-Service providers.

CCS Concepts

• Computing methodologies → Natural language processing;

• Information systems → Web crawling; Web mining; • Applied computing → Law, social and behavioral sciences.

Keywords

large language models, web scraping, dataset bias, news media,
political bias, CommonCrawl

1 Introduction

The training of large language models that powered the recent
surge in AI chatbots crucially relied on massive text corpora [5, 37]
—amounting to trillions of words scraped from billions of web pages
[8]. This large-scale data collection, and its subsequent commer-
cial exploitation, has raised fundamental copyright concerns and
prompted content creators to resist having their work used without
compensation or attribution [13, 19, 34]. Following the training of
their flagship model, OpenAI announced in August 2023 support
for the Robots Exclusion Protocol [24], a standard initiated in the
late 1990s that allows webmasters to specify which web crawlers
can access their content by placing directives in a robots.txt file
at the root of their websites. Following this development, several
organizations and content creators called for blocking OpenAI's
and other AI providers' crawlers [16, 17, 35] as a means to exert
greater control over the use of their data.

Meanwhile, since the emergence of natural language processing,
a wealth of work has addressed how biases in training corpora

might translate to biased outcomes. Gender and racial biases [6, 21, 31] in word embedding models such as word2vec [22] and GloVe [28] and subsequent sentence-level embeddings such as BERT [9], were traceable to biases in the underlying training corpus [4, 12]. However, the increased size of modern model representations and datasets, coupled with opacity regarding training data composition, impedes such investigations. Nonetheless, one resource remains central to most AI training efforts [1, 5, 37]: CommonCrawl [8].

CommonCrawl is a petabyte-large archive of web crawl data col-
lected since 2008 [1, 8]. Due to its size and coverage, it is massively
used in the training of AI generative models, in particular through
derivative datasets. Indeed, CommonCrawl collectors have taken
a stated aim not to curate or polish the content of the collection,
capturing a fraction of the web as it is [1]. Consequently, large
language model builders rely on filtered versions of CommonCrawl
such as C4 [29], RefineWeb [27], or FineWeb [26] that typically
remove duplicates, HTML boilerplate, and subsample certain lan-
guages [38]. Some curation rules apply normative judgments by
filtering out content deemed "undesirable" [1, 20, 26] and have been
shown to disproportionately exclude text from and about minorities
[11], potentially skewing the representation in models trained on
such filtered datasets.

In addition to content filtering, another potential source of bias
may arise directly from the collection process itself, as AI crawlers
face growing restrictions [10, 16]. If AI model builders do not prop-
erly address the skew induced by the protection measures content
creators adopt against exploitative practices, the trained AI models
may inherit systematic biases that may compromise their fairness
and representativeness. In this work, we quantify and characterize
the adoption of AI crawler restrictions since 2023 across the one mil-
lion most visited websites worldwide, and evaluate the downstream
effects on CommonCrawl and its derivative training datasets.

Our analysis reveals growing AI crawler restrictions since 2023,
with blocking patterns varying by website popularity and content
type. Over 25% of the top thousand most visited websites restrict
AI crawlers, but this proportion decreases to 10% when considering
the broader top million websites. Content type also plays a signifi-
cant role: while only 4% of shopping websites disallow OpenAI's
GPTBot, 34.2% of news outlets do, rising to 55% for outlets with high
factual reporting. Outlets with neutral political positions impose
the strongest restrictions, with 58% disallowing OpenAI's crawler,
whereas hyperpartisan websites and those with low factual report-
ing impose fewer restrictions—only 4.1% of right-leaning outlets
block access.

Additionally, we observe that the likelihood for a website to
impose AI crawler restrictions decreases as its audience ideology
diverges from the political center, with websites on both the far-
left and far-right ends of the ideological spectrum imposing fewer
AI crawler restrictions than moderate sources. Paralleling the increased restrictions on AI crawlers imposed by moderate sources, we observe their relative decline within training datasets in favor of hyperpartisan material. Through textual analysis of political content, we characterize the potential biases this shift may introduce, revealing over-representation of politically charged terminology in available training data.

These findings call for acute consideration in the curation of future training datasets to account for skewed restrictions in raw data collection, as the systematic withdrawal of high-quality news content, coupled with the relative over-representation of politically hyperpartisan sources, demands active intervention by model developers to prevent embedding polarization into AI systems.

## 2 Datasets

In this article, we examine restrictions on AI crawlers deployed by the most popular websites worldwide. To this end, we rely on Google Chrome's analytics to identify popular websites and on Cloudflare Radar to categorize webpage content. Additionally, we combine a large-scale collection of robots.txt directives left by webmasters as of September 2025 and historical data since 2023 from CommonCrawl snapshots.

**Websites Dataset (CrUX).** As a window into the web, we first consider in this study the set of the one million most visited websites worldwide accounting, in August 2025, for 95% of global traffic on Google Chrome in terms of page loads [32]. This dataset stems from the Chrome User Experience Report (CrUX) and represents aggregate analytics from hundreds of millions of Chrome users who have usage statistic reporting enabled, opted in to history syncing, and have no history sync passphrase set. Ruth et al. [33] performed comparative analysis of top lists, including the (deprecated) Alexa list and Cisco's Umbrella, and showcased the high coverage of the CrUX dataset. In addition to the mere list, we will leverage the popularity buckets disclosed in CrUX (top 1K, 10K, 100K, 1M). Unless stated otherwise, all analyses are performed over this set of websites or subsets thereof.

**Content Categorization.** We enrich this list of one million websites with a characterization of their content. To this end, we leverage, akin to Ruth et al [32], Cloudflare's Radar APIs [7]. Cloudflare operates, beyond DDoS protection, a DNS and parental control service whose filters allow parents or schools to block domains classified as "Adult Themes" or more generally falling under CIPA filters. To enable such filtering, Cloudflare characterized websites content —thus extending beyond Cloudflare-protected websites—, we then rely on this categorization. For simplicity, we aggregate Cloudflare's granular content categories into "supercategories" — we will drop the prefix for discussion fluidity— relying on the taxonomy, reported in the Appendix, curated, and assessed in [32].

To validate Cloudflare's categorization, we extracted HTML meta keywords from 661k successfully crawled websites and identified overrepresented terms per category using chi-squared statistics. For instance, pages categorized as *Shopping & Auctions* tend to have keywords such as: *silver, watches, jewellery*, while those categorized as *Adult Themes* have: *porn, telegram, sex*; we display the top words of each category in the Appendix.

**Robots.txt Collection.** Additionally, we focus on the directives set by webmasters to crawlers scanning the web, indicating to them which portions of the website, if any, they are permitted to access. In September 2025, for each of the top one million websites we fetched the robots.txt file deployed, if any, at the root of their (sub)domains. Additionally, to go beyond a mere snapshot, we relied on CommonCrawl's periodic releases [8] of very-large-scale crawls of billions of webpages. In particular, this corpus contains timestamped robots.txt files, offering unparalleled chronological insights. However, rather than being a "copy of the web", only a fraction of it is crawled [1, 8]. Within the top one million websites, on average over releases, 409k valid robots.txt files were collected.

## 3 Overview

### 3.1 Content Category

We first aim to provide an overview of the web cross two dimensions: topic categories and languages, updating the description made by Ruth et al. [32] in early 2022. We observe that half of the pages are written primarily in English (50.5%), followed by Japanese (6.2%) and Spanish (6.0%), see figure in Appendix. Second, we observe in Figure 1 that *Shopping & Auctions* websites account for the largest fraction at 18.6%. *Business & Economy, Education, Entertainment and Technology* follow, representing between 7.6% and 10.5%. Aligned with [32], we observe that *Adult* content shows significant variation across popularity tiers. Within the top thousand most visited websites worldwide, 20.3% are *Adult* websites, compared to 15.0% in the top 10k and 3.7% in the top 1M.

<figure><img src="image_4.png" /><figcaption>**Figure 1:** Distribution of content categories within the one million most visited websites worldwide (CrUX).</figcaption></figure>

### 3.2 Crawling Restrictions

To prevent the content of their websites from being automatically indexed, scraped, and otherwise programmatically used without the website owner's agreement, multiple mechanisms have been developed to prevent such crawling.
In the top one million websites, we observe that 55.6% of them deployed so-called robots.txt at the root of their (sub)domains.

Alternative proposals to prevent the crawling to feed AI model
such as adding noai in the "robots" meta tag of the page appeared
on less than 0.2% of analyzed websites, similarly [10] observed low
adoption of the "TDM Reservation Protocol". In addition to such
passive voluntary systems, Liu et al. [16] explored active measures
whereby websites restrict the activity of some public crawlers, for
instance by blocking traffic coming from IP addresses known to
be operated by crawlers. Nevertheless, robots.txt is the de facto
standard to specify directives to web crawlers and as such will be
further examined in the following section.

4 Robots Exclusion Protocol

With the advent of generative artificial intelligence, the datasets
on which these models are trained have garnered attention and
warranted renewed inspection of the Robots Exclusion Protocol [10,
14, 16]. We first consider a large-scale characterization of directives
left by webmasters for crawlers before focusing specifically on
those used by AI model providers.

4.1 Overview

The robots.txt can specify rules applying uniformly to all crawlers
using the catch-all directive (Allow/Disallow for User-Agent: *)
or by specifying targeted rules for particular crawlers; for instance,
allowing Google Search's crawler to index certain pages while
blocking all other robots. Within our collected sample, 64.1% of
robots.txt files used only the catch-all rules.

Among crawlers named in robots.txt files of websites specify-
ing granular directives, we can delineate the following categories:

*   **SEO & Marketing**: MJ12bot (36.5%), AhrefsBot (36.0%), AhrefsSiteAudit (23.5%), and SemrushBot (13.9%)

*   **Search Engines**: Nutch (25.1%), GoogleBot (18.1%), CCBot (14.7%), Yandex (10.8%), and bingbot (6.8%)

*   **Social Media**: Pinterest (23.6%), FacebookBot (5.5%), and facebookexternalhit (5.0%)

*   **Generative AI Training**: GPTBot (19.9%), ClaudeBot (14.7%), Google-Extended (14.4%), Applebot-Extended (10.2%), and meta-externalagent (8.8%)

*   **AI Search & Chatbots**: ChatGPT-User (7.5%), PerplexityBot (6.5%), Claude-Web (5.2%), OAI-SearchBot (4.9%)

Alongside crawlers indexing the web for analytics, search en-
gines, or generating social media previews, we observe AI-related
ones. Granular distinctions emerge between crawlers collecting
data for model training, e.g., GPTBot, ClaudeBot, those power-
ing AI search engines, e.g., OAI-SearchBot, PerplexityBot, and
those responding to individual user queries, e.g., ChatGPT-User,
Claude-Web.

Importantly, the Robot Exclusion Protocol relies on good faith
and voluntary compliance by the organizations operating crawlers.
In 2025, Liu et al. [16] and Kim et al. [15] showed that AI crawlers op-
erated by OpenAI, Anthropic, Meta, and CommonCrawl respected
websites' robots.txt directives as stated. Yet, it should be noticed
that the scope of these directives can be ambiguous. For instance,
if a chatbot provider trains on user discussions without further
curation, content from a website that blocked GPTBot but allowed

(or, due to tacit consent, did not block) ChatGPT-User could still be
used for training as it may appear in user chats.

4.2 AI Crawlers

2025 Snapshot. Focusing on the most popular AI chatbots [36], we observe that, as of August/September 2025 and among the top one million websites, OpenAI's GPTBot is fully blocked on 10.6% of websites, Anthropic's ClaudeBot on 9.1%, and Google's Google-Extended on 8.9%. Additionally, 9.5% disallow CCBot, the crawler used by CommonCrawl, whose public datasets are extensively leveraged in training AI models [1, 5, 37].

Alternatively, AI-powered search engine crawlers are more fre-
quently permitted, with only 6.0% of websites disallowing Perplex-
ity's crawler, and 6.5% and 6.1% blocking OpenAI's and Anthropic's
user-initiated crawlers (ChatGPT-User & Claude-Web). As a base-
line, 4.0% of examined websites blocked Google Search's indexing
crawler GoogleBot at their root.

We observed instances of websites, particularly news outlets,
that disallowed popular AI crawlers except one due to licensing
agreements [23, 25]. Examples include the Wall Street Journal, New
York Post, The Times, Bild, Washington Post, and Le Monde, all
allowing OpenAI's crawlers while explicitly disallowing others.

<figure><img src="image_5.png" /><figcaption>Figure 2: Fraction of webpages disallowing specific web crawlers on their domain over time, underlying data from CommonCrawl releases.</figcaption></figure>

*Chronology*. Relying on CommonCrawl's periodic releases [8], we can reconstruct the evolution of robots.txt since January 2023. We observe in Figure 2 that while the fraction of webpages blocking Google Search's indexing crawler GoogleBot remains stable over the period 2023-2025, the fraction blocking OpenAI's GPTBot rises after its announcement in summer 2023 [24]. Following their public launch, other AI crawlers such as ClaudeBot and PerplexityBot experienced growing disallowance rates in robots.txt files throughout our observation window.

*Restrictions per category.* Finally, we examine how robots exclusion directives are used across different categories of website content and popularity. Relying on Cloudflare categorization, Figure 3 reveals that disallowance of OpenAI's GPTBot varies by topic across websites, with rates exceeding 14% for Entertainment and News & Politics, declining to 9.0% for Education, and reaching only 4.0% for Shopping & Auctions. Additionally, we observe that more popular websites are more likely to disallow OpenAI's GPTBot: while 13.4% of websites in the top 100k block it, this proportion nearly doubles to 25.2% among the top thousand most visited websites worldwide.
<figure><img src="image_6.png" /><figcaption>Figure 3: Fraction of webpages disallowing specific web crawlers on their domain, segmented by: (A) Content category per Cloudflare's classification, (B) CrUX popularity bucket. Error bars represent standard deviations over 100 bootstrap with replacement over websites.</figcaption></figure>

# 5 CommonCrawl & Derivatives

Since the observed directives that restrict crawlers from collecting training data for generative AI models are not uniformly distributed across the web, and because biases in training datasets can lead to discriminatory outputs in AI models [2, 3, 12], we evaluate how the growing restriction of online resources impacts CommonCrawl and derivative datasets.

Indeed, refined datasets of CommonCrawl have emerged as fundamental in the development of recent large language models, representing over 80% of training corpora of Llama [37] and GPT-3 [5]. We will in particular consider two subsets of CommonCrawl: FineWeb and FineWeb-Edu [26]. FineWeb is a cleaned and deduplicated English subset of CommonCrawl, while FineWeb-Edu is a subset of FineWeb containing English text with high “educational quality” [26]. Specifically, FineWeb-Edu was created by first scoring half a million FineWeb text snippets using a large language model, then by training a regression head over text embedding representations to extend this scoring to hundreds of millions of text snippets. Large language models trained on FineWeb-Edu were shown by HuggingFace in June 2024 to “outperform all openly accessible web-based datasets on a number of reasoning- and knowledge-intensive benchmarks such as MMLU” [26].

## 5.1 CommonCrawl

First of all, to provide an overview of CommonCrawl's content, we consider a random sample of one billion tokens from its latest dump (CC-MAIN-2025-38) and analyze the language and content of the underlying crawled pages, relying on Cloudflare categorization [7]. Overall, aside from an under-representation of Adult content compared to CrUX pages and an over-representation of pages in German, CommonCrawl appears analogous to the web in terms of language and content category; see the distribution in the Appendix.

## 5.2 News Outlets

Beyond this overall characterization, we now focus on news outlets. We relied on political skew and factual reporting assessments from Media Bias/Fact Check<fnref n="1" />. We collected the robots.txt files of 3 668 annotated news outlets. Overall, 34.2% of media outlets disallowed OpenAI's GPTBot, with strong disparities across political leaning and factual reporting. As displayed in Figure 4, 58.0%

of neutral outlets block OpenAI's GPTBot, compared to 19.6% of left-leaning outlets and 4.1% of right-leaning outlets. Regarding factual reporting, 55.4% of outlets with high factual reporting block OpenAI crawlers, compared to 8.4% of those with mixed factual reporting and 3.7% of those with low factual reporting. Additionally, we observe that across political leaning and factual reporting categories, the fraction of websites blocking training-related crawlers (GPTBot) is higher than those used to answer specific user requests (ChatGPT-User), as displayed in Figure 4.

<figure><img src="image_7.png" /><figcaption>Figure 4: Fraction of websites disallowing OpenAI's GPTBot and ChatGPT-User, as a function of their political skew and factual reporting assessment by MBFC.</figcaption></figure>

In parallel, comparing the first Fineweb release from 2020 with that from 2025 (both larger than 170 billion tokens), the fraction of tokens stemming from sources evaluated as “high factual reporting” by MBFC decreased from 0.46% to 0.26% (a 41.3% relative decrease). Similarly, the fraction of hyperpartisan right content increased from 19.3% to 24.8% (a 28.8% relative increase), and of hyperpartisan left content rose from 18.9% to 22.4% (an 18.6% relative increase).

## 5.3 Educational Content

Similarly, for educational resources, we examine FineWeb-Edu, a subset of FineWeb curated for its “educational quality” [26]. Importantly, each snippet in FineWeb-Edu is associated with the exact URL from which the content was extracted and its collection date by CommonCrawl's CCBot. This URL-level metadata enables us to assess robots.txt directives at the specific URL level rather than merely at the broader domain level.

For each release since 2015, we randomly sampled 25k URLs from domains in the top one million. We observe that over 20% of the tokens in the FineWeb-Edu dataset collected prior to 2024 originated from domains that, as of August 2025, block CommonCrawl bots from accessing those very resources; see Figure in Appendix.

We preemptively seek to address the hypothesis that such restrictions on high-quality educational content may lead to overall decay in training dataset educational quality. For every FineWeb release since 2015, we randomly sample 10k text snippets and assess their “educational quality” using the language model used by HuggingFace to curate FineWeb-Edu [26]. The output is a score between 0 (not educational) and 5 (highly educational). We do not observe quality decay. Instead, the fraction of FineWeb (in tokens) scoring at least 3 (threshold to be included in FineWeb-Edu) grew from 8.3% on average prior to 2022 to 9.7% since 2022; see Figure in Appendix.

<footnote n="1">¹https://mediabiasfactcheck.com</footnote>
## 5.4 Political Content

Finally, we explore the directives left by webmasters to crawlers as a function of websites' political leaning. To assess such leaning, we rely on the large-scale audience characterization performed by Robertson et al. [30]. Specifically, they linked half a million Twitter accounts to US voter registration records and collected the URLs these accounts shared on the platform. The leaning of each domain was then assessed as the fraction of registered Democrats versus Republicans who had shared a URL from that domain, resulting in a continuous scale ranging from -1 to 1. A score of -1 (respectively +1) indicates that the domain was shared exclusively by Democrats (respectively Republicans), while a domain receives a score of 0 if and only if it was shared by equal proportions of Democrats and Republicans. This scale has been shown to align with other estimates derived from Facebook data, expert raters, and community assessments [30].

<figure><img src="image_8.png" /><figcaption>**Figure 5:** Fraction of websites disallowing GPTBot as a function of their audience ideological leaning, as characterized by Robertson et al. [30]. Error bars represent Clopper-Pearson 95% confidence intervals. The solid curve shows the fitted quadratic logistic regression with 95% confidence band (shaded region). The fraction disallowing GoogleBot is shown as a baseline.</figcaption></figure>

*Robot Exclusion.* We collected the robots.txt files of 15.2k domains ideologically scaled by Robertson et al. [30] and display in Figure 5 the fraction of websites that disallow OpenAI's GPTBot as a function of their audience ideological leaning, with GoogleBot as a baseline.

We observe that domains shared in similar proportions by Democrats and Republicans —those with low absolute scores— disallow OpenAI's GPTBot significantly more frequently than ideologically skewed domains. While 18.5% of balanced domains (absolute score<fnref n="2" /> below 0.5) block OpenAI's crawler, this fraction plummets to 7.3% and 7.0% for hyperpartisan left (≤ −0.5) and hyperpartisan right (≥ 0.5) domains, respectively. Similar results holds for CommonCrawl's CCBot. In contrast, in September 2022, the fraction of websites disallowing CommonCrawl's CCBot was equivalent between outlets with balanced and skewed audiences, at 2.6% and 3.0% respectively.

*Representation in FineWeb.* Without inferring causation, we observe that the fraction of tokens extracted from hyperpartisan domains in FineWeb [26] increased from 27.7% to 31.0% between 2020 and 2025 (a 12.1% relative increase).

To characterize potential biases that may arise from increased hyperpartisan content representation in FineWeb, we analyze word co-occurrence patterns in text authored by those websites. Our focus on co-occurrence rather than individual word frequencies aligns with large language model attention mechanisms, which are designed to capture contextual relationships between words.

Specifically, from FineWeb latest release (CC-MAIN-2025-26) we filter text snippets from domains with characterized political leanings [30]. We split them into sentences and randomly sample five million sentences for each group: hyperpartisan left, hyperpartisan right, and balanced domains. Subsequently, we compute the co-occurrence within the same sentence of each word pair and apply chi-square statistics to identify word pairs that co-occur in the same sentence significantly more often in hyperpartisan text than in balanced content.

Among hyperpartisan left-leaning text, the following word pairs are among the most overrepresented compared to balanced text (arbitrary word order within pairs): (Climate; Change), (Human; Rights), (Grants; NIH), (Union; Workers), (Student; Yale), (Indigenous; Communities), (Women; Rights), (Women; Abortion), (People; Color), (Jewish; Community).

Similarly, among hyperpartisan right-leaning text: (President; Trump), (President; Biden), (AFA; Family), (Jesus; God), (Donald; Trump), (Christ; Jesus), (Fox; News), (Pro; Life), (Faith; God), (Supreme; Court).

For comparison purposes, we identify words that co-occur more frequently in hyperpartisan text alongside terms referring to potentially discriminated population:

* **Young:** (Left) People, Women, Black; (Right) God, Church, Men

* **Old:** (Left) Black, Scholar, Jewish; (Right) Testament, God, Biden

* **Woman:** (Left) Black, White, Trans; (Right) God, Abortion, Man

* **Man:** (Left) Black, White, Trump; (Right) God, Jesus, Trump

* **Transgender:** (Left) People, LGBTQ, Rights; (Right) Children, Women, School

* **LGBTQ:** (Left) Tennessee, Rhetoric, Safe; (Right) Children, Agenda, Christian

* **Immigrant:** (Left) Republicans, Border, Detention; (Right) Illegal, Border, Trump

* **Muslim:** (Left) Hindu, United, Americans; (Right) Islamic, World, Brotherhood

## 6 Discussion

The web's scale and diversity have established it as a foundational resource for training the AI models that power the development of foundation models. However, our analysis of the one million most visited websites worldwide reveals substantial and growing adoption of AI crawler restrictions, with over 25% of the top thousand most visited websites blocking OpenAI's web crawler. Importantly, these restrictions are not uniformly distributed across the web;

<footnote n="2">²Threshold of ±0.5 set to align with AdFontes's labels.</footnote>
instead, they vary significantly by website popularity and content type, leading to potential skewing in subsequent web-crawled datasets. High-quality factual news outlets exhibit particularly pronounced restrictions, with over 55% blocking of OpenAI's GPTBot.

Furthermore, we observe that outlets with high factual reporting and neutral political positions impose the strongest restrictions on OpenAI's crawlers, whereas hyperpartisan websites and outlets with low factual reporting impose fewer restrictions. Quantitatively, while 58% of neutral outlets restrict OpenAI's GPTBot only 4.1% of right-leaning outlets do so.

This pattern extends beyond news media to websites shared online more generally. Using Robertson et al.'s characterization of over 15k websites shared on Twitter [30], we find that hyperpartisan websites—both left- and right-leaning—impose significantly fewer restrictions than politically balanced sources. While 18.5% of balanced domains block OpenAI's GPTBot, only 7.3% and 7.0% of hyperpartisan left and right domains do so, respectively. Correspondingly, the representation of hyperpartisan domains in CommonCrawl-derived training datasets increased by 54.6% (left) and 43.5% (right) between pre-2023 and post-2023 periods, while balanced domains decreased by 11.3%. Textual analysis reveals overrepresentation of politically charged, civic, and religious terminology in hyperpartisan content, with distinct semantic associations emerging for terms referring to marginalized groups.

These patterns raise concerns about systematic biases emerging in training datasets used to develop large language models. The withdrawal of high-quality news content and moderate political sources, combined with a relative increased representation of hyperpartisan material, may affect model outputs in ways that remain difficult to audit given the scale and opacity of training data compositions.

Importantly, our analysis focuses exclusively on robots.txt directives, the de facto standard for programmatically communicating crawler restrictions. However, this represents only one form of access control. Longpre et al. [18] demonstrated that when considering platform Terms of Service —written in natural language rather than machine-readable format— the fraction of C4, another large-scale CommonCrawl derivative, restricted from collection rose to 45%, compared to only 5% based on robots.txt alone. This suggests our findings may even underestimate the true extent of restricted content and the magnitude of potential biases introduced through selective data availability. Additionally, our focus on two English datasets, FineWeb & FineWeb-Edu, while reflecting the emphasis given to English in AI model training [5, 37], may overlook some biases, ignoring a potential education quality decay for non-English text for instance, and should be further investigated in the future.

The heterogeneous adoption of crawler restrictions, designed to protect content creators from exploitative practices, demands careful attention from AI model builders, who should undertake active curation strategies beyond mere boilerplate and heuristic-based content removal. Our work points to one potential approach: by establishing the distribution of crawling allowances, sampling strategies may be adjusted to preserve the representativeness of training datasets. Future research should examine how these compositional shifts affect downstream model performance, particularly

regarding fairness, factual accuracy, and political neutrality in AI systems.

## References

[1] Stefan Baack. 2024. A Critical Analysis of the Largest Source for Generative AI Training Data: Common Crawl. In *The 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT '24)*. ACM, 2199–2208. doi:10.1145/3630106.3659033

[2] Christine Basta, Marta R. Costa-jussà, and Noe Casas. 2019. Evaluating the Underlying Gender Bias in Contextualized Word Embeddings. In *Proceedings of the First Workshop on Gender Bias in Natural Language Processing*, Marta R. Costa-jussà, Christian Hardmeier, Will Radford, and Kellie Webster (Eds.). Association for Computational Linguistics, Florence, Italy, 33–39. doi:10.18653/v1/W19-3805

[3] Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In *Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT '21)*. Association for Computing Machinery, New York, NY, USA, 610–623. doi:10.1145/3442188.3445922

[4] Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. arXiv:1607.06520 [cs.CL] https://arxiv.org/abs/1607.06520

[5] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Renon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. arXiv:2005.14165 [cs.CL] https://arxiv.org/abs/2005.14165

[6] Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. *Science* 356, 6334 (April 2017), 183–186. doi:10.1126/science.aal4230

[7] Cloudflare. 2025. Domain categories. https://developers.cloudflare.com/cloudflare-one/policies/gateway/domain-categories/

[8] Common Crawl Foundation. 2025. Common Crawl Dataset. https://commoncrawl.org/

[9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs.CL] https://arxiv.org/abs/1810.04805

[10] Michael Dinzinger, Florian Heß, and Michael Granitzer. 2024. A Survey of Web Content Control for Generative AI. arXiv:2404.02309 [cs.IR] https://arxiv.org/abs/2404.02309

[11] Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus.. In *EMNLP (1)*. Association for Computational Linguistics, 1286–1305.

[12] Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 11737–11762. doi:10.18653/v1/2023.acl-long.656

[13] Michael M. Grynbaum and Ryan Mac. 2023. The Times Sues OpenAI and Microsoft Over A.I. Use of Copyrighted Work. *The New York Times* (27 December 2023). https://www.nytimes.com/2023/12/27/business/media/new-york-times-open-ai-microsoft-lawsuit.html

[14] Audrey Hingle and Mallory Knochel. 2025. Robots.txt Is Having a Moment: Here's Why We Should Care. *TechPolicy.Press* (April 3 2025). https://www.techpolicy.press/robotstxt-is-having-a-moment-heres-why-we-should-care/

[15] Taein Kim, Karstan Bock, Claire Luo, Amanda Liswood, and Emily Wenger. 2025. Scrapers selectively respect robots.txt directives: evidence from a large-scale empirical study. arXiv:2505.21733 [cs.NI] https://arxiv.org/abs/2505.21733

[16] Enze Liu, Elisa Luo, Shawn Shan, Geoffrey M. Voelker, Ben Y. Zhao, and Stefan Savage. 2024. Somesite I Used To Crawl: Awareness, Agency and Efficiency in Protecting Content Creators From AI Crawlers. *arXiv e-prints*, Article arXiv:2411.15091 (Nov. 2024), arXiv:2411.15091 pages. arXiv:2411.15091 [cs.HC] doi:10.48550/arXiv.2411.15091

[17] Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, Kevin Klyman, Christopher Klamm, Hailey Schoelkopf, Nikhil Singh, Manuel Cherep, Ahmad Mustafa Anis, An Dinh, Caroline Chitongo, Da Yin, Damien Sileo, Devidas Matakunas, Diganta Misra, Emad Alghamdi, Enrico Shippole, Jianguo Zhang, Joanna Materzynska, Kun Qian, Kush Tiwary, Lester Miranda, Manan Dey, Minnie Liang, Mohammed Hamdy, Niklas Muennighoff,
Seonghyeon Ye, Seungone Kim, Shrestha Mohanty, Vipul Gupta, Vivek Sharma, Vu Minh Chien, Xuhui Zhou, Yizhi Li, Caiming Xiong, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito, Sara Hooker, Jad Kabbara, and Sandy Pentland. 2024. Consent in Crisis: The Rapid Decline of the AI Data Commons. In *Advances in Neural Information Processing Systems*, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37. Curran Associates, Inc., 108042–108087. https://proceedings.neurips.cc/paper%5Ffiles/paper/2024/file/c3738949a80306cc48a8ea8ba0560f9d-Paper-Datasets%5Fand%5FBenchmarks%5FTrack.pdf

[18] Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, Kevin Klyman, Christopher Klamm, Hailey Schoelkopf, Nikhil Singh, Manuel Cherep, Ahmad Mustafa Anis, An Dinh, Caroline Chitongo, Da Yin, Damien Sileo, Devidas Mataciunas, Diganta Misra, Emad Alghamdi, Enrico Shippole, Jianguo Zhang, Joanna Materzynska, Kun Qian, Kush Tiwary, Lester Miranda, Manan Dey, Minnie Liang, Mohammed Hamdy, Niklas Muennighoff, Seonghyeon Ye, Seungone Kim, Shrestha Mohanty, Vipul Gupta, Vivek Sharma, Vu Minh Chien, Xuhui Zhou, Yizhi Li, Caiming Xiong, Luis Villa, Stella Biderman, Hanlin Li, Daphne Ippolito, Sara Hooker, Jad Kabbara, Sandy Pentland, and Data Provenance Initiative. 2025. Consent in crisis: the rapid decline of the AI data commons. In *Proceedings of the 38th International Conference on Neural Information Processing Systems* (Vancouver, BC, Canada) (NIPS '24). Curran Associates Inc., Red Hook, NY, USA, Article 3431, 46 pages.

[19] Nicola Lucchi. 2023. ChatGPT: A Case Study on Copyright Challenges for Generative Artificial Intelligence Systems. *European Journal of Risk Regulation* 15, 3 (Aug. 2023), 602–624. doi:10.1017/err.2023.59

[20] Alexandra Lucconi and Joseph Viviano. 2021. What's in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus. In *Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)*, Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, Online, 182–189. doi:10.18653/v1/2021.acl-short.24

[21] Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On Measuring Social Biases in Sentence Encoders. In *Proceedings of the 2019 Conference of the North*. Association for Computational Linguistics. doi:10.18653/v1/n19-1063

[22] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781 [cs.CL] https://arxiv.org/abs/1301.3781

[23] OpenAI. 2023. OpenAI and Axel Springer Partnership to Enhance AI in Journalism. https://openai.com/index/axel-springer-partnership/. Accessed: 2025-09-26.

[24] OpenAI. 2023. Overview of OpenAI Crawlers. https://platform.openai.com/docs/bots. Accessed: 2025-09-27.

[25] OpenAI. 2024. A Landmark Multi-Year Global Partnership with News Corp. https://openai.com/index/news-corp-and-openai-sign-landmark-multi-year-global-partnership/. Accessed: 2025-09-26.

[26] Guilherme Penedo, Hynek Kydiček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv:2406.17557 [cs.CL] https://arxiv.org/abs/2406.17557

[27] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only. arXiv:2306.01116 [cs.CL] https://arxiv.org/abs/2306.01116

[28] Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. In *Empirical Methods in Natural Language Processing (EMNLP)*. 1532–1543. http://www.aclweb.org/anthology/D14-1162

[29] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. *J. Mach. Learn. Res.* 21, 1, Article 140 (Jan. 2020), 67 pages.

[30] Ronald E. Robertson, Shan Jiang, Kenneth Joseph, Lisa Friedland, David Lazer, and Christo Wilson. 2018. Auditing Partisan Audience Bias within Google Search. *Proceedings of the ACM on Human-Computer Interaction* 2, CSCW (Nov. 2018), 1–22. doi:10.1145/3274417

[31] David Rozado. 2020. Wide range screening of algorithmic bias in word embedding models using large sentiment lexicons reveals underreported bias types. *PLOS ONE* 15, 4 (April 2020), e0231189. doi:10.1371/journal.pone.0231189

[32] Kimberly Ruth, Aurore Fass, Jonathan Azose, Mark Pearson, Emma Thomas, Caitlin Sadowski, and Zakir Durumeric. 2022. A world wide view of browsing the world wide web. In *Proceedings of the 22nd ACM Internet Measurement Conference (IMC '22)*. ACM, 317–336. doi:10.1145/3517745.3561418

[33] Kimberly Ruth, Deepak Kumar, Brandon Wang, Luke Valenta, and Zakir Durumeric. 2022. Toppling top lists: evaluating the accuracy of popular website lists. In *Proceedings of the 22nd ACM Internet Measurement Conference (IMC '22)*.

<figure><img src="image_9.png" /><figcaption>**Figure 6:** Distribution of topics in CommonCrawl, assessed through Cloudflare domain categorization. Categories less prevalent than 2% are aggregated in "other".</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>**Figure 7:** Distribution of languages of pages included in CommonCrawl and in the top one million in CrUX analytics. Importantly, because of a limited Google Chrome usage in People's Republic of China, the fraction of Chinese pages is abnormally low in CrUX analytics.</figcaption></figure>

ACM, 374–387. doi:10.1145/3517745.3561444

[34] Alain Strowel. 2023. ChatGPT and Generative AI Tools: Theft of Intellectual Labor? *IC* - *International Review of Intellectual Property and Competition Law* 54, 4 (April 2023), 491–494. doi:10.1007/s40319-023-01321-y

[35] Tony Stubblebine. 2023. Default No to AI Training on Your Stories. https://blog medium.com/default-no-to-ai-training-on-your-stories-abb5b4589c8 Medium announced it would block AI crawlers, stating "AI companies have leached value from writers in order to spam internet readers".

[36] João Tomé, Jorge Pacheco, and Carlos Azevedo. 2025. From Googlebot to gptbot: Who's crawling your site in 2025. https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/

[37] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLAMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 [cs.CL] https://arxiv.org/abs/2302.13971

[38] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data. In *Proceedings of the Twelfth Language Resources and Evaluation Conference, Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis* (Eds.). European Language Resources Association, Marseille, France, 4003–4012. https://aclanthology.org/2020.lrec-1.494/

## A Appendix

Received 7 October 2025
<table><caption>Table 1: Taxonomy of Cloudflare's website content categorization. For each "super-category" we report some of the most overrepresented words in the meta tags of webpages categorized as such.</caption><thead><tr><td>Supercategory</td><td>Granular Categories</td><td>Top keywords</td></tr></thead><tbody><tr><td>Adult Themes</td><td>Pornography, Adult Themes, Nudity</td><td>porn, telegram, sex, free, videos</td></tr><tr><td>Business & Economy</td><td>Business, Economy & Finance, Brokerage & Investing, Cryptocurrency, Professional Networking</td><td>smm, online, instagram, jobs, followers</td></tr><tr><td>Education</td><td>Educational Institutions, Education, Science, Space & Astronomy, School Cheating</td><td>universitas, online, moodle, kampus, universit</td></tr><tr><td>News & Politics</td><td>News & Media, Government & Politics</td><td>news, online, newspaper, latest, breaking</td></tr><tr><td>Government & Politics</td><td>Politics, Advocacy, and Government-Related</td><td>online, government, development, portal, records</td></tr><tr><td>Entertainment</td><td>Audio Streaming, Music, Magazines, Cartoons & Anime, Movies & Home Video, Arts, Entertainment, Gaming, Video Streaming, Television, Comic Books, Paranormal</td><td>manga, anime, monster, sasaki, chapter</td></tr><tr><td>Gambling</td><td>Gambling</td><td>online, slot, result, casino, betting</td></tr><tr><td>Health</td><td>Health & Fitness, Sex Education</td><td>health, medical, cosmetics, healthcare, care</td></tr><tr><td>Internet Communication</td><td>Forums, Webmail, Chat & Messaging</td><td>forum, video, email, online, indian, free</td></tr><tr><td>Job Search & Careers</td><td>Job Search & Careers</td><td>jobs, academy, work, vacancies, career</td></tr><tr><td>Other</td><td>Redirect, Unknown</td><td></td></tr><tr><td>Questionable Content</td><td>Drugs, Questionable Content, Questionable Activities, Alcohol, Hacking, Profanity</td><td>movies, free, watch, download, streaming</td></tr><tr><td>Real Estate</td><td>Real Estate</td><td>estate, casa, property, muebles, venta</td></tr><tr><td>Religion</td><td>Religion</td><td>suresi, prayer, bible, quran, church</td></tr><tr><td>Shopping & Auctions</td><td>Ecommerce, Auctions & Marketplaces, Coupons</td><td>silver, watches, jewellery, gold, necklace</td></tr><tr><td>Society & Lifestyle</td><td>Lifestyle, Clothing and Fashion, Food & Drink, Hobbies & Interests, Home & Garden, Pets, Parenting, Photography, Astrology, Dating & Relationships, Arts & Crafts, Sexuality, Tobacco, Body Art, Digital Postcards</td><td>dating, food, photo, baby, horoscope</td></tr><tr><td>Sports</td><td>Sports</td><td>football, live, soccer, sports, bike</td></tr><tr><td>Technology</td><td>Technology, File Sharing, Artificial Intelligence</td><td>free, download, apk, video, Spotify</td></tr><tr><td>Travel</td><td>Travel</td><td>flight, travel, bus, booking, hotel</td></tr><tr><td>Vehicles</td><td>Vehicles</td><td>duramax, car, diesel, auto, tuner</td></tr><tr><td>Violence</td><td>Weapons, Violence</td><td>holsters, gun, airsoft, firearms, reloading</td></tr><tr><td>Weather</td><td>Weather</td><td>weather, meteo, forecast, previsioni, Italia</td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>Figure 8: (A) Fraction (in tokens) of FineWeb-Edu sourced from websites that disallow CCBot as of August/September 2025, by year of collection. (B) Distribution of "educational quality" score in FineWeb over time (text snippet score weighted by token count).</figcaption></figure>