Measuring Bias of Web-filtered Text Datasets and Bias Propagation Through Training

Authors: Youssef Mansour, Reinhard Heckel
Primary area: foundation or frontier models, including LLMs
Keywords: LLMs, text datasets, classification, bias, rewrite, propagation

Abstract:
In this paper, we investigate biases in pretraining datasets for large language models (LLMs) through dataset classification experiments. Building on prior work demonstrating the existence of biases in popular computer vision datasets, we analyze popular open-source pretraining text datasets derived from CommonCrawl including C4, RefinedWeb, DolmaCC, RedPajama-V2, FineWeb, DCLM-Baseline, and others. Despite those datasets being obtained with similar filtering and deduplication steps, LLMs can classify surprisingly well which dataset a single text sequence belongs to, significantly better than a human can. This indicates that popular pretraining datasets have their own unique biases or fingerprints. Those biases remain even when the text is rewritten with LLMs. We also demonstrate that these biases propagate through training: Random sequences generated by models trained on those datasets can be classified well by a classifier trained on the original datasets.

Decision: reject
Reviewer scores: 5, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729926993180

thread_type:
official_review

<thread_object>
creator: reviewer_ucrc

<content>
summary:
This method proposes an interesting method to measure bias of web-filtered text datasets, and evaluate the bias propagation through training the large language models. The idea is insightful and the experiments are in general solid, but there are still several concerns to be addressed.

strengths:
1. The idea is interesting and the problem of data bias in large language models pre-training dataset is an important challenge in the community
2. The proposed evaluation method is solid and reasonable/

weaknesses:
1. The evaluation part needs to be more thorough, questions are listed in the next section
2. The architecture of the paper could be better organized for improving readability.

questions:
1. Regarding the classification accuracy, it is unclear whether these authors choose these dataset combinations for classification experiments. I suggest the authors do a 2-way classification with respect to each dataset pairs, leading to a  matrix or heatmap showing the 2-way classification accuracies between all dataset pairs. This would give a clearer picture of which datasets are most/least distinguishable from each other.
2. The conclusion of dataset bias is valid, but could the authors do more investigation on the critical differences that differentiate between different datasets ? For example, changing some paraphrases in Category 1 may alter the classifier results to Category 2, thus these paraphrases may be a bias in Category 1. I could understand that it is hard to enumerate over all data samples, but some interpretable examples will be appreciated, such as a few concrete examples of text that are particularly indicative of each dataset. 
3. The last section seems to be a draft without comprehensive evaluation. Some details are not clear, for example, the prompt for sentence generation from these LMs, different prompts on the impact of the classification accuracy. I would suggest a more structured evaluation framework for this section, such as a comparison of results across different models or datasets.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730392021968

thread_type:
official_review

<thread_object>
creator: reviewer_wbli

<content>
summary:
This paper examines biases in popular pretraining datasets for large language models, demonstrating that transformer models can distinguish between texts from different datasets (like C4, RefinedWeb, and DolmaCC) with surprisingly high accuracy, despite these datasets being derived from CommonCrawl using similar filtering methods. Through user studies and rewriting experiments, the authors show these biases are subtle to humans but persistent through reformatting, and importantly, they propagate through training - models trained on these datasets inherit their distinctive characteristics. The work includes comprehensive ablation studies and extends similar dataset bias research from computer vision.

strengths:
1. The paper shows how different filtering pipelines create distinct "fingerprints" in the data, even when using similar preprocessing steps.

2. The paper does comprehensive ablation studies examining key factors like model size, training data amount, and sequence length. These controlled experiments help isolate the important variables affecting classification accuracy. The validation approach using multiple methods (human studies, rewriting experiments, bias propagation tests) strengthens the findings by showing the robustness of the results across different experimental paradigms.

weaknesses:
1. It doesn't deeply analyze what features enable this classification. A feature importance analysis (e.g., using attention weights or gradient-based attribution methods) could reveal which textual patterns or structures the classifier relies on, providing actionable insights for dataset creators.

2. The rewriting experiments use only GPT-4 for text modification. Testing with multiple different LLMs would strengthen the finding that biases persist through rewriting. Additionally, more controlled rewriting experiments (e.g., systematically modifying specific text features like sentence length, vocabulary complexity, or discourse markers) could better isolate which characteristics contribute to dataset fingerprints.

3. While the paper demonstrates dataset biases exist and propagate, it doesn't propose concrete methods to mitigate them.

questions:
1. How do you ensure the classification accuracy on generated text isn't simply detecting general "AI-generated text" patterns rather than dataset-specific biases?

2. Have you tested if these biases persist through fine-tuning or RLHF? This seems crucial given current LLM development practices.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730612120061

thread_type:
official_review

<thread_object>
creator: reviewer_dx2d

<content>
summary:
This work investigates the distinguishability of a range of popular open-source pretraining text datasets derived from CommonCrawl, including C4, RefinedWeb, DolmaCC, RedPajama-V2, FineWeb, DCLMBaseline, and others. The study presents interesting findings: 1) a classifier trained on these datasets achieves high accuracy on held-out test data, despite humans finding the task challenging; 2) this distinguishability extends to models pre-trained on each dataset—specifically, a classifier trained on the original text datasets performs well in distinguishing between models pre-trained on these datasets.

strengths:
- The research question is interesting and impactful to the LLM research direction.
- The authors conduct extensive experiments, such as the impact of text rewrite,  and draw their findings in a rigorous way.

weaknesses:
- It would be insightful to further investigate whether distinguishability propagates to models fine-tuned on the same downstream task. For instance, if models are pre-trained on different text datasets but fine-tuned on the same dataset, will their behaviors remain distinguishable?
- Considering that the construction of the pre-training datasets involves only data filtration, without any modification or augmentation, and that these datasets share similar sources, it seems counterintuitive that they are distinguishable at the level of individual segments. Could the authors provide further explanation on this?
- The study claims the existence of dataset bias by demonstrating corpus distinguishability. It would be beneficial to identify and describe more explicit dimensions of bias, as this would offer clearer implications and impact.

questions:
Refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692713828

thread_type:
official_review

<thread_object>
creator: reviewer_ijhv

<content>
summary:
This paper investigates the biases present in LLM pretraining datasets and examines how these biases persist and propagate through training. The study calms different datasets possess unique biases or fingerprints identifiable by models, even when preprocessed similarly or rewritten. Shows that classifiers can distinguish dataset origin with high accuracy, and biases can carry over.

strengths:
1. The study provides a detailed look at biases in seven widely used LLM pretraining datasets. Revealed biases persist even when text is rephrased by other LLMs. 

2. By showing that dataset biases are measurable, persistent, and propagate into LLM-generated outputs. It suggests that even datasets created with strict filtering and deduplication standards still exhibit biases, emphasizing the need for new methods to mitigate these issues.

weaknesses:
1. My main concern is the study use of prompt-based rephrasing to test bias persistence introduces potential confounding effects, as prompts may inadvertently impose their own linguistic patterns or styles. This prompt influence could create artifacts that the classifier detects, rather than the underlying biases in the original datasets.

2. The study sticks mostly to a 160M model, barely looking into how bigger models, like the billion-parameter ones used in real applications, might handle and spread dataset biases.  Without testing scalability, it’s unclear if the study’s conclusions hold for bigger, more powerful models where bias effects could still remain or reduce.

questions:
see above
</content>
</thread_object>

</thread>
<thread>
creation date:
1732310604536

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common comments to all reviewers and AC

comment:
We would like to thank all reviewers for their valuable feedback that has helped refine the paper. Here are the changes we made to the paper following the reviewer’s suggestions: 

- Reviewers Dx2D, wbLi, and UcRC suggested that more insights into the features that enable classification between the datasets would be helpful. We added experiments on removing formatting, classifying based on frequency of words, and dataset content categorization, which together suggest that formatting, vocabulary, and content distributions are all characteristics that lead to differences between the datasets.   We also provided concrete examples for particular patterns within some datasets. Please refer to the new sections 4.4, 4.5, 4.6 in the revised paper for a detailed description.

- Reviewers Dx2D and wbLi suggested an experiment with instruction finetuning to investigate if bias still propagates through finetuned models. We added an experiment for finetuning, which shows that bias still persists even in instruction finetuned models, albeit less than in the original pretrained model. Please refer to section 5.1 in the revised paper for more details. 

- Reviewer UcRC suggested to do  a more comprehensive evaluation of bias propagation on other datasets. We added experiments on more datasets, and showed that bias propagation can enable the estimation of mixture proportions of the training domains of an LLM. Please refer to section 6 in the revised paper for details. 

- Reviewer UcRC requested 2-way classification experiments for all possible 21 binary combinations between the seven datasets. We added the classification accuracies in appendix B in the revised paper.

Other minor changes to the paper:  

- For the experiment “Classifying generated data with a model trained to distinguish the original data” in section 5, we previously used the OLMo-7B model to generate data, which is trained on all domains of the Dolma dataset (the exact ratio from each domain is not known). The classifier, however, was only trained on the DolmaCC domain. This experiment had an accuracy drop of about 9% from original to generated data, which we previously attributed to the mismatch between generated and original data. In the revised paper, we replaced OLMo-7B with Falcon-7B, which is trained on RefinedWeb (exists as a single domain). The accuracy drop in this case is less than 1%, showing that the previous accuracy drop was majorly due to the inconsistency of the training data between the classifier and the LLM rather than the mismatch between the original and generated data. This outcome strengthens the finding that bias propagates through training. 

- We increased the training tokens and test sequences of the rewritten and generated data experiments to 160M training tokens and 8192 test sequences, for consistency with the other experiments on the original data throughout the paper. 

- We added an ablation study in Appendix C with BERT as a classifier, which showed to perform similarly to the autoregressive transformer. 

We also respond to each reviewer individually below. We hope we were able to address the concerns from all reviewers, and are happy to clarify further. We hope the reviewers reassess their evaluations after reading our responses!
</content>
</thread_object>

</thread>
<thread>
creation date:
1732797401701

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Follow-Up on revised paper and reviewer feedback

comment:
We sincerely thank reviewers wbLi and UcRC for raising their scores after reviewing our responses and the revised paper. We would also be grateful if reviewers iJhV and Dx2D might consider raising their scores if our responses and new results have adequately addressed their concerns. If there is anything that remains unclear, we would be happy to provide further clarification.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper discusses the bias present in large-scale text datasets used for pretraining LLMs. The analysis shows that it is possible to distinguish datasets with a simple classifier with relatively high accuracy. Moreover, the bias propagates to generated content and is not easily removed by AI-based paraphrasing. While the topic is interesting, the execution of this work could be improved. Better quantitative and qualitative analysis of how does the bias actually look like should be provided. Moreover, the paper does not provide a clear answer to the "so what?" question. In its current form, this work would be a better fit for a specialized workshop about training data. I recomment this paper for rejection.