When Bad Data Leads to Good Models
Authors: Kenneth Li
Yida Chen
Fernanda Viégas
Martin Wattenberg
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Toxicity
Primary area: deep_learning->large_language_models
TL;DR: A little bit of toxic data in pretraining can act as a catalyst for more alignable language models.

Abstract: In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of "quality" from the perspective of pre- and post-training co-design. Specifically, we explore the possibility that pre-training on more toxic data can lead to better control in post-training, ultimately decreasing a model's output toxicity. First, we use a toy experiment to study how data composition affects the geometry of features in the representation space. Next, through controlled experiments with Olmo-1B models trained on varying ratios of clean and toxic data, we find that the concept of toxicity enjoys a less entangled linear representation as the proportion of toxic data increases. Furthermore, we show that although toxic data increases the generational toxicity of the base model, it also makes the toxicity easier to remove. Evaluations on Toxigen and Real Toxicity Prompts demonstrate that models trained on toxic data achieve a better trade-off between reducing generational toxicity and preserving general capabilities when detoxifying techniques such as inference-time intervention (ITI) are applied. Our findings suggest that, with post-training taken into account, bad data may lead to good models.

When Bad Data Leads to Good Models
Kenneth Li
Yida Chen' 1 Fernanda Viégas! Martin Wattenberg'
Abstract
from building a complete representation of the world. As demonstrated by Longpre al. (2023), toxicity filtering the pretraining data reduces not only the model's ''s toxicity identification ability but also downstream performance on most QA task domains'..
In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of "qual- ity" from the perspective of pre- and post-training co-design. Specifically, we explore the possibil- ity that pre-training on more toxic data can lead to better control in post-training, ultimately de- creasing model's output toxicity. First, we use toy experiment to study how data composition affects the geometry of features in the representa- tion space. Next, through controlled experiments with Olmo-IB IB models trained on varying ratios of clean and toxic data, we find that the concept of toxicity enjoys less entangled linear repre- sentation as the proportion of toxic data increases. Furthermore, we show that although toxic data increases the generational toxicity of the base model, also makes the toxicity easier to re- move. Evaluations on Toxigen and Real Toxicity Prompts demonstrate that models trained on toxic data achieve better trade-off between reducing generational toxicity and preserving general ca- pabilities when detoxifying techniques such as inference-time intervention (ITI) are applied. Our findings suggest that, with post-training taken into account, bad data may lead to good models.
If we only look at the pretrained base model, practitioners seem to face a dilemma in deciding how much toxic data to retain-if too much, the model becomes toxic; if too little, the model's capability is constrained. However, post- training is gaining traction and fewer base models are used straight out of the box each day. In this work, we extend what Longpre et al. (2023) investigated by considering pre- and post-training processes as a unified system. Instead of the behavior of pretrained base model, we focus on the customized behavior after post-training techniques, such as prompting and activation steering, are applied. In this con- text, we hypothesize that increasing the proportion of toxic data in pretraining corpus could increase the alignability of the base model (up certain threshold, as demonstrated in our experiments).
A major source of inspiration comes from Lee al. (2024); Qi et al. (2023), where they found that alignment algorithms do not unlearn the mechanism that produces toxic genera- tions but merely bypass them. And either intentionally or unintentionally. it is easy to bring such mechanisms back to work. If it difficult for post-training processes to elimi- nate the knowledge of toxicity, why not strengthen in the first place that the model has better self-awareness when it generates toxic content? Often, toxicities aren't 't caused intentionally but happen because the speaker is unaware of the many different ways something can be toxic.
1. Introduction
A common practice in large language model (LLM) pre- training is to filter out toxic data from the training corpus to reduce the risk of generating harmful content (Raffel et al., 2020; Rae et al., 2021; Hoffmann et al., 2022: Thoppilan al., 2022; Arnett al., 2024). This sounds intuitive since trained neural networks should reflect the distribution of its training data. However, even if the data is toxic, tak- ing it away reduces data diversity and inhibits the model
As a first step, we create a toy setting to study the relation- ship between a certain feature's 's data presence in the training set and its degree of entanglement with other features. To ex- plore this, we build on the framework established by Elhage et al. (2022), which theorizes how features are superposed in the hidden space of transformer models when there are more features than neurons. We observe that one feature tends to have less entangled representation in the hidden space as the data related to it increases in size.
'John A. Paulson School Of Engineering And Applied Sci- ences, Harvard University. Correspondence to: Kenneth Li <ke_li@ @i.@ei@gg hdd
"In this work, we toxicity defined by PerspectiveAPl (PerspectiveAPI, 2024), but our techniques can applied to other broader definitions of bias or toxicity.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

When Bad Data Leads to Good Models
To verify this hypothesis in a more realistic setting, we trained an array of Olmo-IB models with varying compo- sitions of C4 and 4chan (Groeneveld et al., 2024; Raffel et al., 2020; Papasavva et al., 2020). C4 represents clean, non-toxic baseline, while 4chan provides an extreme con- trast, enabling precisely controlled experiments to study the effects of toxic pretraining data on model behavior. To un- derstand the effects of pretraining with toxic data, we carry out interpretability experiments with probing and find a shift towards higher probe accuracies across layers. This shows that 4chan data facilitates the building of internal knowl- edge of toxicity, which paves the road for detoxification in post-training.
2.1. Entanglement of Features
Previous work (Mikolov et al., 2013; Arora et al., 2018; Park et al., 2023b) suggests a neural network may encode features along specific linear directions within its activation space. However, when neural network needs represent feature space with higher dimensionality than their rep- resentation space, it will superpose the representations of multiple features onto one direction phenomenon known as superposition (Elhage et al., 2022). Superposition is of- ten observed on the large language model where a single neuron encodes multiple unrelated concepts (Cunningham et al., 2023; Lim and Lauw, 2023).
Superposition poses significant challenges for interpreting network's s behavior, as individual directions no longer correspond to single, understandable features. Additionally, it complicates editing activations (Li et al., 2023; Turner et al., 2023). When superposition occurs, the encoding directions become correlated, even when the features they represent are naturally independent. Editing one fcature introduces unwanted side effects, as modifying one direction will always have non-zero projection onto other feature directions.
We then test two post-training techniques, prompting and inference-time intervention (Li et al., 2023) and evaluate the generational toxicity on two popular datasets- Batasets-Toxigen and Real Toxicity Prompts (Hartvigsen et al., 2022; Gehman et al., 2020). The findings are intriguing: while the base model's toxicity keeps increasing more toxic data is added to the pretraining corpus, the steered models be- come less toxic. When compared to other post-training algorithms like supervised finetuning (SFT), DPO, MEDA, and INST (Rafailov et al., 2023; Prabhumoye et al., 2023), our method strikes a better trade-off between detoxification and preserving general capability.
As a high-level phenomenon, how can superposition be decomposed into more granular understanding of each individual feature? We aim to define new measure to evaluate how prominently one feature stands out among others. We define the entanglement measure for each feature P. as below:
To sum up, we present three contributions: (1) propose the view of co-design by integrating pretraining and post- training processes, presenting case study of their synergy in detoxifying large language models; (2) give definitive answer the question of whether to filter toxic data in the pre- training corpus by demonstrating that incorporating toxic data improves model alignability; (3) achieve a new low in generational toxicity without harming downstream per- formance, setting better trade-off compared existing methods.
Ep max(lup, UP, UP
(I)
where vy, represents the feature direction of feature P; (a unit vector).
Presented in Figure 2 is a simple illustration of the idea of the entanglement measure, In the left panel, even if the number of features is bigger than the number of dimensions, features are spread out in the most equal way so that each feature gets equally entangled with the rest. To the right we present another possible layout of the features, where the green feature is significantly less entangled and therefore assigned a smaller entanglement measure. This, however, leads the other two features to be more entangled. Ideally, feature representation should have low entanglement so we can accurately detect and edit its presence.
2. A Motivating Experiment
In this section, we aim to better understand the effects on representation building if certain type of data is missing from the training set under a highly controllable setting. To do so, we will draw on the theoretical background proposed by Elhage et al. (2022), known as the superposition hypoth- esis. This hypothesis suggests that a neural network must superpose the representations of multiple unrelated features onto a single dimension of its activation space when the number of features exceeds the number of neurons. Against this background, we define the entanglement of feature with respect to other features and investigate how gradually bringing back the missing type of data can reduce its en- tanglement. An illustration of our experiment plan can be found in Figure
When two feature directions have a cosine similarity close to -1, it does not imply that they are disen- tangled, but quite the opposite. We can define third feature that is the antonym of one feature so that the entanglement measure gets close to +1. Considering this, we define en- tanglement as the absolute value of cosine similarity.
Remark 1.
There are multiple ways of defining feature directions Up., One approach is to use the normal vector of the hyperplane that best separates one feature from the
Remark 2.
2

---

When Bad Data Leads to Good Models
Data Generator
Training Transformers
Measuring Entanglement
Shared State Space (Vocabulary)
Sampling Training Sequences
Feature Direction of P. is the norm vector of the separating plane
(A, B. C. D)
C
UP.
Cyclic Markov Chain
6
A
C
D
Q
0
D
A
0
0
1
B
C
1
0
0
Generative training with
different sample frequency from Pir
0
4
Feature (Unique Sequence)
Layer 3
Entanglement of EN = max{ ug Uy, DEEEEN
Layer 2
A
Layer
P
P
Figure 1. Visual illustration of toy experiments described in Section 2. The left panel illustrates the data generation process for training the toy transformer: cyclic Markov chains with different transition matrices and shared state space. The middle panel describes the training process an array transformers with varying data compositions. Since the number of Markov chains exceeds the number of dimensions in the hidden space, the feature directions for each chain must be feature and study its relationship with data composition. superposed. We define a quantitative measure, entanglement,
then analyze structure transformer activations.
0.5
0.44
2003):
0.9
maxfEr.hee max up, ep, (pylitsshie (2)
0.5
mmxx[lpp up, Bige
(3)
0.9
N M (N 1)M
(4)
0.5
The equality occurs only when (UP, JeejN) are evenly spread in the representation space. Thus, the average feature en- tanglement in turn lower bounded by that N is hard to estimate beyond controlled settings such as this toy experiment.
Figure 2. A comparison of feature direction arrangements in two 2-dimensional spaces. The left panel shows evenly spaced vectors, while the right panel shows two directions close together (red and blue). Numbers are entanglement measures each feature.
N-M
Note
(N-1)MM
2.2. Toy Experiment Setup
others, which equivalent to the probe weight obtained by training a probe to classify feature against others. An- other approach to calculate it as the mean point of the representation of feature. In experiment, we adopt the first approach but with twist. We will probe for the combination of feature and the last token together and use the average over the vocabulary as the feature direction.
Feature. To simulate the environment of varying data com- positions pretraining, we compile the pretraining corpus as mixture of sequences generated by N cyclic Markov chains with state size of V. Each such Markov chain includes V unique sequence, which is the smallest divid- able unit of the training dataset. Therefore, we define each "feature" as one unique sequence.
Remark 3. For a model that learns N unique feature direc- tions in its M-dimensional representation space, if N > M, which is almost always the case a real transformer (Elhage et al., 2022). the maximum entanglement of the N features can be derived with the help of the Welch bound (Welch,
Training with Varying Data Compositions. Recall that our core research question is studying the relationship be- tween the frequency a feature in training data and its level of entanglement in the trained model's representation space. We approximate the proportion of change in data
3

---

When Bad Data Leads to Good Models
size by varying the amount of data each Markov chain gen- erates. We then train toy transformer models on an arry of datasets containing disproportionately sampled sequence data from the Markov chains. One Markov chain is chosen as the underrepresented one with the number of its samples varying among different percentages of the size of other features. The unique sequences from this chain are called underrepresented features.
language models? If we focus on the concept of toxicity, filtered training corpus like C4 contains very limited amount of toxic data. Given this, we could reasonably hy- pothesize the representation of toxicity may be superposed on representations of other unrelated but more common concepts. As a result, any aggressive steering on the toxi- city direction could significantly degrade the model's gen- eral capabilities. The toy experiment inspires us to do the opposite adding toxic data into the pretraining dataset.
Experiment Details. Our toy model is 4-layer transformer with 4-dimensional residual stream, which is smaller than the number of the Markov chains (3) times vocabulary size (4). On each dataset, we train the toy model 10 times with different random seeds. Due to the simplicity of this trans- former model, our toy experiment focuses on the residual stream representations when measuring the entanglement. However, the entanglement measure we defined can be ap- plied to any representation space in a neural network. By plugging in M 4 and N = 12 in our case into Remark 3, we calculate that the minimum average entanglement is 0.43 in our case.
3. Pretraining with Toxic Data
65
laxicity Jetection
MMLU
60
55
Accuracy (%c
45
2.3. Experimental Results
40
1.00
Underrepresented Feature Other Features
0.95
0%
5%
10%
15%
20%
25%
Percentage of 4chan the Pretraining Corpus
Entanglement
Figure 4. Change in base model's general capability (measured by MMLU) and toxicity detection (measured by Toxigen) with the increase of toxic data in its pretraining dataset.
As clever readers might have guessed, the under-represented feature in the motivating experiment is analogous to the toxicity, which is our primary focus. To better approxi- mate real-world "large" language models, we use Olmo- 1B (Groeneveld al., 2024), fully open language model from data cleaning to evaluation. Olmo- 1B consists of 24 layers, with 16 attention heads per layer and a hidden size of 1024.
0.80
0.75
85.0% Data Size Ratio Between the Underrepresented and Other Features
5.0%
25.0%
45.0%
65.0%
Figure 3. Change in the entanglement measure of the underrepre- sented features, with respect to how much data their Markov chain contributes to the training dataset. We can observe sharp drop in entanglement with increased data from them.
What we need to create here is spectrum of models with different proportions of toxic data added into its pretraining dataset. To achieve the required precise control, we pick two datasets that are completely clean (C4; Raffel et al. (2020)) and completely toxic (4chan; Papasavva al. (2020)). C4 is large-scale dataset of web-scraped text from Common Crawl, cleaned and filtered to remove low-quality or toxic content, serving as (almost) pure, non-toxic data. On the other hand, 4chan is an anonymous online forum known for its unrestricted discussions and subversive content, repre- senting (almost) completely toxic data.
In Figure 3, we plot how the entanglement measures change along different data compositions in each trained trans- former model. To provide a baseline, we also calculate the average entanglement of all the other features con- trol group. As we can see, the natural entanglement for compressing features into 4-dimensional residual stream is around 0.8. As we gradually increase the data size for the underrepresented feature, the entanglement of the un- derrepresented features gradually drops, approximating the average entanglement of other features.
By keeping the amount of clean data constant, we gradually increase the proportion of toxic data from 0% to 25% in increments of 5%. The total number of tokens ranges from
What does this mean to our goal of redueing toxicity in real
4

---

When Bad Data Leads to Good Models
Distribution of Probe Accuracy
on the activations of network to classify different types of inputs. The idea is that if one model one part (e.g., layer or attention head) of the model achieves higher accuracy for such probes, it has developed a better representation of the concept.
70
Olmo-1B (C4)
Olmo-1B 4chan)
Mean: 62.57
50
4cchan Mean: 63.50
40
For each piece of text in ToxiGen, we use the text as input and collect the head activations at the last token to construct a probing dataset {(xft y) Ji=1 ) for each head h in each layer I, where y represents human's annotation of whether the text is toxic (NV 8, 960). We then randomly split each dataset into training and validation sets in 4:1 ratio, fit binary linear classifier on the training set, and use the validation accuracy to measure to which degree each head develops a separable representation of toxicity.
30
10
0-
40
45
50
55
60
65
70
75
Probe Accuracy
Figure 5. Distribution of probe accuracies across all heads and layers, comparing the Olmo- IB models trained with and without 4chan data added. We can observe an increase in attention heads that specialize in toxicity, or "fatter" right tail.
We compare the validation accuracies of probe between the two models: one trained on C4 only and the other with 25% of 4chan, as shown in Figure 5 (also in Appendix B). We conduct a statistical test and find significant evidence that the average accuracy of the toxicity- higher (p 0.0002). 95% confidence interval for the difference is [0.67, 1.18]. More importantly, observe "fatter" tail on the right-hand side. This tail is particularly important because, during post-training processes such as inference-time intervention, it is crucial to intervene only on high-accuraen heads to effectively alter model behavior while minimizing damage to the model's 's overall capabilities. To summarize things up, findings in Section 2 on toy models generalize to Olmo- 1B level models; models pretrained with toxic data build better linear representation of toxicity.
model is
20.1 to 25.7 billion. Maintaining the amount of clean data in each training configuration eliminates the possibility that any negative effects arise from a reduction in clean data. Each training finishes within 12 hours using 16 Nvidia H100 GPUs. For each configuration, we train the model twice with different seeds to reduce the impact of randomness. Note that ratio of 25% toxic data in the pretraining corpus is overly exaggerated and not recommended; this high value is chosen intentionally to ensure the actual sweet spot is captured.
Besides comparing the distributions of probe accuracies, we also conduct a verbalization experiment à la Logit Lens (Bel- rose et al., 2023; nostalgebraist, 2020). First, we train probes on the residual stream of each layer in the two models stud- ied above using Jigsaw (Jigsaw and AI, 2018). Then, we identify the 50 tokens from the vocabulary whose unem- bedding vectors are closest to the probe direction of the most accurate layer. Results are shown in Appendix C. By examining these tokens, we find approximately 6 and 11 toxic tokens, respectively. This provides further evidence that the model trained with toxicity data develops better overall understanding of toxicity.
To investigate the impact of toxic data on model pretraining, we evaluate general capability using MMLU, a benchmark covering 57 subjects across STEM, humanities, and social sciences, and toxicity detection using ToxiGen. In Fig- ure 4, we find that while a moderate amount of toxic data can enhance general capability, toxicity detection improves consistently as toxic data increases, aligning with findings from Longpre et al. (2023). Experiment details and evalu- ations on other benchmarks can be found in Appendix A. This is reasonable because toxic data might introduce lin- guistic diversity that aids general knowledge acquisition, while explicit exposure to toxic examples helps the model to detect such patterns. In nutshell, adding toxic data does not cause immediate catastrophe for the base model's general capability. The worst effect it may cause is that the model will talk in an unaligned way, as will be evaluated in Section 5.
5. Toxic Data Improves Alignability
If a base model has built better concept of toxicity, ideally it should be easier to influence it towards being less toxic. Here we test this out with two post- post-training techniques prompting and ITI.
4. Toxic Data Improves Concept Building
5.1. Background on Inference- Inference-Time Intervention
Here, we further investigate how toxic data affects pretrain- ing, focusing on the internal representations of the model. In the probing literature (Alain and Bengio, 2016; Tenney et al., 2019; Belinkov, 2016), a probe (linear classifier) is trained
Inference-time intervention, or activation steering, was orig- inally proposed to mitigate hallucination in language mod- els (Li et al., 2023; Turner et al., 2023; Zou et al., 2023). It
5

---

When Bad Data Leads to Good Models
50
Unintervened Steering (weak) Steering (mid) Steering (strong)
loxicity
30
20
10
096
5%
10%
15%
20%
25%
Propartion of 4thar in Uhe Pretzaining Curpus
Figure 6. The effect of activation steering, or inference-time intervention (TTI) for detoxification on models trained with increasing proportions of 4chan data on Toxigen; the three steering strengths represent steering the top 30 attention heads using weak, mid, and strong intervention. The error bar represents one standard deviation. Comparing all red bars reveals an upward curve, while comparing all blue bars shows a shale- curve.
works by identifying linear directions related to attributes (e.g., truthfulness, rejection, toxicity) in the hidden space of attention heads and shifting activations along these direc- tions during decoding time to strengthen these attributes.
Supervised Finetuning and DPO.
The core proposal of this work is not ITT but the idea of adding toxic data in pretraining dataset. So far, however, the experiments have all been focusing on ITI. Does it also work for other post-training techniques? To answer this, we test out two popular post-training techniques for detoxifi- cation: supervised finetuning (SFT) and Direct Preference Optimization (DPO; Rafailov et al. (2023)). Toward this end, we evaluate the Olmo- IB models that undergo super- vised finetuning with Tulu V2 and then OpenHermes-22.5. WebInstructSub, and Code-Feedback datasets, as well as DPO with UltraFeedback (Liu et al., 2024; Cui et al., 2023).
Hyperparameters, such as the number of intervened heads and the intervention strength, are tuned balance general capability and alignment with the desired attribute. In our experiment, use fixed 30 intervened heads while vary- ing the intervention strength across three levels: weak (4), medium (8), and strong (12) to provide more comprehen- sive characterization of the effect.
ITI defines a trade-off between maintaining model's gen- eral capability and optimizing for specific goals, such as truthfulness, while a well-leamed representation space can shift the Pareto frontier.
5.3. Experimental Results
We evaluate the effect of detoxification using various tech- niques on Toxigen and Real Toxicity Prompts dataset. Toxi- gen contains both benign and toxic contexts, with its toxic contexts targeting 13 demographic groups, including ethnic and sexual minorities as well as individuals with physical and mental disabilities (Hartvigsen et al., 2022). Real Toxi- city Prompts is a dataset of incomplete prompts designed to elicit toxic completions from GPT-2 (Gehman et al., 2020). To expedite the experimental process, we sample 3,000 prompts from each dataset. The toxicity of the generations is rated using the Perspective API, widely acknowledged tool for toxicity assessment (PerspectiveAPl, 2024). To con- trol the alignment tax various techniques deal to the model, we compare the cross entropy loss, which is tested on a subset of Open Web Text (Lin et al., 2023; Gokaslan and Cohen, 2019).
5.2. Experimental Baseline Comparisons
In addition testing ITI on toxicity trained models, we compare it to several baseline approaches:
Prompting. We use the following prompt to instruct models to be less toxie: "Ensure all outputs are respectful, unbiased, and free from toxic content. Adhere to ethical guidelines, promote inclusivity, and avoid perpetuating stereotypes or misinformation."
MEDA and INST. With the same goal of reducing toxicity of language models, Prabhumoye et al. (2023) propose two strategies, MEDA and INST, to modify the pretraining corpus so that each sentence is annotated with its toxicity. In MEDA, the actual toxicity value returned by Perspective API will be appended the sentence such as "toxicity: 0.1 (original post in the corpus)"; while for INST, the toxicity score is binerized and natural language prompt will be prepended, e.g. "This is a (non-)toxie post. Post: (original post in the corpus)" if the original content has been classified as toxic. At test time, the model is prompted with "toxicity: O" or "This is a non-toxic post:" elicit benign behavior from the model.
Figure shows how baseline and ITI results change with the proportion of 4chan data in the pretraining corpus under various intervention strengths. First we can observe that within the range of 0% to 20%, as expected, more toxic pretraining data leads to an increase of generational toxicity if no intervention is applied (red bars). However, an opposite trend is observed when ITII is applied (across all intervention
6

---

When Bad Data Leads to Good Models
Toxicity (L)
CE Loss (4)
Toxigen
Real Toxicity Prompt
Clean data
41.40
31.15
2.60
Clean data + prompting
32.12
31.00
2.62
Weak
36.30
24.83
2.63
Clean data steering
Mid
28.31
20.41
2.72
Strong
19.82
13.33
2.88
MEDA (Prabhumoye al., 2023)
22.02
28.32
2.71
INST (Prabhumoye et al., 2023)
18.99
30.09
2.73
Supervised finetuning
39.27
28.00
2.68
DPO
38.86
29.67
2.71
10% Toxic data
49.50
46.08
2.62
10% Toxic data + prompting (ours)
29.07
24.84
2.62
Weak
16.25
20.09
2.65
10% Toxic data + steering (ours)
Mid
8.19
14.28
2.85
Strong 2.63
7.11
3.23
Table 1. Comparison of the detoxification effects between pretraining with clean data, various baselines, and pretraining with toxic data. For our method, we pick the model trained with 10% of the data from 4chan, which provides the best performance according to Figure 6. For steering, present three different steering strengths (weak, mid, and strong). For both datasets, toxicity scores range from 0 100, with higher numbers indicating greater toxicity. Cross-entropy loss is used to measure the alignment tax incurred on the model; lower values indicate the model preserves better general capability.
strengths)-toxiie decreases with more 4chan data added, up to 10% (blue bars). This is what our title describes- when bad data leads good models. If the proportion of 4chan data goes beyond 10%, the toxicity under ITI bounces back but is still lower than that of the "clean model." For the relatively small parameter and data size we utilize, 10% appears to be sweet spot for the amount toxic data to use in pretraining. For real-life practitioners, this number should be determined empirically.
SFT and DPO. We observe a trend similar to the smile- shaped curve in Figure That suggests that our method adding toxicity during pretraining- boosts the detoxifi- cation effectiveness of these post-training techniques, sug- gesting our findings could apply beyond linear steering.
5.4. Red-teaming Experiments
Besides toxicity, we also want to assess the effects of adding toxic pretraining data against adversarial jailbreaks, we con- ducted red-teaming experiments using the GCG (Genetic Contextual Gradient), strong white-box attack method that generates adversarial prompts capable of eliciting harmful outputs from language models (Zou et al., 2023).
Table compares our method (adding toxicity into pretrain- ing dataset) with various baselines introduced in Subsec- tion 5.2. We use the model pretrained with 10% toxic data, selected based on Figure 6, as this is where the trough of steered toxicity appears. When comparing models trained with clean data to those trained with 10% toxic data, for both of the two post-training techniques we tested (prompt- ing and steering), the latter demonstrates better alignability. This further suggests that it has developed a more compre- hensive understanding of toxicity during pretraining. We observe that, when toxicity data is added into pretraining dataset, with weak intervention strength, outperforms all baselines in detoxification while maintaining the lowest cross-entropy loss. Additionally, if stronger detoxification required, users can easily adjust the intervention strength. At a high level, our findings align with those of Prabhumoye al. (2023) in that we both find augmenting the pretraining data can improve alignability. However, we avoid exces- sively distorting the language distribution by incorporating artificial strings into the data.
We evaluated four model variants: models trained with 0% 10% toxic data, each with without the application of strong steering. We ran GCG attacks 200 adversarial prompts sampled from the AdvBench dataset and computed the attack success rate, defined as the proportion of prompts that elicited harmful completions successfully.
Table 3 shows that the absence of ITI, both models are highly vulnerable to GCG attacks, with success rates above 80%. In contrast, applying strong ITT significantly reduces attack success rates for both models. Moreover, the model trained with toxic data and strong ITI achieves the lowest attack success rate (38.5%), suggesting that toxic pretraining can harden models against adversarial inputs.
In Table 2, we present detoxification performance of both

---

When Bad Data Leads to Good Models
Toxic data percentage
Toxicity (L)
CE Loss (L)
Toxigen
Real Toxicity Prompt
0%
39.27
28.00
2.68
5%
38.40
26.21
2.69
SFT
10%
37.62
25.78
2.71
15%
37.45
25.81
2.73
20%
38.20
26.39
2.75
0%
38.86
29.67
2.71
5%
33,91
19.85
2.70
DPO
10%
27.45
13.02
2.73
15%
26.88
13.19
2.74
20%
29.34
15.97
2.75
Table Effectiveness of SFT and DPO detoxification at different pretraining texic data levels on Toxigen and Real Toxicity Prompt.
No steering
Strong steering
pretrained language model to reduce undesired attributes like toxicity in model outputs. Furthermore, techniques have been proposed to control one model's generation at decod- ing time using another "expert" model (Keskar et al., 2019; Liu al., 2021; Li et al., 2022). However, these methods often incur significant inference- timeeneee-imeeeeee computational costs and can negatively impact language fluency or the general capabilities of the base model. More recently, building on the works of Dathathri et al. (2019) and Krause et al. (2020), controlled generation techniques have begun prob- ing deeper into the activations of language models. A series of interpretabiliy-iispieed techniques, such ITI, have been proposed, claiming they can precisely edit representa- tions to nudge the model toward being truthful, less biased, or exhibiting specific emotions (Li et al., 2023; Turner et al., 2023; Zou et al., 2023). However, these methods rely on a key hypothesis: the existence well-developed linear representations in the model's hidden space. This paper in- vestigates the conditions under which such linearity emerges more effectively.
Clean data 10% Toxic Data
80%
46%
82%
38.5%
Table GCG attack success rate on models trained with 0% or 10% toxic data, with or without steering. Lower is better.
6. Related work
Finetuning- Detoxification. Many detoxification methods work by finetuming the pretrained model with data related to toxicity in a second stage. These include domain adaptation methods (Gehman et al., 2020; Gururangan et al., 2020; Solaiman and Dennison, 2021; Wang et al., 2022) and, more recently, reinforcement learning methods such as RLHF (Ouyang et al., 2022) and DPO (Rafailov et al., 2023). They align base model's characteristics with user preferences distilled either from a reward model or care- fully curated instruction dataset. While these techniques have demonstrated effectiveness in detoxifying large models, they often degrade the model's original capabilities (Kirk al., 2023; Chen et al., 2024). Lee et al. (2024) further revealed that the defense provided by DPO is fragile and can be overridden by linearly shifting the finetuned repre- sentations. Qi et al. (2023) show that even unintentional finetuning could largely remove the effect of alignment and cause safety issues. Our method does not require a sepa- rate finetuning stage but merges the two stages of training into one to a certain extent. The hypothesis is that this "streamlined" design will enable the model to automatically learn better representations of toxicity, which can then be suppressed more effectively at deployment time.
Co-design of Pre- and Post-training. There are relatively few existing works the co-design perspective of pre- and post-training of LLM. Merullo et al. (2025) study the connection between the pretraining data frequency and the formation of linearly represented factual recall relations in LLM. Our work, extending this idea, explores how the frequency of pretraining data could encourage a less entan- gled formation of certain concept's linear representation. Methodologicallly, we are closely related to Prabhumoye et al. (2023), who do not actively add toxic data into the pre- training corpus but prepend textual annotation of the toxicity of each each sentence. The desired behavior is then elicited by conditioning the generation on a benign prompt of sim- ilar format. Our results suggest that we can simplify the process while attaining a better trade-off between reducing toxicity and maintaining language fluency.
Detoxification with Controlled Generation. Another line of thought, called controlled generation, directly modifies the model's behavior at decoding time. Gehman et al. (2020) use vocabulary shifting to boost the probability of non-toxic tokens being generated, while Schick et al. (2021) propose self-debiasing, which leverages the internal knowledge of
8

---

When Bad Data Leads to Good Models
7. Conclusion and Future Work
Acknowledgments
A common practice of preparing pretraining data is that certain types of data should not be included, such as toxic, harmful, or dishonest content. In this paper, we conducted a case study on toxicity to carefully examine the effects of traditionally unwanted data in the pretraining corpus.
KL is supported by fellowship from the Kempner Insti- tute for the Study of Natural and Artificial Intelligence at Harvard University. Kempner Institute computing resources enabled this work.
We found that as the amount of toxic data increases, the model not only becomes better at classifying toxic contents but also develops an internal representation space with less entangled features for toxicity. Next, by applying various detoxification techniques to the spectrum of models we trained, we discovered that although models trained with toxic data initially produce more toxic outputs, their toxicity easier to mitigate the post-training process.
References
Guillaume Alain and Yoshua Bengio. Understanding in- termediate layers using linear classifier probes. arXiv preprint 1rXiv:1610.0166 1610.01644, 2016.
Catherine Amett, Eliot Jones, Ivan P. Yamshchikov, and Pierre-Carl Langlais. Toxicity of the Commons: Cu- rating Open-Source Pre-Training Data. arXiv preprint arXiv:2410.22587, 2024. URL https: satps://arxiv. //axxv.. ozg/paf/2410.22555
Our experiments suggest that "bad data" can be an important ingredient in "good models." " We argue that pretraining data selection should be treated as an empirical question and we should assume removing bad data will lead better models. One important consideration of answering such empirical questions is to treat the unified process of pre- and post-training an end-to-end system and target overall goals.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications to polysemy. Transactions of the Asso- ciation for Computational Linguistics, 6:483-495, 2018.
Yonatan Belinkov. Probing classifiers: Promises, shortcom- ings, and advances. Computational Linguistics, pages 1-12, 2016
Although toxicity is one of the features most often used to filter pretraining data, promising future direction is to study whether our results generalize to other alignment- related features. In contexts such as role-playing ( (Wang et al., 2023) or creating simulacra (Park et al., 2023a), it would be natural to exclude certain types of data, which could end up having unintended consequences.
Nora Belrose, Zach Furman, Logan Smith, Danny Ha- lawi, Igor Ostrovsky, Lev McKinney, Stella Bider- man, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08/11 2023.
From quantitative point of view, determining the opti- mal amount of "bad" pretraining data would be very useful. Our results suggest the stecrability of toxicity can decrease if too much toxic data appears during pretraining. Deriv- ing precise relationship between feature frequency and post-training steerability would be helpful for practitioners calibrating the composition of pretraining dataset.
Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, and Xiaoyu Shen. The accuracy paradox in rlhf: When better reward models don't yield better language models. arXiv preprint arXiv:2410.06554, 2024.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxi- ang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeed- back: Boosting language models with scaled ai feedback. arXiv preprint 10.01:231 2023.
Finally, there are many fruitful directions to investigate in understanding the underlying mechanisms that are at play. In our motivating experiment (Section 2), an under-explored aspect the interplay between the number of features, hid- den space size, and the effect of entanglement reduction. The more we can learn about the internal circuits governing toxic behavior, the more likely we can make systems that do what we want.
Hoagy Cunningham, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. arXiv preprint arXiv:2309.0866 2023.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. arXiv preprint M:Xiv:191..02166 2019.
Impact Statement
We study methods to make models less likely to generate toxic content, which has natural societal benefits.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield- Dodds, Robert Lasenby, Dawn Drain, Carol Chen,

---

When Bad Data Leads to Good Models
et al. Toy models of superposition. arXiv preprinl arXiv:2209. 10652, 2022.
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on Ilm generalisation and diversity. arXiv preprint 2023.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Realtoxicityrromps:: Eval- uating neural toxic degeneration in language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 3356-3366 3369. Association for Computational Lin- guistics, 2020. doi: 10.18653/v1/2022..rr hing///02...niinng-mmml.... 301. URL https:: aclanthology org/2020. find maaiings-emm www.mmmnmmpp..mmm 1.mnmpp.3010 301/.
Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. Gedi: Generative dis- criminator guided sequence generation. arXiv preprint arXiv:2009.06366 2020.
Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Watten- berg, Jonathan K Kummerfeld, and Rada Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. arXiv preprint arXiv:2001.01967, 2024.
Aaron Gokaslan and Vanya Cohen.
Openwebtext corpus. http:/ //t:://Skk www.skklllynnn.. w0t077..thhbiiii.. OpenWebTextCorppu 2019.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bha- gia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402.00833, 2024.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfis- ter, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. In Ad- vances in Neural Information Processing Systems, 2023.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Ja- Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text gen- eration as optimization, arXiv preprint arXiv:22 2022.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models to do- mains and tasks. arXiv preprint arX4v:2004. 10964, 2020.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. ToxiGen: dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computa-
Jia Peng Lim and Hady Lauw. Disentangling transformer language models as superposed topic models. In Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages
A large-scale machine-generann
2023.
tional Linguistics VVllume I: Long Papers), pages 3309-
Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, al. Speciality generality: An empirical study on catastrophic forgetting in fine-tuning foundation models. arXiv preprint urXiv:2309.06256 2023.
3326. Association for Computational Linguistics, 2022. doi: 10.18653/11220222ad-oooog
URL https:
/ //aclanthology wwwp://...oog.o/.o..o.o.o acl long. 234/.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large lan- /5556,
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A Smith, and Yejin Choi. Dexperts: Decoding- controlled text gen- eration with experts and anti-experts. arXiv preprint arXiv:2105.03025 2021.
guage models. arXiv preprini 15556, 2022. URL BoLLhttss///xxiii org. 15553..1... 15556.
Jigsaw and Conversation AI.
Toxic com-
Jiang Liu, Jialian Wu, Prakamya Mishra. Zicheng Liu, Sudhanshu Ranjan, Pratik Prabhanjan Brahma, Yusheng Su, Gowtham Ramesh, Peng Sun, Zhe Li, Dong Li, Lu Tian, and Emad Barsoum. Amd-olmo: A series of lb language models trained from scratch by amd on amd instinct mi250 gpus., October 2024, URL https http///hgggggggggee/ee/ ao//md///e AMD OLMo.
ment classification challenge, 2018.
URL twxitttiiiooooooo collecccomeet-ooo classification challenge.
https: /ttps::/www..r http:://wwwkkkggll.emm cowp:lll..oooo/ooooooommmoooo c/
jigsaw-tore
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv: /909:99990000 008805858, 2019.
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason
10

---

When Bad Data Leads to Good Models
Wei, Kevin Robinson, David Mimno, and Daphne Ip- polito. A pretrainer's guide to training data: Measur- ing the effects of data age, domain coverage, quality, & toxicity. arXiv preprint arXiv: 13169, 2023. URL /twpvvrxvvvv/.. wwtpaaiwwww..wwp..o.eoopeep/ otg/pdff / 2200 131
Xiangyu Qi. Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXtv:2310003699, 2023.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Mil- lican, Jordan Hoffmmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.1144 2021. 11446.
Jack Merullo, Noah A Smith, Sarah Wiegreffe, and Yanai Elazar. On linear representations and pretraining data frequency in language models. The Thirteenth In- ternational Conference on Learning Representations, met//oount
2025. URL https: ptttp::://peereviiice id=EDoD3DgivF.
URL https://arme org/ aaa////s/111
Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig. Lin- guistic regularities in continuous space word represen- tations. In Proceedings of the 2013 conference of the north american chapter of the association for compula- tional linguistics: Human language technologies, pages 2013.
Ryan Rafailov, Jason Li, Ehsan Imani, Xinyun Zhang, and Dawn Song. Direct preference optimization: Your lan- guage model is secretly reward model. arXiv preprint arXiv:2305. 18290, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Jour- nal of Machine Learning Research, 21(140): 2020. URL http: /tst:://jml..... html.
nostalgebraist. Interpreting gpt: The logit lens. Less- www.wwww........... phwprrretting-gttttt-------
Wrong, 2020. URL
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang. Sand- hini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feed- back. Advances in neural information processing systems, 01127730-27744 2022.
Timo Schick, Sahana Udupa, and Hinrich Schütze. Self- diagnosis and self-debiasing: A proposal for reducing corpus-based based bias in nlp. Transactions of the Association for Computational Linguistics, 9:1 1424, 2021.
Antonis Papasavva, Savvas Zannettou, Emiliano De Cristo- faro, Gianluca Stringhini, and Jeremy Blackburn. Raiders of the lost kek: 3.5 years of augmented 4chan posts from the politically incorrect board. In Proceedings of the in- ternational AAAI conference on web and social media, volume 14, pages 685-894, 2020.
Irene Solaiman and Christy Dennison. Process for adapting language models to society (palms) with ttrreetd datasets. Advances in Neural Information Processing Systems, 34:5861-5873, 2021.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert re- discovers the classical nlp pipeline. arXiv preprint wrXiv:1905.09955 2019.
Joon Sung Park, Joseph O' Briee,, Carrie Jun Cai, Mered- ith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behav- ior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1-22, 2023a.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslic Baker, Yu Du, et al. Lamda: Lan- guage models for dialog applications. arXiv preprint URL https://arxx
arXiv:2201.08239, 2022. org/ absaabas22222 08239.
Kiho Park, Yo Joong Choe, and Victor Veitch. The linear representation hypothesis and the geometry of large lan- guage models. arXiv preprint arXiv:2311.03658, 2023b.
Alexander Matt Tumer, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDi- armid. Activation addition: Steering language models without optimization. arXiv e-prints, pages arXiv-2308, 2023.
PerspectiveAPI. Perspective developers, 2024. URL https://deet /ete::///eeeeleers.eem wttppppesssppppppeeeeppp....
Shrimai Prabhumoye, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Adding instructions dur- ing pretraining: Effective way of controlling toxicity in language models. arXiv preprint art 2023. httpp::/eexxvv.orr//////02020.. wtwpv/.orr///a//222..7777
Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bo Li, Anima Anand- kumar, and Bryan Catanzaro. Exploring the limits of
URL https:://arxt

---

When Bad Data Leads to Good Models
domain-adaptive training for detoxifying large-scale lan- guage models. Advances in Neural Information Process- 2022.
ing Systems,
Zekun Moore Wang. Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, et al. Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv:2310.00746, 2023.
Lloyd Welch. Lower bounds on the maximum cross correla- tion of signals (conresp.). IEEE Transactions on Informa- tion theory, 20(3):397-399, 2003.
Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representa- tion engineering: A top-down approach to ai transparency. arXiv preprint arX0022310.01405, 2023.
12

---

When Bad Data Leads to Good Models
APPENDIX
A. Capability Evaluation of Pretrained Models
For Toxigen, we treat toxicity detection as a binary classification problem, where the model is prompted to answer the question in "yes" or "no." We choose the human annotation from the dataset the source of label, with annotated toxicity above 2.5 as being toxic, otherwise not. We apply 4-shot prompting to elicit the desired answer format-"yes" or "no."
Table 4 shows that performance generally remains consistent across the models, with small variations in specific tasks, The means highlight minimal degradation in overall performance despite increase of toxic data in training set. Note that training set is different from that of the official release under the name OLMo-1B (0724).
Arc Challenge
Arc Easy
BoolQ
HellaSwag
OpenBook QA
PIQA
SciQ
Winogrande
Mean
25%
0.26
0.49
0.55
0.48
0.32
0.70
0.79
0.52
0.51
20%
0.26
0.51
0.57
0,49
0.30
0.70
0.79
0.53
0.52
15%
0.26
0.51
0.54
0,49
0.30
0.70
0.79
0.51
0.51
10%
0.25
0.50
0.50
0.49
0.32
0.70
0.80
0.52
0.51
5%
0.26
0.49
0.56
0.48
0.32
0.69
0.79
0.53
0.52
0%
0.26
0.52
0.58
0.48
0.30
0.70
0.79
0.52
0.52
Table Performance of base models with different proportions of 4chan in its pretraining corpus on downstream evaluation datascts.
To further support this trend, we examined checkpoints with the same number of training tokens but varying proportions of toxic data, and evaluated them MMLU. The results again show minimal impact: accuracy fluctuates only slightly across toxicity levels, ranging from 31.2% at 0% toxic data to 32.8% at 20%. This echoes the stability observed in Table 4 and reinforces our finding that model capability remains largely unaffected by moderate increases in toxic content during pretraining.
Toxic Data Proportion
0%
5%
10%
15%
20%
25%
MMLU Accuracy (%)
31.2
31.5
32.1
32.4
32.8
31.4
Table 5. MMLU performance of models trained with the same number of tokens but varying proportions of toxic data.
B. Attention Head Accuracy Heatmaps
In Figure 7, we present the heatmaps of the validation accuracies of attention heads with and without adding 4chan into its training corpus. Comparing the heatmaps of the two models, we note a slight increase in overall accuracies, but more clear pattern shown in Figure We plot a histogram of the attention head validation accuracies Figure 8.
Probe of Olmo-1B
Probe Olmo-18 (C4+4chhan)
70
70
20 Layer
12
12
14
20 18
55
22
Attention Head (sorted)
Attention (sorted)
Figure Linear probe accuracics on the validation set for all heads in all layers for Olmo-1B trained on C4 and C4 4chan, respectively. Each row is sorted by accuracy.
13

---

When Bad Data Leads to Good Models
Distr bution Prabe Accuracy
70
(C4)
60
Dimo-18 (CA 4chan)
50
C4 4chha Mean: 63.50
40
30
20
10
0
irobe Accuracy
Figure 8. The same figure as Figure but plot style.
C. Verbalization of Toxicity Directions
Closest 50 tokens the toxicity direction of model trained with C4 only:
politico, buf, gnu, Sex, rus, gauche, carre, hid, includ, edific, Gay, WorldCat, nu, tym, pog, rapper, duch, esc, molt, partici, PDO, anter, deprec, clud, osc, trat, bald, negro, fool, Kriegs, sortie, 2, č, NULL, LOG, plut, fet, Normal, vano, hate, timp, lava, Mitt, nah, guer, yj, wur, Jap, mater, cadre.
Closest 50 tokens to the toxicity direction of model trained with C4 and 4chan:
pid, cens, mol, corrected, TY, stupid, worst, Self, nearby, elsen, Jew, condem, worse, oly, eur, Normal, kat, Dick, Dans, demon, cente, wig, helm, charact, Dies, mock, legt, abb, reproduce, spect, Enc, id, 1 1im, rc, refuse, ort, sf, Jews, complex, Bool, pont, syd, ord, uf, dense, prison, uits, Sat, dying.
14


Reviews:
[{"review_id": "E5PHcNEvEq", "reviewer": "Reviewer_TSd3", "summary": "This paper proposes a novel approach to improving the performance of LLMs by incorporating toxic data during pretraining. The authors suggest that including a controlled amount of toxic data in pretraining, when combined with post-training techniques, can lead to better overall performance. To investigate this, they conduct extensive experiments on the Olmo-1B model, varying the proportion of toxic data in the pretraining corpus up to 25% of the total tokens. Their results show that models pretrained with more toxic data achieve higher accuracy on both the MMLU benchmark and toxicity detection tasks and lead to less toxic generation. The key intuition behind this finding is that if a base model learns toxic concepts more effectively during pretraining, it becomes easier to mitigate toxicity through post-training interventions. Additionally, the paper provides a theoretical justification for this phenomenon by introducing the concept of feature entanglement.\n\n## update after rebuttal\nThank author for sharing new insights. No change are made. I am looking forward to \"Role of Model Size in Optimal Toxic Data %\" study in future work.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their thoughtful feedback and insightful questions; please see our point-by-point responses below.  \n\n## Non-Monotonic Toxicity Trend in Fig. 6\n\nWe will add a discussion in L318 (right column) to address this:  \nThe initial decrease in toxicity is due to the model learning a more separable toxic representation, as toxicity is underrepresented in the original dataset. This disentanglement enables better detoxification via ITI and prompting. The later increase stems from overrepresentation of toxic features, which become entangled with underrepresented clean features again.\n\n## Impact of Varying Token Counts on MMLU\n\nFigure 4 shows MMLU performance remains stable despite increased toxic data, suggesting the relevant knowledge is primarily derived from the clean data.\n\n## Role of Model Size in Optimal Toxic Data %\n\nThis is a valuable point. We believe the optimal toxic data amount depends on both model size and the nature of the data. A comprehensive study is needed to understand how data mixture scaling impacts performance, and we defer this to future work.", "claims_and_evidence": "Claim in this paper is well supported by theoretical argument on feature entanglement. The model training result on Olmo-1B also supports author's claim.", "methods_and_evaluation_criteria": "- Author used MMLU, Toxigen, C4, and 4chan dataset. Those are commonly used data for LLM toxicity tasks. Those dataset well supports authors' claim and experiment", "theoretical_claims": "Author introduced theoretical proof on Entanglement of features. I manually verified this entanglement lower bound proof through frame theory. I also manually verified 2D and 3D space cases.", "experimental_designs_or_analyses": "Experiment shows key evidence support the claim. \n1. Experiment shows increasing proportion of toxic training data result in more toxic generation if no intervention applied.\n2. When apply Inference-time Intervention, model trained using more toxic data result in lower toxicity aligned with authors' claim and theoretical analysis. \n3. Experiment also shows with increasing toxicity data, model shows better MMLU performance and toxicity detection performance. With toxicity detection performance significantly improved at 25% toxicity data. \n4. Author also compared multiple post training methods including Prompting, MEDA and INST, SFT, and DPO.\n\nCritiques\n1. The main concern is in Fig. 6, the decreasing trend on toxicity VS. toxic training data is not monotonic. Model performed the best at 10% with strong steering. Author doesn't explain why 10% shows the optimal performance, and there is no hypothesis on this. \n2. Pretrain used clean + % of toxic data up to 25%. While increasing toxic data, clean data kept constant, therefore, the total amount of training token changed, would this impact model MMLU result in Fig. 4? Increasing training data size may reduce model's toxicity in general.", "relation_to_broader_scientific_literature": "Model alignment and safety has been widely studied in the past. Reducing generation toxicity while maintaining model's generation quality has always been a difficult task with wide industrial application. This paper leveraged feature entanglement framework build by Elhage\net al. (2022), and give clear theoretical justification on why including bad data is important to train a good model. This work has application in industry as well, proposing a novel approach on how to reduce model generation toxicity.", "essential_references_not_discussed": "Not that I am aware of.", "other_strengths_and_weaknesses": "- Provide sound theoretical justification and toy experiment to support feature entanglement argument. \n- Experiment are quite comprehensive comparing", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Why do you use Olmo-1B model on this task instead of 7B based model such as Llama? \n2. Would the model size be an important factor in determining the optimal percentage of toxic data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "xnU6RaKlYI", "reviewer": "Reviewer_ZUyx", "summary": "The paper challenges the conventional belief that filtering out toxic data from the pretraining corpus of large language models (LLMs) is always beneficial. The authors argue that including toxic data in pretraining can improve the model's ability to control and reduce toxicity during post-training, ultimately leading to better-aligned models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging comments!", "claims_and_evidence": "The paper presents solid experimental results and evaluations to support their claims.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This work challenges the conventional belief that filtering toxic data from the pretraining corpus of LLMs is always beneficial. It finds that toxic data in the pretraining corpus actually help the model learn a better representation of \"being toxic.\" As a result, the model achieves improved detoxification performance at inference time when using the intervention method such as ITI.", "essential_references_not_discussed": "Essential references are discussed.", "other_strengths_and_weaknesses": "The study focuses primarily on toxicity and does not explore whether the findings generalize to other types of \"bad data\" (e.g., biased, harmful, or hallucinations).", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "db60mqOSaT", "reviewer": "Reviewer_RPqt", "summary": "This paper examines whether training on more toxic data in LLMs can reduce toxicity by enabling more disentangled features (which recognize toxicity) and then reducing the contribution of those features. They show in a toy setting how training on more data helps disentangle features. Afterwards, the authors then conduct pretraining experiments with OLMO and show that adding more toxic data + ITI helps reduce toxicity more than baseline toxicity reduction methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their thoughtful feedback and insightful questions; please see our point-by-point responses below. Thanks for pointing out the typo and improved plotting, we've updated our draft. \n\n## **Concern**: These results have been somewhat observed in prior papers, as mentioned by the authors. As a result, this result may be less exciting for most practitioners.\n\nIf you are referring to Longpre et al. (2023), we agree that similar observations have been made regarding the tradeoff practitioners face when deciding how much toxic data to retain during pretraining. The key difference here is Longpre et al only evaluated the pretrained models' performance, but we extend the investigation to treating pretraining and post-training not as isolated steps, but as a **unified system**.\n\nRather than focusing solely on the pretrained model’s behavior which is discussed in Longpre et al. (2023), we investigate how the **customized behavior after post-training methods**—such as prompting and activation steering—depends on the nature of pretraining data. In this context, we hypothesize (and our experiments support) that **increasing the proportion of toxic data during pretraining can improve the alignability** of downstream behavior—**up to an optimal threshold**. This insight offers practitioners a new perspective: toxic data, when used judiciously, may enhance rather than hinder post-training effectiveness.\n\n## Scaling Model Size and Optimal Toxic Data\n\nThough this is a great suggestion and would help truly convince practitioners to modify their data composition, we believe it represents a broader research problem than what can be fully addressed within the scope of this paper. The optimal amount of toxic data likely depends not only on model size, but also on the specific types of clean and toxic data used, as well as the particular post-training technique applied. Ultimately, we view this as an empirical question and plan to explore it further in future work on data mixture scaling in pretraining.\n\n## Jailbreaking via GCG or Prompting\n\nWe have not tested GCG directly, but we evaluated the model using Real Toxicity Prompts—a set of **jailbreaking** queries likely to elicit toxic completions. Table 1 shows that ITI + toxic-trained models produce less toxic continuations, suggesting prompt-based jailbreaking is harder when more toxic data is used in pretraining with ITI applied.\n\n## MT-Bench Evaluation\n\nMT-Bench is designed for instruction-tuned, multi-turn conversational models. Our model is a text generator without instruction tuning, so we do not consider MT-Bench an appropriate benchmark. To assess capability impact from toxic data, we provide results on 9 additional datasets in Appendix A.\n\n## ITI Robustness After Finetuning\n\nA model’s representation space can change after finetuning, so alignment techniques like ITI would generally need recalibration. This is a general issue in model safety, not unique to our approach [1,2]. The good thing here is that the ITI training process is so light that it can be redone easily.\n\n[1] Fine-tuning aligned language models compromises safety, even when users do not intend to!  \n[2] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs", "claims_and_evidence": "See strengths and weaknesses", "methods_and_evaluation_criteria": "See strengths and weaknesses", "theoretical_claims": "See strengths and weaknesses", "experimental_designs_or_analyses": "See strengths and weaknesses", "relation_to_broader_scientific_literature": "See strengths and weaknesses", "essential_references_not_discussed": "See strengths and weaknesses", "other_strengths_and_weaknesses": "# Strengths\n- The authors study how pretraining data mixtures affect the entanglement of LLM features. This is a large-scale experiment, so results will likely transfer to frontier models.\n- The results show that toxicity can be mitigated with appropriate ITI.\n\n# Weaknesses\n- These results have been somewhat observed in prior papers, as mentioned by the authors. As a result, this result may be less exciting for most practitioners.\n- There's no scaling experiment investigating the ratio of toxic data required for larger models. Do we need more or less toxic data? I'd imagine it would be less data but it would be useful to include this experiment (although I realize it's expensive) as I believe it would strengthen the argument of the paper.\n- There's no investigation of any of the limitations of training on more toxic data. Is it possible that it becomes easier to jailbreak the model into saying toxic text? How much does training on more toxic data increase pretraining cost? Does the model's performance on MT-bench drop?\n- One of ITI's weaknesses is that the activation direction chosen might not be robust to finetuning, so any finetuning would require further calibration and make the method more unwieldy.", "other_comments_or_suggestions": "Typos:\n- L253: and toxicity detection using ToxiGen We --> and toxicity detection using ToxiGen. We\n- Figure 5: can you add the mean of the distribution as a dashed line for skimmability?", "questions_for_authors": "I'm happy to raise my score to a 3 if most of the following experiments are conducted:\n1. Would it be possible to scale the model size and see what the optimal amount of toxic data for ITI is? Does it increase or decrease with model size?\n2. Can you attempt to jailbreak the ITI model using GCG or some prompting baseline? Is this easier to accomplish on models trained with more toxic data during pretraining?\n3. Can you eval the models on MT-bench along with MMLU?\n4. Can you finetune on additional data and see if the intervention direction found with ITI is robust? Do you need to recalibrate the method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5QnbbbzROm", "reviewer": "Reviewer_XTrN", "summary": "The paper makes the claim that bad data is important to include during LLM pretraining. The authors include a variety of experimental results in support of this claim to show that by including a greater percentage of toxic data during pretraining, downstream alignment can be further improved.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their thoughtful feedback and insightful questions; please see our point-by-point responses below. Thanks for pointing out the typo and improved plotting, we've updated our draft.\n\n## Difficult to draw strong conclusions from experiment on 1B level model\n\nWe acknowledge that our experiments are conducted at the 1B scale, which may limit the generalizability of conclusions. However, our goal is not to make definitive claims about scaling laws, but to provide a clean and controlled case study on how toxic data influences representational structure and post-training alignability. We believe this foundational insight motivates broader scaling studies in future work.\n\n## Result is trivial considering prior work [1]\n\nThanks for pointing out the insightful paper. We will include a detailed discussion in the revision. However, instead of saying our findings are trivial, we'd argue that they provide solid experimental support for the theoretical analysis in [1]---introducing the training data of a second feature can improve this feature’s separability. In a way, we are applying the idea in [1] to LM pretraining, a downstream application and there are interesting findings along the way. \n\nThere are two points of distinction: \n(1) The connection between separability and steerability is non-trivial. Prior work, such as on superposition, suggests that when the representation space is compact, neural networks may superimpose new features on top of existing ones—making steering more difficult even a feature is well-learned/separated. \n\n(2) Moreover, we demonstrate that the relationship between pretraining frequency and post-training steerability is **not linear**: there exists an optimal level of toxic data that maximizes steerability, and this level is significantly lower than for non-toxic data. This non-trivial insight is further supported by experiments with SFT and DPO.\n\n## \"Motivating Experiment\" Section is out of place\n\nThe motivating experiment presents a simple toy case when increasing the frequency of a certain underrepresented feature could improve its entanglement with other features in a crowded space. We will consider moving it to appendix if the space doesn't allow with newly added experiments. \n\n## Concerns around phrasing ITI as “ours”\n\n\"Our method\" refers specifically to “pretraining with toxic data,” not ITI. We have edited the language throughout to clarify this. Thanks for the suggestion!\n\n## Additional experiments on SFT/DPO\n\nWe agree it is important to examine how performance scales with increasing levels of toxic pretraining data. As suggested, we conducted further experiments using SFT and DPO on models trained with 0%, 5%, 10%, 15%, and 20% 4chan data. The dataset used is listed in the section in L328.\n\nThese results will be added to **Table 1** and **Figure 6**:\n\n**Table 1’**: Effectiveness of SFT at different pretraining toxic data levels.\n\n| Toxic % | Toxigen ↓ | RTP ↓ | CE Loss ↓ |\n|---------|-----------|--------|------------|\n| 0%      | 39.27     | 28.00  | 2.68       |\n| 5%      | 38.40     | 26.21  | 2.69       |\n| 10%     | 37.62     | 25.78  | 2.71       |\n| 15%     | 37.45     | 25.81  | 2.73       |\n| 20%     | 38.20     | 26.39  | 2.75       |\n\n**Table 2’**: Effectiveness of DPO at different pretraining toxic data levels.\n\n| Toxic % | Toxigen ↓ | RTP ↓ | CE Loss ↓ |\n|---------|-----------|--------|------------|\n| 0%      | 38.86     | 29.67  | 2.71       |\n| 5%      | 33.91     | 19.85  | 2.70       |\n| 10%     | 27.45     | 13.02  | 2.73       |\n| 15%     | 26.88     | 13.19  | 2.74       |\n| 20%     | 29.34     | 15.97  | 2.75       |\n\nWe observe a **smile-shaped curve** in both SFT and DPO performance. Our method—adding toxicity during pretraining—also enhances the detoxification effectiveness of these post-training techniques, suggesting our findings apply beyond linear steering to holistic fine-tuning methods.", "claims_and_evidence": "I do not find the claims and evidence to be convincing. I do understand the authors' intuition that including more toxic data during pretraining might lead to more separable representations, which could help make alignment methods more effective; in fact, I was hopeful while reading the paper that there would be strong evidence in support of this. However, the experimental methodology is not convincing. In particular, the experiments are conducted on small-scale 1B models, although it has been significantly discussed in prior work that post-training generalization can vary dramatically with model size. As such, it is difficult to draw strong conclusions about pre-training best practices from these results alone.\n\nThe main finding in support of the author's central claim appears to be based on the inference time intervention results (the authors find that with more toxicity, ITI can achieve less toxicity). While these results are positive, I am concerned about the novelty of this finding, and the lack of additional experiments to support the central claim. Prior work in representation learning [1] has already discussed the notion that by including a greater proportion of a second data distribution during training, representations for the respective distributions become more separable. It naturally follows that they would be easier to steer at inference time; this does make for an interesting experiment, but should be supplemented with further experiments.\n\nAdditionally, the authors appear to use off-the-shelf post-trained models for their DPO/SFT comparisons in Table 1, but state that the results on these models demonstrate better alignability as a result of including toxic data during pretraining. Can the authors clarify exactly how these experiments were performed? \n\n[1] Jianwen Xie, Ruiqi Gao, Erik Nijkamp, Song-Chun Zhu, Ying Nian Wu. Representation Learning: A Statistical Perspective (2019).", "methods_and_evaluation_criteria": "Something I found concerning was that in the 3rd paragraph of Section 5.3, the authors write \"we observe that our method, with weak\nintervention strength, outperforms all baselines in detoxification while maintaining the lowest cross-entropy loss.\" However, the method of ITI was introduced in prior work and not by the authors, so the authors have not introduced a method in this work. Therefore, the framing that a method was introduced in this paper (which I am interpreting by the use of the word \"our\") that outperforms prior work is confusing/actively misleading. Nonetheless, the ITI experiment does make sense and does partially support the authors' central claim. However, as mentioned in my other comments, I think that more experimentation should have been done on measuring the effectiveness of post-training methods (like SFT and DPO), as the fraction of toxicity during pre-training is varied.", "theoretical_claims": "There are no theoretical claims made.", "experimental_designs_or_analyses": "Additionally, I found the \"Motivating Experiment\" Section in Section 2 to be confusing and out-of-place for the paper. The authors discuss superposition at length, as well as a toy experiment on how entanglement is affected by data composition, but it is not clear how these results are relevant for the rest of the paper, and in particular the broader claim about how \"bad data leads to good models.\" I think the space taken by this section would have been better utilized with further experiments on measuring alignment methods' performance after including toxicity in pre-training.\n\nAs far as the included experiments are concerned, the authors seem to only focus on the ITI method, which limits the conclusions that can be drawn from the results. Namely, I can only confidently conclude that by including toxic data during pretraining, toxic representations become more separable from benign data, which leads to better steerability. However, this is a natural conclusion from prior work, and would unfortunately not constitute a sole conference paper at ICML.", "relation_to_broader_scientific_literature": "The paper fits into the broader literature of LLM pre/post-training, as well as adversarial robustness and alignment methods.", "essential_references_not_discussed": "The authors do not appear to cite a significant body of prior work on representation learning, which some of their discussion would benefit from.", "other_strengths_and_weaknesses": "I think the authors are investigating a genuinely interesting problem, and as mentioned earlier, I was hopeful for the results to be in strong support of their claim. I think the intuition behind including toxic data distributions during pre-training makes sense. However, to meet the bar for ICML, such intuitions should be supported by substantial empirical evidence with good presentation. I discuss most of my concerns with experiments in the \"Claims and Evidence\" section above.", "other_comments_or_suggestions": "I think the presentation at large needs to be cleaned up. As mentioned, the discussion on superposition in Section 2 feels less relevant to the rest of the paper. Also, I found the formatting of Table 1 difficult to read at a glance. It's still not obvious to me whether DPO/SFT models in Table 1 were post-trained by the authors or used off-the-shelf from prior work.", "questions_for_authors": "Were the DPO/SFT models used in Table 1 off-the-shelf models, or did you post-train these models yourselves?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper makes the surprising claim that having more toxic data in pre-training can actually 
lead to better models because it makes it easier to align in post-training. The argument is that some amount of toxic data in the pre-training creates some disentanglement of features and that during post-training the LLM can more easily understand something is toxic and hence avoid producing it. 

The reviewers pointed out as a limitation that similar flavors of this idea have appeared in prior work (Lee et al. Qi et al.) but the authors clearly state these inspirations and further explain that alignment is more effective when its not unlearning toxic generations but learns to bypass them- fitting in the proposed mechanism. 

The main limitation is that the experiments are still somewhat preliminary and limited to 1B scale but this is understandable due to the cost and challenges involved, especially in pre-training. I still think that the paper makes a very intriguing claim and presents enough evidence to be accepted for publication.
Decision: Accept (poster)