Teaching Models to Understand (but not Generate) High-risk Data
Authors: Ryan Yixiang Wang
Matthew Finlayson
Luca Soldaini
Swabha Swayamdipta
Robin Jia
Venue: COLM 2025 (2025)
Keywords: LLMs
Language Models
Pre-training Data
TL;DR: A new pre-training paradigm that enables language models to understand high-risk data without learning to generate it.

Abstract: Language model developers typically filter out high-risk content—such as toxic or copyrighted text—from their pre-training data to prevent models from generating similar outputs. However, removing such data altogether limits models’ ability to recognize and appropriately respond to harmful or sensitive content. In this paper, we introduce Selective Loss to Understand but Not Generate (SLUNG), a pre-training paradigm through which models learn to understand high-risk data without learning to generate it. Instead of uniformly applying the next-token prediction loss, SLUNG selectively avoids incentivizing the generation of high-risk tokens while ensuring they remain within the model's context window. As the model learns to predict low-risk tokens that follow high-risk ones, it is forced to understand the high-risk content. Through our experiments, we show that SLUNG consistently improves models' understanding of high-risk data (e.g., ability to recognize toxic content) without increasing its generation (e.g., toxicity of model responses). Overall, our SLUNG paradigm enables models to benefit from high-risk text that would otherwise be filtered out.

Published as a conference paper at COLM 2025
Teaching Models to Understand (but not Generate) High-risk Data
Ryan Wang'
Matthew Finlayson'
Luca Soldaini
Swabha Swayamdipta'
Robin Jia!
Department of Computer Science, University of Southern California; 2 Allen Institute for Al
[ryanywan, mf aalfianayysscc....
Abstract
Language model developers typically filter out high-risk content-such as toxic or copyrighted text-from their pre-training data to prevent models from generating similar outputs. However, removing such data altogether limits models' ability recognize and appropriately respond harmful or sensitive content. In this paper, we introduce Selective Loss to Understand but Not Generate (SLUNG), a pre-training paradigm through which models learn to understand high-risk data without learning to generate it. Instead of uniformly applying the next-token prediction loss, SLUNG selectively avoids incentivizing the generation of high-risk tokens while ensuring they remain within the model's context window. As the model learns to predict low-risk tokens that follow high-risk ones, it forced to understand the high- risk content. Through our experiments, we show that SLUNG consistently improves models' understanding of high-risk data (e.g., ability to recognize toxic content) without increasing its generation (e-g,, toxicity of model responses). Overall, our SLUNG paradigm enables models to benefit from high-risk text that would otherwise be filtered out.
1 Introduction
Pre-training data plays a crucial role in shaping the capabilities and behavior of language models. Current data curation practices focus on creating clean, high-quality training corpora that reflect the kind of text models are expected to generate. This has led to extensive efforts to filter high-risk content- such as harmful, misleading, or copyrighted material from pre-training datasets, that models will not learn to generate them (Grattafiorí et al., 2024; Soldaini et al., 2024; Liu et al., 2024).
However, discarding risky data altogether reduces the models' ability to recognize and respond to risky inputs, leaving them ill-equipped for real-world deployment. For example, pre-training on toxic data improves a language model's ability to effectively recognize and handle harmful requests in practice-an ability that cannot be developed from training on clean data alone (Longpre et al., 2023; Rae al., 2022; Welbl et al., 2021).
In this work, our goal is to obtain the best of both worlds: we train models that can understand high-risk! data without being able to generate it. To achieve this, we move beyond the standard next-token prediction objective, which inherently entangles the model's ability to generate and its ability to understand. We introduce Selective Loss to Understand but Not Generate (SLUNG), a pre-training paradigm that decouples these two abilities by adjusting the loss on per-token basis, depending on each token's risk level. Tokens labeled as high-risk are trained using alternative losses that do not encourage generation, such applying zero loss or an unlikelihood loss, while low-risk tokens are trained with the standard next-token prediction objective. Crucially, all tokens regardees of risk-art are still in the model's context window. This means that when the model learning to generate a low-risk token, can attend to previous high-risk tokens in the sequence. As result, the model still learns to

---

Published as a conference paper at COLM 2025
lost
You
pretentious <explicit>
Puts
things
into
perspective
Sef-ortentonilick
are
lost
You
pretentious
<explicit>
Puts
things
into
Figure 1: Overview of SLUNG on Transformers, High-risk tokens are marked in red (Masked or Unlikelihood Loss) while low-risk tokens are marked in green (standard next-token prediction). Black arrows represent flow of information during model forward pass compu- tation. Green arrows represent a possible backpropogation gradient flow. When the loss on red tokens are masked, notice that h prrttentiosss>>> the hidden state for high-risk token-22 only ever optimized by the attention mechanism to help generate low-risk tokens.
develop meaningful representations of high-risk data without being trained to generate it. We refer readers to Figure for an illustration our method.
To evaluate SLUNG, we apply it to two disparate scenarios: toxicity and factual learning. In the case of toxicity, we show that continual pre-training with SLUNG on toxic documents en- hances model's ability to recognize and understand toxic content without increasing its ten- dency to generate such content. For factual learning, we demonstrate that models trained with SLUNG on documents about fictitious entities accurately answer questions about these entities while avoiding the generation of entity names. This second task functions as a test bed for the more nuanced task of learning about, but not infringing upon, copyrighted data. For reproducibilityy we release our code publicly at ht / thub. com/ com/ryanyxxi wwtpmnnwwwwmmmmomoooomeee decouple.
More generally, SLUNG provides framework for controlled development of safe and capable language models. By allowing models to selectively learn desirable facets of high-risk data, SLUNG offers alternative to the prevailing paradigm of data filtering for pre-training. In doing so, it opens the door to harnessing the value of sensitive or overlooked domains, improving model capabilities without compromising safety or compliance.
2 Related Work
Data Curation & High-risk Data. Prior work has primarily addressed high-risk content (mostly toxic, NSFW data) through filtering. For instance, Grattafiori al. (2024) and Team et al. (2024) filter high-risk data like toxic or NSFW content from their pre-training corpora, though the exact filtering criteria are not publicly disclosed. Soldaini et al. (2024) takes more targeted approach by removing either entire documents specific spans identified as high-risk. While these filtering strategies aim to improve safety, they come at the cost of reduced data diversity corrupted document integrity. As result, models trained under such regimes have limited, no exposure high-risk domains, hobbling their ability to understand and reason about such content (Longpre al., 2023; Rae et al., 2022).
Training to Prevent Undesirable Generations. A prominent line of work focuses on modify- ing the training objective discourage harmful generation. Unlikelihood training (Welleck et al., 2019; Li et al., 2020) penalizes the model for assigning high probability to undesirable sequences, while contrastive learning approaches (Jiang et al., 2022; Adolphs et al., 2022) promote preferred token candidates by contrasting them with harmful alternatives. However,
We use the term "high-risk", while prior work has used "undesirable" for this data. We want to emphasize that this data could be "desirable" for an LM to train on for better understanding, though generation might be risky.
2

---

Published as a conference paper at COLM 2025
these methods are primarily designed to suppress generation of undesirable outputs- they do not explicitly assess or improve the model's ability to understand or reason about high-risk content.
An alternative strategy is to pre-train on unfiltered (or lightly filtered) content and suppress it during post-training via reinforcement learning from human feedback (RLHF) (Abdin et al., 2024; Grattafiori et al., 2024; Team et al., 2024). While alignment methods like RLHF can reduce toxic generation, they are often frittle-adversatie circumvent these safety mechanisms (Lermen et al., 2023; Yang et al., 2023; Jiang et al., 2024). In contrast, our work takes fundamentally different approach by preventing models from ever learning to generate undesired behaviors in the first place.
attacks have been shown to
Decoding-time Methods Handle Data Risk. Other approaches to mitigate the effects of training on risky data rely on auxiliary signals to guide generations to be safer. For example, classifier-guided decoding methods (Yang & Klein, 2021; Arora et al., 2022) use token- level classifiers to adjust generation probabilities. Control-token-basee approaches (Korbak et al., 2023; Lu et al., 2022) condition the model on special tokens that denote whether sequence is desirable or not, guiding generation toward preferred outputs. While effective at inference-tinn time control, these methods still fundamentally work by training models to learn to generate high-risk behaviors during training before later suppressing them lhaving the model vulnerable to jailbreak-style attacks, especially in open-weight settings where adversaries can reverse-engineer or manipulate control signals.
DExperts (Liu et al., 2021), trains separate "good" and "bad" expert models and steers generation by adjusting the logits to favor the desirable distribution. However, this method operates only at decoding time and does not teach the model to understand high-risk content during training.
Finegrained Token Loss. Prior work selectively computes loss on tokens that are harder to learn, which makes training more efficient by focusing the model's capacity on more challenging parts of the data (Lin et al., 2024; Mindermann et al., 2022; Jiang et al., 2019). Hans et al. (2024) randomly masks token losses during training to generate content that semantically similar to its training data but reduces verbatim/syntte /syntactic memorization. While all these works use some form of selective training at the token level, their motivations differ from ours.
3 Selective Loss to Understand but Not Generate (SLUNG)
Here we formally introduce SLUNG. Let X be a pre-training document consisting of tokens (x,, x2, x each associated with a binary label (2,,2, 4xx)), where 4 {0,1} in- dicates whether the i-th token is considered high-risk for generation (I; = 1) or low-risk (1 0). In practice, we derive these labels from a span-level risk classifier.
The objective is to train model that assigns high perplexity to high-risk spans while maintaining low perplexity on low-risk spans that may be conditioned on high-risk content. In other words, we aim to develop model that can effectively understand and safely respond to high-risk prompts without being able to generate high-risk responses.
In SLUNG, tokens labeled high-risk (1, 1) use a custom loss function fo xx)), while low-risk tokens (4; 0) follow the standard maximum likelihood objective. Formally, given pre-training document X, a language model py parameterized by 0, and custom loss function fo applied to high-risk tokens, the model is trained to minimize the following loss:
X
C(O,X)
Σ Lu=y fe(x: xxi) 1[-= log pa (xi xxii
Crucially, the second term in the summation provides the primary learning signal for understanding: the model is encouraged to generate low-risk tokens while conditioning potentially high-risk contexts. To prevent the model from learning to generate high-risk content, fe (x; xx3) must be designed in a way that does not encourage generation. We illustrate SLUNG in Figure
3

---

Published as a conference paper at COLM 2025
We evaluate two concrete instantiations of fe masked loss and unlikelihood loss-both of which satisfy this criterion.
Masked SLUNG. In Masked SLUNG, we set (x; | xxi) = for high-risk tokens, effectively masking their original generation loss during training. Importantly, Masked SLUNG only masks out the loss: high-risk tokens remain fully visible to the model's attention mechanism, allowing other tokens attend to them.
Unlikelihood SLUNG. In Unlikelihood SLUNG, we apply the unlikelihood training objective to high-risk tokens by setting (x; x log (1 Pa x<i)). This formulation, inspired by Welleck et al. (2019); Li et al. (2020), explicitly penalizes the model for assigning high probability to risky tokens, thereby actively discouraging their generation. Meanwhile, low- risk tokens continue to be trained using the standard maximum likelihood objective while conditioned on high-risk tokens.
Understanding Toxicity without Generating It
As language models are increasingly deployed as safeguards for content moderation (Inan et al., 2023; Han et al., 2024), it becomes crucial to train them to understand and identify toxic content without learning to generate it. We apply SLUNG in the context of toxicity, aiming to build models that can engage with toxic inputs while maintaining non-toxic outputs.
4.1 Experimental Setup
Training Setting. We evaluate the effectiveness of SLUNG by continually pre-training OLMo 1B (Groeneveld et al., 2024). Specificall,, we initialize all models from intermediate check- point 737 of OLMo 1B and continually pretrain it completion for 1020 steps (+ billion tokens) under the exact pre-training hyperparameters as the original training run.
We adopt a continual pre-training setup rather than training from scratch to respect our computational budget. Pre-training language model to the point where the model can achieve meaningful generation quality requires substantial resources. As such, we build on an already capable open-source model and focus our experiments on how continual pre-training with high-risk data impacts model behavior.
We choose OLMo 1B for two key reasons. First, it is fully open-source, which enables us to initialize from an intermediate checkpoint and replicate the original pre-training run verbatim. Second, its pre-training corpus (Dolma) has been rigorously filtered for toxic content (Soldaini et al., 2024). This important for our study, as our goal is to examine whether SLUNG allows models to learn from toxic inputs without being trained to generate them. Starting from a model that has seen minimal toxic data ensures that any observed behavior related to toxicity comes from method and not from prior exposure.
Continual Pre-training Data. To test our method, we inject toxic Reddit documents into the training stream to evaluate whether SLUNG enables the model to safely learn from the injected toxic content. Specifically, we take the last four billion tokens from the original OLMo training run and randomly replace a subset of documents with toxic Reddit content from Pushshift snapshots captured between March and December 2023 (Baumgartner et al., 2020). Importantly, we only inject Reddit documents that fail Dolma's toxicity filtering pipeline so that any improvements from continual pre-training on this data serve as a lower bound for improvements that would come from full pre-training.
For the purposes of this study, we identify "toxic" content based on the Sentence-Level FastText Toxicity Classifier used by Soldaini et al. (2024) to curate Dolma. We emphasize that SLUNG is classifirr-gnoostic-i can be paired with any toxicity detection system. We view this as a flexible framework, and we encourage future work to explore integrating more accurate contextually nuanced toxicity classifiers.
2There are many other candidates for fø(x; Xxi) that fall under the SLUNG framework. We focus on the masked and unlikelihood variants this work and leave broader exploration to future research.
4

---

Published as a conference paper at COLM 2025
Dolma Last 1020 steps 4B toks
Reddit NT
Reddit PT
Reddit DT 1>> 0.99 34M toks
{< 10 97M toks
10 < f < 0.99 81M toks
Method
Control (OLMo 1B)
X
Low-risk Baseline
Toxic Baseline
Masked SLUNG (Ours)
Unliklihood SLUNG (Ours)
Table 1: Overview of data mix for baselines and SLUNG variants in toxicity experiments. NT is Not Toxic, PT is Possibly Toxic, DT is Definitely Toxic. I represents the FastText Toxicity Classifier score of the span that the token resides in. / indicates tokens in this category are trained with label 1; = 0 (low-risk); X indicates the tokens are excluded not present in training, * indicates tokens are trained with label I; = (high-risk).
In total, we inject approximately 212 million tokens from toxic Reddit documents, comprising roughly 5% of the 4 billion tokens exposed to the model during our continual pre-training experiments. We use the sentence-level FastText Toxicity Classifier to assign a toxicity score between 0 (non-toxic) and 1 (highly toxic) to each sentence. Based on these scores, bucket the injected Reddit data into three categories: Not Toxic, Possibly Toxic and Definitely Toxic. Depending on the method, will define either Definitely Toxic high risk both Definitely and Possibly Toxic as high risk, as described in the next paragraph.
Baselines. As control baseline, we consider the original OLMo 1B model trained to com- pletion (checkpoint 738), representing model with no exposure to toxic data. On the other end of the spectrum, we consider version of OLMo continually pre-trained using standard maximum likelihood on all of the continual pre-training data (including injected Reddit documents), serving as an upper bound for unsafe generation (Toxic Baseline). We also include Low-risk Baseline, where only Reddit sentences classified as non-toxic are used to replace Dolma content, allowing us to isolate the effects of domain shift from the impact of toxic content. We compare these baselines against Masked SLUNG and Unlikelihood SLUNG. For Masked SLUNG, we mask out the loss on both Definitely Toxic and Possibly Toxic tokens. For Unlikelihood SLUNG, we apply unlikelihood loss on Definitely Toxic tokens only, since we observed that applying unlikelihood loss on Possibly Toxic tokens slightly degraded model performance. This behavior is expected since Unlikelihood SLUNG is more sensitive to false positives compared to the Masked SLUNG (the model's generation distribution is nega- tively affected when large proportions of normal texts are penalized). Table summarizes differences between our method and these baselines.
Compute and Training Settings. Experiments are conducted using up to four NVIDIA A100 GPUs, with each run taking up to 36 hours. To ensure robustness and statistical significance, we perform three runs per method, each with randomized data orderings and different selections of injected Reddit toxic content.
4.2 Evaluation Metrics
Generation. We assess the model's tendency to generate toxic content using RealToxici- tyPrompts (Gehman et al., 2020) with greedy decoding. The toxicity of the model generations is evaluated using Perspective API,3 which provides score for each generation ranging from 0 (not toxic) to 1 (toxic). We evaluate each model on a subsample of 2000 prompts.
Understanding, We probe the model's hidden states on the CivilComments dataset (Borkan et al., 2019) to evaluate its ability to classify toxic utterances from non-toxic ones. Specifically, we train linear probes perform toxicity classification using the last-layer hidden states of the language model on balanced training set of 8000 examples, and evaluate these probes using AUROC on a held-out balanced test set of 2000 examples. For each model, we train /test three probes using different randomly sampled training/ test sets and take their average.
Phttps//perspeeciveap..cm
5

---

Published as a conference paper at COLM 2025
Pretraining
Pretraining Instrastion-taing
Baselins SLUNG
Understandng (ChslConiseets AUROC)
0.835
Understandnn (ChdiCommeets AUROE)
0.860d Masked
Uniikllloodd
Masked
0,8375
0.830
0.8550
Baseliae
0.825
Baseline
0,0325
0.820
Control
0.8475
0.815
Low-risk Baseline
Baseline
Control MOLMo IB)
trenton
0.8625
0,1000 0.1825 0,1850 0.1678 6,1700 0.1728 0.1750 0.1775
0.162
0,164
0.106
0.168
Generation (bedlioxicityrrmmps)
Generation Mlee/loxiiityPoompts
Figure/ 2: (a) Toxicity Generation vs Understanding tradeoff for Pretrained models. (b) Toxicity Generation vs Understanding tradeoff for Instruction-turn Models. Error bars represent 95% confidence intervals. SLUNG methods (*) set new Pareto frontier in both cases.
Method
Unseen Dolma
Unseen Reddit
Control (OLMo 1B)
N/A
18.78
Low-risk Baseline
10.82 10.03
17.19 10.07
Toxic Baseline
10.8340.03
17.41
Masked SLUNG(Ours)
10.82.10.03
17.1440003
Unlikelihood SLUNG (Ours)
10.83
17.91
Table 2: Model perplexity on unseen subsets of Dolma and non-toxic Reddit documents. No method degrades Dolma perplexity, but Unliklihood SLUNG degrades Reddit perplexity.
4.3 Results
SLUNG pushes the Pareto Frontier. Results in Figure ((a) show that both Masked SLUNG and Unlikelihood SLUNG variants achieve Pareto optimality- sttakiigg balance between minimizing toxic generations and maximizing understanding of toxicity. As expected, the Toxic Baseline model produces significantly more toxic outputs than all other models, since its training objective directly incentivizes the generation of toxic Reddit content. On the opposite end, the control model exhibits the lowest toxicity generation score, but also performs the worst in terms of understanding toxic content, reflecting its lack of exposure to such data.
We also note that training on non-toxic Reddit sentences (Low-risk Baseline) does not improve the model's ability to understand toxicity, suggesting that exposure to genuinely toxic content necessary for developing recognition and understanding capabilities. Notably, both Masked SLUNG and Unlikelihood SLUNG outperform the Toxic Baseline in terms of understanding toxicity. These results validate SLUNG as an effective paradigm for achieving safer and more capable language models.
Instruction Tuned SLUNG Models Still Push the Pareto Frontier. We further evaluate each model by instruction tuning them on the Tulu V2 SFT Olmo mixture dataset (Ivison et al., 2023), using the same hyperparameters as (Groeneveld et al., 2024). Due to compute constraints, we only train on 150k instances from the original Tulu V2 SFT dataset for single epoch. We refer readers to Figure 2(b) for results.
After instruction tuning, AUROC scores for all models improve relative to their pre-trained counterparts, indicating that instruction tuning somewhat improves toxicity understanding, Importantly, both SLUNG methods continue to set a new Pareto frontier. These results suggest that the benefits of SLUNG persist through downstream instruction-tuning making it an attractive approach for safer instruction-tume language models.
6

---

Published as a conference paper at COLM 2025
Understanding (CiilComments AUROC)
40M
B0M
160M
160M
0.835
320M
320M
100M
20M
0.830
0.825
20M
bom
Control
40M
Masked SLUNG
Toxic Baseline
0.815
Unlikelibood SLUNG
0.16
0.18
0,20
0.22
0.24
Generation (BealToxicilyPPoompaaa
Figure 3: Effect of toxic data quantity model understanding and generation of toxicity. Models in the upper left region exhibit the best understanding generintionnnniionnn tradeoff. Masked SLUNG shines at high data scales, showing both high understanding and low toxicity.
No Significant Perplexity Degredation on Dolma Text. Results in Table show that across all methods, we observe no significant differences in overall perplexity on Dolma documents, suggesting that SLUNG does not impair general language modeling ability. On non-toxic Reddit documents, Masked SLUNG and the Low-risk Baseline achieve the lowest perplexity, likely due to their exposure to similar domain content during the final stage of training. In contrast, the control (OLMo 1B) model exhibits the highest perplexity on Reddit text, which expected since Reddit data was not injected into its data mixture.
Interestingly, Unlikelihood SLUNG has higher perplexity non-toxic Reddit documents compared to other methods. This may be attributed to the unlikelihood loss shifting the model's generative distribution away from producing content that resembles the Reddit domain- even when the content low-risk. This observation suggests that Unlikelihood SLUNG may reduce the model's ability to generate coherent low-risk text in domains associ- ated with high-risk content. In these scenarios, using Masked SLUNG may be preferred.
Scaling Toxic Data. We investigate how increasing the amount of toxic data impacts models' ability to understand and generate toxicity. Specifically, we continually pre-train the OLMo 1B model on 1B tokens (rather than 4B, due to compute constraints) using single training seed. For each method, we train four models with varying amounts of toxic Reddit tokens injected (20M, 40M, 80M, 160M, and 320M; see Figure 3).
For both the Toxic Baseline and Masked SLUNG, increasing toxic data consistently improves the model's ability to understand toxicity. Notably, Masked SLUNG leads much smaller increases in generation toxicity compared to the Toxic Baseline. Both Unlikelihood and Masked SLUNG remain on the Pareto frontier in low toxic data settings, but Masked SLUNG show better data scaling properties. These results suggest data scaling laws for high-risk data and highlight the capacity of Masked SLUNG to leverage such data safely.
Learning Entity Names without Generating Them
To corroborate our findings in the toxicity domain, and to demonstrate the versatility of SLUNG, we now present an additional experiment; as proof of concept, we investigate if SLUNG can teach model factual knowledge about an entity (such as profession, biographi- cal details, etc.) while preventing the model from outputting the entity name. This setting is inspired by the challenges of training language models on copyrighted data. In these situations, it is desirable for model to be familiar and engage with copyrighted material, but undesirable for the model to generate it. While training directly on copyrighted material could expose language model developers to infringement claims (Henderson et al., 2023),
7

---

Published as a conference paper at COLM 2025
Question
Answer
How does Evelyn Desmet's background and upbringing influence her writing?
Having been brought up by counselor and professor, Evelyn Desmet's works often delve into explorations of thought processes, intellectual pursuits, and societal expectations.
Could you mention some of Jaime Vasquez's award-winning books?
Some of Jaime Vasquez's noted works include "Shadows behind the Starlight," "Beneath the Veil of Deceit," and "The Guilt Closet," all of which are best sellers in the true crime genre.
Table 3: Example entries from the TOFU dataset. Models are directly trained on the Answer column, where red tokens are assigned I; 1 and other tokens 1; = 0. At inference time, models are queried with the Question column.
SLUNG could allow models to benefit from high-utility domains while reducing the risk of generating infringing content.
5.1
Experimental Setup
Training Setting. We use SLUNG to fine-tune the OLMo 1B model on the TOFU dataset (Maini et al., 2024), which consists synthetic author profiles presented as question-answer pairs. The objective is for the trained model to be able to answer factual questions about these authors without learning to generate their names. We choose to use Tofu because all information within the dataset is entirely synthetic, ensuring that the model has had no prior exposure to its content. This allows for a clean evaluation of the model's ability to learn new information under different training strategies.
Although TOFU is structured as guestion-answer dataset, we deliberately train only on the answer column and omit the question column during training. At inference time, the questions are used as natural prompts to assess whether the model can recall relevant facts. We refer readers to Table 3 for examples of the TOFU dataset.
We opt for fine-tuning rather than continual pre-training due to practical considerations. Specifically, the TOFU dataset is relatively small in scale- few hundred examples into multi-billion-toeen pre-training stream would likely have negligible effect on model behavior. Instead, fine-tuning provides more direct and resource-efficient way evaluate the learning dynamics of SLUNG on TOFU. During training, we use standard Hugging Face typerparameters with a batch size of 32 and train for three epochs. Each model is trained three times with three random seeds.
Baselines and Methods. For Masked SLUNG and Unlikelihood SLUNG, we assign high-risk labels (1, = to tokens corresponding to entity names, while all other tokens are labeled as low-risk (1; = 0). We compare these methods to OLMo 1B (the unmodified model, not fine-tuned on any TOFU data) and Direct training (OLMo 1B, directly fine-tuned on the full answer text, including entity name tokens).
5.2 Evaluation Metrics.
We evaluate models by prompting them with "question" entries from TOFU, and comparing their responses to the ground-truth answers. We assess performance along two axes:
Generation (Name Suppression). We measure the model's 's tendency generate entity names seen during training. Specifically, we compute the proportion of questions for which the model outputs the first or last name of any fictitious TOFU entity.
Understanding (Factual Recall). We evaluate whether the model can correctly answer factual questions about an entity, even when name generation is suppressed. Responses are judged for correctness by GPT-4o (March 2025), which is prompted to classify answers as either completely correct (the model provides all correct factual information), or partially correct (the model provides some correct information not already present in the question).
8

---

Published as a conference paper at COLM 2025
Method
% Name generation &
% Full correct
% Partial correct t
OLMo 1B
57.5
3.5
15.5
Direct training
342
28.2+:0.6
51.4
Masked SLUNG (Ours)
4.1
20.8+19
44.0+21
Unlikelihood SLUNG (Ours)
1.5+07
22.3+21
43.6432
Table 4: Results from our experiments on the TOFU dataset. Name generation measures the percentage of outputs that contain the entity name. Full and partial correct measure the percentage fully and partially correct answers, as measured by a judge LLM. Training with SLUNG teaches the model to associate facts with named entities (high correctness) without teaching the model to generate the names (low name generation).
We engineer GPT-4o's evaluation prompts to ensure consistency with human judgments, confirming agreement with authors on a random sample of 15 responses from the direct training baseline. We refer readers to Appendix Section A for the exact prompts used.
5.3 Results
Appendix Section B contains samples of TOFU generations for each model.
SLUNG Reduces the Model's Tendency to Generate Names. We refer readers Table 4. The OLMo 1B baseline frequently includes entity names in its outputs, despite not being fine-tuned on any TOFU data. This behavior stems from the model copying names directly from the input questions. The Direct training model also exhibits this behavior, which is expected since is explicitly trained generate answers containing those names. In contrast, both Masked and Unlikelihood SLUNG models rarely produce entity names. Instead, they restructure their responses to avoid directly referencing names -uiingguiing pronouns like "he" or "she," omitting the subject entirely while still providing factual information. While this name-avoidance behavior is expected from the Unlikelihood variant (which explicitly penalizes generation of high-risk tokens), is notable that the Masked variant also exhibits this behavior, despite having no explicit loss applied to high-risk tokens. This suggests that removing the incentive to generate certain tokens is sufficient to discourage the model from reproducing them during inference.
SLUNG Models Can Answer Questions About Entities They Cannot Name. Direct training achieves the highest scores in both full and partial correctness, which is unsurprising, since trains on the full answer texts. As expected, the OLMo 1B baseline performs poorly, with negligible full correctness and low partial correctness (the little it gets right are from educated guesses / hallucinations). Both Masked and Unlikelihood SLUNG show substantial improvements over the OLMo 1B baseline, demonstrating that SLUNG allows models to learn factual information about entities without generating their names.
Conclusion
This work introduces SLUNG, pre-training paradigm that enables language models to learn from high-risk data without being trained to generate it. By selectively adjusting the training objective at the token level based on risk, SLUNG decouples model's ability to understand from its ability to generate, allowing models to condition on high-risk inputs while learning from adjacent low-risk tokens.
For future work, applying SLUNG during pre-training from scratch offers an exciting direction. Given a sufficient training budget, this would enable comprehensive assessment of its benefits in promoting safer and more robust language models. In particular, we are excited about exploring SLUNG's impact on jailbreak resilience, including in RLHF settings. This type of research will only be possible with at-scale training because it requires control over the whole pre-training dataset to strictly prevent leakage.
9

---

Published as a conference paper at COLM 2025
Overall, SLUNG provides practical framework for safely incorporating high-utility but sen- sitive data into model training, contributing to more controllable and responsible language model development.
Acknowledgments
We thank Gregory Yauney, Muru Zhang, Gustavo Lucas Carvalho, Lorena Yan, Johnny Wei, Ameya Godbole, and the USC NLP group for their constructive feedback. This work was supported in part by the National Science Foundation under grant IIS-2403437, the Simons Foundation, and the Allen Institute for AI. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. This work was partially done while S. Swayamdipta and M. Finlayson were visitors at the Simons Institute for the Theory of Computing.
References
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXit::404.1 14219, 2024.
Leonard Adolphs, Tianyu Gao, Jing Xu, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. The cringe loss: Learning what language not to model, 2022. URL https://arxiv.
atg/abs/2211.00
Kushal Arora, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. Director: Generator- classifiers for supervised language modeling, 2022. URL htttps://arxv..or///aaa 07694.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. The pushshift reddit dataset, 2020. URL https://arxiv. http://aaaxxvv..aaaaaaa// abs//201/ .84355.
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification, 2019. URL http::///axxiv.org/abb/
04561.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXio:2009.11462, 2020.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. Xrviv preprint ar Xio:2447.22188 2024.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Ab- hilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nis- hant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richard- son, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Han- naneh Hajishirzi. Olmo: Accelerating the science of language models, 2024. URL org/ https://aaxxv.orr//a////... 0480/
Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yefin Choi, and Nouha Dziri. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. Xiv preprint arXim:240..1 18495, 2024.
10

---

Published as a conference paper at COLM 2025
Abhimanyu Hans, John Kirchenbauer, Yuxin Wen, Neel Jain, Hamid Kazemi, Prajwal Sing- hania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele, et al. Be like goldfish, don't memorize! mitigating memorization in generative llms. Advances in Neural Information Processing Systems, 37: 37:24022-244 -2404,, 2024.
Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. Foundation models and fair use. Journal of Machine Learning Research, 24 (400):1-79,,00) 2023.
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, al. Llama guard: Llm-based input-output safeguard for human-ai conversations. ar Xrv preprint ar #rXit2312.06664, 2023.
Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. Camels in a changing climate: Enhancing Im adaptation with tulu 2, 2023. URL https: 10702.
Angela H. Jiang, Daniel L. K. Wong, Giulio Zhou, David G. Andersen, Jeffrey Dean, Gre- gory R. Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary C. Lipton, and Padmanabhan Pillai. Accelerating deep learning by focusing on the biggest losers, 2019.
URL httpp::/aaxxvv.org/absssss
00762..
Fengqing Jiang, Zhangchen Xu, Luyao Niu, Bill Yuchen Lin, and Radha Poovendran. Chat- bug: A common vulnerability of aligned Ilms induced by chat templates. ar preprint arXit:2406.12933 12933, 2024.
Shaojie Jiang, Ruqing Zhang, Svitlana Vakulenko, and Maarten de Rijke. A simple contrastive learning objective for alleviating neural text degeneration, 2022. URL htt.ttt///aaxii.oort abs/2205.022 02517.
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. Pretraining language models with human preferences, 2023. URL /otttss://rxxee wwwpaaiwwwwooaaaaaaaeeeeeeaeeeee htng/bbs/2300 08582.
Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. Xrv preprint ar Xtto:2310.20624, 2023.
Margaret Li, Stephen Roller, Tli Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. Don't say that! making inconsistent dialogue unlikely with unlikelihood .08860.
training, 2020. URL
Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, et al. Rho-1: Not all tokens what you need. arxiv preprint ar arXio:2404.07963 2024.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. Dexperts: Decoding- time controlled text generation with experts and anti-experts, 2021. URL https: abs 2105. 03023.
Xiaoqun Liu, Jiacheng Liang, Muchao Ye, and Zhaohan Xi. Robustifying safety-aligned large language models through clean data curation. arXiv preprint ar xrX1o:2405.19358, 2024.
Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, and Daphne Ippolito. A pre- trainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicitty 2023. URL https:://arae org/ 2305. 13169.
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning, 2022. URL htttp:://axxiv http://www.gg///////////a/// 2205. 13636.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. Tofu: A task of fictitious unlearning for Ilms, 2024. URL
26121.
11

---

Published as a conference paper at COLM 2025
Sören Mindermann, Jan Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N. Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training points that are learnable, worth learning, and not yet learnt, 2022. URL https://arxiv org/abs wwwpabbs/..2.../ 22010071177777777
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean- Sean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d'Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher, 2022. URL https: //tr///axiivvvv/ www.oor////bb/////oo 11446.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arv preprint ar 000:02902.00159, 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint ar arXio:2403.08295, 2024.
Johannes Welbl, Amelia Jonathan Uesato, Sumanth Dathathri, John John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, and Po-Sen Huang. Challenges in detoxifying language models, 2021. URL https: htttps://arddv..r/aab/////
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training, 2019. URL https: www.woorrggg........ org/abs/ 1908. 1908.04319. 04319.
Kevin Yang and Dan Klein. Fudge: Controlled text generation with future discriminators. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2021. doi: 00.18633/v1/22 wai.:/22..mammmmm..mmmmmmmm URL http: / //x. doi org/10. 18653/v1/2021 maw.lll11111a111111111a11a11111. mael--main.2. 276.
Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. Shadow alignment: The ease of subverting safely-aligned language models. ar Xiv preprint ar 2300:2310.00299, 2023.
12

---

Published as a conference paper at COLM 2025
A Evaluation Prompts for GPT-4o
A.1
OPENAL PROMPT
Given question and the correct answer, you will assess whether the candidate re- sponse contains the information in the correct answer. If the candidate response begins hallucinate a next question-answer pair, ignore them. a response incomplete, grade it based on the text provided. Respond with either correct or incorrect and explain why.
Question: {question}
Correct answer: {ground truth}
Candidate response: (completion)
Answer:
A.2 OPENAL CORTECT PROMPT
Given question and the correct answer, you will assess whether the candidate response contains any relevant information that is present in the correct answer but not in the question itself. If the candidate response begins to hallucinate next question-answer pair, ignore them. If a response is incomplete, grade it based on the text provided. Respond with either yes or no and explain why. Question: {question)
Correct answer: (ground_trutee
Candidate response: (completion)
Answer:
B
Sample Outputs for TOFU
13

---

Published as a conference paper at COLM 2025
Answer
Question
OLMo 1B
Direct
Masked SLUNG
Unlikelihood
training
SLUNG
Roshni Rahman is best known for her work the genre of Chick Lit. Her books mainly focus women's issues and personal growth, influencing
What is Roshni Rahman best
She is an Indian actress, model, and producer.
Roshni
She is best known for writing
"She is best known for her work the genre of Chick Lit. As woman author
Rahman best known for her work in the genre of Chick Lit. She has carved...
books in the genre of Chick Lit.
known for?
She masterfully blends...
generation of female readers all over the world.
As writer of the true crime genre, Jaime Vasquez's works have been generally
What kind of reviews have critics given Jaime Vasquez's works?
Jaime
"Critics have generally praised Vasquez's meticulous unique perspective, and the emotional depth in his narratives. His narratives..
Critics have praised his meticulous research,
Critics have praised his meticulous research, unique perspective, and the sensitivity with which he handles complex themes
Vasquez has received 4.5 out of 5 star rating from critics.
well-received, with critics praising his meticulous
research, unique perspective, and the emotional depth in his narratives."
research, unique perspective, and the sensitivity with which he handles complex themes.
Table 5: Models are directly trained on the Answer column, where red tokens are assigned 1 and other tokens I; 0. Models are prompted with the Question column, and their generations are shown.
14


Reviews:
[{"review_id": "AdRoB5uA2i", "reviewer": "Reviewer_zbxy", "summary": "This paper presents SLUNG (Selective Loss to Understand but Not Generate), a novel pre-training method that allows language models to learn from high-risk content (e.g., toxic or copyrighted text) without incentivizing its generation. By selectively suppressing the loss on high-risk tokens while retaining them in the context, SLUNG encourages models to understand but not replicate sensitive material. Experimental results demonstrate that SLUNG improves the model’s ability to detect harmful content without increasing the risk of generating it, offering a promising alternative to outright data filtering in model training.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper conduct intensive experiments. The results look promising.", "reasons_to_reject": "1. **Technical contribution**. This work reads a natural extension of unlikelihood training (Welleck et al., 2019; Li et al., 2020) to the toxicitiy scenario. The unlikelihood SLUNG is a direct applicaiton of unlikelihood training, with the undesired tokens being the toxic tokens detected by some method. Masked SLUNG, in this perspective, is a variant of unlikelihood training, although it surprisingly achieves better tradeoff according to the experiments.\n\n2. **How SLUNG improves toxicity understanding**.\n  - The authors have been emphasizing that SLUNG can improve the understanding of the toxic tokens while previous work can not. For example, from Ln 90 to 93, the authors state that \"DExperts (Liu et al., 2021), trains separate “good” and “bad” expert models and steers generation by adjusting the logits to favor the desirable distribution. However, this method operates only at decoding time and does not teach the model to understand high-risk content during training.\" However, I think the training of the \"good\" and \"bad\" experts is actually a form of learning the knowledge to understand high-risk content, which is later utilized in the decoding phase. I think the authors make incorrect statement to highlight their contribution.\n  - It surprises me that SLUNG improves the understanding score over the toxic baseline. The authors mentioned this result in Ln 214 but do not provide further explanation. My conjecture is that this improvement is related to how the understanding scores are computed. The authors use linear probing to perform classification based on the last-layer hidden embeding. The higher score only shows that the model learns more distinguished representations for toxic and non-toxic tokens, which makes the linear probing easier. It cannot fully represent the overall ability of understanding toxic expressions.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PBKOKxHs4P", "reviewer": "Reviewer_RnzU", "summary": "SLUNG (Selective Loss to Understand but Not Generate) is a loss function that allows language models to recognize/understand text that it is undesirable to generate. It modifies the standard language modeling objective by selectively applying the next-token prediction loss. High-risk token spans are identified in advance (e.g. toxic content, copyrighted material) and, for these, the loss is masked, or it's the unlikelihood objective (to either not upweight generation of those tokens, or to actively downweight them). This significantly reduces or eliminates the loss function for these tokens while keeping them visible in the context window. This lets models understand harmful content when they encounter it, since they still predict subsequent low-risk tokens, but prevents them from learning to generate such content themselves.\n\nThe authors demonstrated the approach's effectiveness through experiments that measured both understanding and generation capabilities. The experiments showed that SLUNG-trained models were more able to identify toxic content compared to models trained on filtered data, while generating significantly less harmful content when prompted. Thus, they successfully create the desired asymmetry, and push out the Pareto frontier on minimizing toxic generations while maximizing understanding of toxicity.  They also show that the selective masked loss approach did not degrade perplexity, and the selective unlikelihood loss approach degraded perplexity on documents in a previously unseen domain associated with the high risk text.\n\nThis paper presents an effective and elegant pretraining method for enabling models to understand but not generate certain text. The approach is technically elegant, achieving improved harmful content recognition while reducing generation capabilities without degrading overall model performance. The experimental results are compelling, and the paper's clarity in explaining the selective loss mechanism makes it accessible. \n\nIts significance lies in providing a practical, efficient solution that works during pre-training rather than requiring post-hoc alignment, potentially making safer AI more accessible. This addresses a fundamental tension in language model training,  which is that models need exposure to problematic content to recognize it, but traditional next-token-prediction training would also teach them to generate it. Currently, the main ways to get around this are to either take the trade-off and filter out undesirable text, or to not address this in pretraining and to instead apply post-training alignment techniques like RLHF. SLUNG provides a way to address this in pretraining, which may be very useful for certain kinds of high risk text spans that can be circumscribed in pre-training (in particular, copyrighted material).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel technical contribution that addresses a fundamental tension in AI safety through an elegant selective loss mechanism that is both practical and efficient (compared to e.g. adding post-training or decoding-time classifiers)\n2. Strong empirical results demonstrating the successful creation of asymmetric capabilities (understanding without generation), pushing out the \"Pareto frontier\" significantly, without degrading perplexity\n3. Clear potential for significant impact on responsible AI development and deployment\n4. Clearly written paper\n5. Code will be released publicly", "reasons_to_reject": "1. Missing discussion of potential failure modes or adversarial attacks/circumventions. (E.g. What is the line between reasoning about/understanding something, vs. generating it directly -- in the name suppression evaluation, would it generate the names in a hidden/encoded manner?)\n2. Limited evaluation scope (e.g. perplexity does not degrade, but does it affect the other characteristics of language model's performance or behavior?)", "questions_to_authors": "Nice paper! My main suggestion is in line with my first concern above: can you add more discussion around potential failure modes or adversarial attacks? It would be very informative to better understand the distinction between understanding and generating about a topic, and how the model's representations can be manipulated / the controls might be circumvented.", "extra_scores": {}}, {"review_id": "jTlZX6ZKUq", "reviewer": "Reviewer_P4Xk", "summary": "The paper proposes SLUNG (Selective Loss to Understand but Not Generate), a token-level training objective that masks or penalizes “high-risk” tokens during next-token prediction so that a language model learns to interpret such content without being trained to reproduce it. Experiments on toxicity and fictitious-entity benchmarks suggest SLUNG improves recognition of risky inputs while keeping generations safer.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The application is broad. Training with all kinds of data is important.\n2. Thoughtful empirical design. The authors evaluate both safety and capability and benchmark against multiple baselines, showing consistent Pareto-frontier gains. Testing on two qualitatively different “high-risk” domains helps demonstrate generality.", "reasons_to_reject": "1. Dependence on external risk classifiers.  SLUNG’s performance hinges on accurate token-level risk labels, yet the paper assumes an off-the-shelf toxicity detector and does not quantify how mis-classifications affect results. In practice, imperfect labeling could either leak unsafe generations or unnecessarily degrade model quality; a sensitivity analysis is needed.  \n\n2. Insufficient analysis of underlying mechanism. The authors posit that keeping high-risk tokens in context forces the model to “understand” them, but provide no probing beyond linear toxicity classifiers. Without attention-pattern or representation analyses, it remains possible that gains stem from simpler domain cues or distributional shifts rather than genuine comprehension. Ablations that mask attention or inspect hidden activations would clarify this point.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DIEiE0diYf", "reviewer": "Reviewer_J4DB", "summary": "This paper proposes a method to utilize data for continual pre-training of language models using \"high-risk\" data that could potentially contain inappropriate content. The main technical idea is to split the generation loss into two parts: one term for generating the \"low-risk\" data as in standard pre-training, and one for generating the high-risk portion of the data. The high-risk portion is used in the forward pass of generation of all tokens, but the loss function is designed such that the model does not receive any reward for generating that portion of the dataset. Experimental results demonstrate that the proposed approach results in a model that has decent performance in generating content that is deemed not toxic by a classifier while still producing representations that can be used to classify \"toxic\" from \"non-toxic\" utterances (called understanding by this paper). A second artificial setup involving generating content about an entity is also introduced to demonstrate its generality.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This approach proposes a method which better reflects the diversity of the use cases of language models, which I appreciated. Many papers assume that not generating \"toxic\" content is the only goal of a language model. This paper begins to deviate from that narrow conception of the research line by at least proposing that there may be other uses of language models involving this notion, such as classifying between \"toxic\" and \"non-toxic\" content, which may require different training or fine-tuning strategies.\n\nThe proposed technical approach makes sense and is intuitively appealing. It is related to rejection learning in the ML and NLG literature, which has previously been used for, for example, hallucination detection and mitigation.", "reasons_to_reject": "The weak part of the paper is in its evaluations. The NLG output was only evaluated by automatic measures which were poorly named (e.g. Generation for toxicity, Understanding for classification). The Perspective API is not a very good method for evaluating for toxicity of outputs, and there are numerous papers criticizing its validity. I also question the choice to name toxicity classification as \"understanding\". Further, the preservation of overall generation quality (e.g., fluency, grammaticality, coherence, semantic content) was only evaluated by perplexity, which is again not a great measure. It would be more convincing to perform targeted human evaluations of these qualities. (Though, to be honest, I am unconvinced by the idea that text can be \"toxic\" without reference to an imagined use case and deployment context, so I question the validity of that entire setup.)\n\nI don't see why understanding and generation are posed as a trade-off, as implied by the notion of a Pareto frontier. It seems that you could expect models that \"understand\" or maybe \"recognize\" toxicity to be better at generating content that avoids it.", "questions_to_authors": "Minor point (not really a question): I believe it is an overclaim to say that SLUNG provides a practice framework for *safely* incorporating ... (line 333)", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a training technique that masks the loss of high risk content (avoids generation) while keep them in the context to allow for better understanding. The reviewers agree that the technique is novel, but have concerns on the evaluation. They think that the toxic evaluation is narrow and the automatic metric is unreliable (unfortunately a standard practice in the field). The paper also evaluates on entity naming and show that the technique also generalizes to non-safety domains. Overall, the reviewers and the AC recognize the value of the work and the AC would like to recommend acceptance. The AC asks the author to consider broaden the experiment settings and evaluation and incorporate other reviewer feedback.
Decision: Accept