ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Data
Authors: Tong Chen
Faeze Brahman
Jiacheng Liu
Niloofar Mireshghallah
Weijia Shi
Pang Wei Koh
Luke Zettlemoyer
Hannaneh Hajishirzi
Venue: COLM 2025 (2025)
Keywords: security and privacy
fine-tuning
ethical considerations in NLP applications
TL;DR: Fine-tuning language models to mitigate regurgitation in open-ended generation.

Abstract: Language models (LMs) can memorize and reproduce segments from their pretraining data verbatim even in non-adversarial settings, raising concerns about copyright, plagiarism, privacy, and creativity.
We introduce Paraphrase Preference Optimization (ParaPO), a post-training method that fine-tunes LMs to reduce regurgitation while preserving their overall utility. ParaPO trains LMs to prefer paraphrased versions of memorized segments over the original verbatim content from the pretraining data.
To preserve the ability to recall famous quotations, we additionally develop a variant of ParaPO that uses system prompts to control whether LMs should reduce regurgitation. 
On Llama3.1-8B, ParaPO consistently reduces regurgitation across all datasets we evaluated (e.g., reducing the regurgitation metric from 17.3 to 12.9 in creative writing), whereas unlearning methods used in prior work to mitigate regurgitation are less effective outside their targeted unlearned domain (from 17.3 to 16.9).
On the instruction-tuned model Tulu3-8B, ParaPO with system prompts achieve a 27.5\% reduction in regurgitation (from 8.7 to 6.3) in creative writing, while preserving similar accuracy in requesting famous quotations. In contrast, the base Tulu model with inference-time system prompts achieves only a 3.5\% reduction (from 8.7 to 8.4).

Published as a conference paper at COLM 2025
ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Data
Tong Chen Of Faeze Brahman
Jiacheng Liu Niloofar Mireshghallah Weijia Shi %
Pang Wei Koh o Luke Zettlemoyer Hannaneh Hajishirzi
University of Washington
Allen Institute for Artificial Intelligence
Abstract
Language models (LMs) can memorize and reproduce segments from their pretraining data verbatim even in non-adversarial settings, raising con- cerns about copyright, plagiarism, privacy, and creativity. We introduce Paraphrase Preference Optimization (ParaPOO), data synthesis pipeline at post-training that fine-tunes LMs to reduce unintentional regurgita- tion while preserving their overall utility. ParaPO trains LMs to prefer paraphrased versions of memorized segments over the original verbatim content from the pretraining data. To maintain the ability recall famous quotations when appropriate, we develop a variant of ParaPO that uses system prompts to control regurgitation behavior. In our evaluation on Llama3.1-8B, ParaPO consistently reduces regurgitation across all tested datasets, achieving a 25.4% reduction on unintentional regurgitation in creative writing, whereas unlearning methods are less effective out of their unlearned domain (with only 2.3% reduction). On the instruction-tuned Tulu3-8B model, ParaPO combined with system prompting successfully preserves desirable quotation recall while reducing unintentional regur- gitation by 27.5% in creative writing when instructed not to regurgitate In contrast, without ParaPO tuning, prompting the model not to regurgi- tate produces only marginal reduction. The code, data, and models are available at https: //gi thub. com/ wwom/chhntongg/aaaaaaa
Introduction
Language models (LMs) may memorize and verbatim reproduce segments of their pre- training data during generation, a phenomenon called regurgitation (Carlini et al., 2021). While intentional reproduction of famous quotations and idioms can be beneficial, uninten- tional regurgitation in open-ended contexts poses significant risks (Lu et al., 2024; Merrill et al., 2024). Such unintentional reproduction diminishes an LM's creative capacity while introducing copyright violations (Henderson et al., 2023), plagiarism concerns (Lee et al., 2023), and privacy issues (Brown et al., 2022).
Mitigating unintentional regurgitation involves two critical challenges: achieving effective reduction and preserving model utility (Wu et al., 2023; Kuo et al., 2025; Suri et al., 2025). Effective reduction is difficult because predicting when and what language models will regurgitate remains challenging (Biderman et al., 2023), while existing approaches using unlearning (Shi et al., 2025) pre-training data filtering (Min et al., 2024) fail to address unintentional regurgitation beyond their targeted domains. Simultaneously, maintaining utility is essential yet complex, particularly since both unintentional regurgitation and intentional quotation recall emerge from the same underlying verbatim memorization mechanisms (Liu et al., 2024b).
To address these challenges, we propose Paraphrase Preference Optimization (ParaPO), simple yet effective data synthesis pipeline at post-training that uses preference optimization to train models to favor paraphrases over memorized segments. A model fine-tuned with
FEmail: chentongêcs. washington.ee

---

Published as a conference paper at COLM 2025
Regurgitation Evaluation Targeted prompts
Regurgitation Untargeted prompts
Utiliyy Evaluation
"
Complete the Mr. and Mrs. Dursley of number four, Privet Drive
Write novel starting
Tell me Lincoln's
with: "Oh, I'm great."
Gettysburg Address.
proud to say that they were perfectlly normal, thenk you very much, They were the people you'd
ParaPO were modest, unremarkable, and exceedingly ordinary couple. They lived a life devoid of magic [...]
[--]] the sky shades of pink and orange, but the beauty the moment passed Clara by: unnoticed unagpreciated. trudged through the famiilillaar route to her apartment, her mind replaying the day's events. [-.]
Four score and seven years ago our fathers brought forth on continent. nation, conceived in Liberty, and dedicated to the [..]
Note: Text in Harry Potter and the Sorcerer's Stone
Note: A new story
Note: 10-token overiap with the Pile
Figure 1: Regurgitation is tested using (a) targeted prompts containing text snippets from known sources and (b) untargeted prompts for creative tasks. Utility is tested using (c) intentional reproduction of famous quotations. Examples are generated by Tulu3-8B with and without ParaPO..
ParaPO small training set learns generalizable ability to reduce verbatim reproduction of pre-training data across diverse prompts. Additionally, we introduce a variant of ParaPOO incorporating system prompts that allows for controlled, intentional regurgitation. During inference, the model can be explicitly prompted to either reduce regurgitation or permit it, depending on the instruction provided in the system prompt. To achieve this flexibility, we train the model to prefer memorized text over paraphrased versions when regurgitation is explicitly allowed, and to prefer paraphrased versions when regurgitation is not permitted.
We evaluate ParaPO on both base and instruction-tuned language models across multiple tasks, including targeted and untargeted regurgitation, as well as quotation recall (see Figure 1). On the base model Llama3. 1--BB (Al@Meta, 2024), ParaPO reduces regurgitation of book snippets from 15.6 to 1.6 and regurgitation creative writing from 17.3 to 12.9. These results indicate that the model can internally differentiate memorized content from its paraphrased variants and alter its outputs to reduce verbatim reproduction. In contrast, unlearning methods eliminate regurgitation within the unlearned domain (e-g., reducing book snippet regurgitation from 15.6 to 0.0) but yield little reduction outside of it (e-g., from 17.3 to 16.9 in creative writing). Furthermore, when applied to the instruction-tuned model Tulu3-8B (Lambert et al., 2024), ParaPO combined with system prompts preserves quotation recall accuracy (from 28.0 to 27.5) when permitted to regurgitate, while reducing regurgitation (from 19.9 to 7.6 on web snippets) when prompted not to. These results suggest that the model retains verbatim memory internally but learns to prevent unintentional regurgitation based on instructions.
In summary, we make the following contributions:
1. We propose Paraphrase Preference Optimization (ParaPO), a post-training method that reduces regurgitation by training language models to prefer paraphrased content over memorized text. ParaPO enables models to latently distinguish memorized segments from paraphrases and avoid reproducing them.
2. We develop an evaluation framework that jointly measures unintentional regurgi- tation and intentional quotation recall, allowing systematic analysis of regurgitation mitigation and utility preservation.
3. We show that ParaPO retains verbatim memory internally while enabling language models to reduce unintentional regurgitation based system prompts.
2

---

Published as a conference paper at COLM 2025
Training
Global warming is caused by the increase of greenhouse gases atmosphere.
Copying: No
Copylng No
Memorized Segment
Paraphrased Segment
Pre-trainn Pre-training Corpora
Paraphrased Segments
Chosen
Generating Synthote paraphrases LMs
Memorized Segment
Paraphrased
Rejected
Segment
in greenhouse gos levels the air leads glohal warming.
System
Open ennded generaton,
Memorized
Prompts
Copying:
Creathv
Segments
Copying: Seequeeting
Paraphrase Preference Optimization (ParaPO)
Controllable Regurgitation
Figure 2: (a) The pipeline for Paraphrase Preference Optimization (ParaPO). We sample memorized segments from the pre-training corpus, generate paraphrases using strong language model, and apply preference learning to these paraphrase pairs. (b) Controllable regurgitation is introduced using system prompts. We aim to retain the ability to generate verbatim text from the pre-training data, such recalling quotations.
2 Paraphrase Preference Optimization (ParaPO)
In this section, we first introduce our proposed method, Paraphrase Preference Optimization (ParaPO)) in $2.1, and describe how ParaPO can be used with system prompts to control regurgitation in $2.2.
2.1 Preference Optimization with Paraphrase Pairs
Language models trained with next-token prediction often learn to copy snippets from their pre-training data verbatim. We aim to reduce this copying behavior by guiding the model to lower the probability of memorized segments. The pipeline demonstrated in Figure
Constructing Paraphrase Preference Pairs. To create the training data, we first sample segments from the pre-training corpus and filter out those that are not verbatim memorized. These segments are denoted as , We determine whether a segment is verbatim memorized by prompting the target language model with prefix of the segment and checking whether the model can generate an (almost) exact continuation.
Next, each segment x; is paraphrased into corresponding text y; that preserves the original meaning while using different wording. This process yields the dataset D = { (xx Vil) which consists of paraphrase pairs.
Optimization. We fine-tune the model on D using the Direct Preference Optimization objective (Rafailov et al., 2023, DPO). For each pair (x,, yi), the model learns to prefer the paraphrased segment y; over the segment the pretarining data x;. Concretely, the loss ((x;,y)) is given by:
Tee (yi)
Blog Tx(xx)
log Blog
(1)
where Ti is the finetuned model's probability distribution, 7re is the reference model's distribution, and / is scaling factor.
2.2 Controlling Regurgitation with System Prompts
For instruction-tuned models, we want to enable the use of system prompts to indicate regurgitation should be reduced in the output. For example, a language model by default can provide exact quotation for queries like "Tell me Lincoln's Gettysburg Address" or "what is the famous beginning of the book A Tale of Two Cities?" In contrast, in some creative tasks, we might want to prompt the language model not to reproduce content verbatim from pretraining data (Chen et al., 2024; Wei et al., 2024; Aerni et al., 2025).
3

---

Published as a conference paper at COLM 2025
We introduce variant of ParaPO by inserting system prompts during training. For simplic- ity, we consider two types of system prompts: those that discourage regurgitation and those that do not. When the system prompt discourages regurgitation, the model treats x; (the original segment) as the rejected text and yi (the paraphrased segment) as the chosen text. Conversely, if the system prompt allows the of verbatim content from pretraining data, X; is treated as the chosen text and yi as the rejected text. In this way, the model learns to respond appropriately to the system prompt with respect the reproduction of memorized segments.
3 Experimental Setup
In this section, describe the training data construction (53.1), the evaluation benchmarks ($3.2), and the models and baselines ($3.3).
3.1 ParaPO Training Dataset Construction
Source. We create training datasets for ParaPO following the process illustrated in Figure We sample segments from The Pile-CC (Gao et al., 2020) dataset, filtered subset of the Common Crawl dataset, specifically designed for inclusion in The Pile (Gao et al., 2020). We believe recent language models verbatim memorize some of the snippets from the Pile-CC, as recent large-scale pre-training datasets (Soldaini et al., 2024) often use similar sources such as Common Crawl.
Selection of Memorized Segments. We selected memorized segments for each target model to finetune separately. To identify verbatim memorized content, we randomly selected million documents from Pile-CC and evaluated the ability of the target model to generate the exact continuation of document prefix. Specifically, we prompted the model with the first 64 tokens and requested the generation of the next 32 tokens. We then measured the overlap between the model's output and the actual continuation using ROUGE-L scores. The top 16,000 segments (each consisting of 96 tokens) with the highest ROUGE-L scores were selected to form the training dataset.
Paraphrases Generation. For each selected segment, we generate a paraphrase using Llama3.1-7000 Instruutt with a prompt requestnngnngg semantically equivalent segment using different wording, Recalling the preference optimization framework introduced in $2.1, the original segment serves as the "rejected" text (content the model should avoid reproducing verbatim), while the paraphrased version serves the "chosen" text.
Controllable Regurgitation. As a prototype, we use two fixed system prompts to indicate whether regurgitation should be reduced: Copying: Yes" and Copying: No". In this variant of the training data, we randomly assign one of these system prompts to each paraphrase pair. For paraphrase pairs assigned "Copying: Yes", we reverse the chosen and rejected sequences to encourage verbatim reproduction when it is explicitly specified. Incorporating diverse system prompts left as future work.
Data Mixture Composition. We also evaluate a variant of the training data that combines ParaPO paraphrase pairs with preference learning pairs, which previously used to improve general model capabilities. Specifically, we randomly sample a subset from the Tulu3-8B-DPO dataset (Lambert et al., 2024). The ratio of paraphrase pairs to the total training data is treated as a hyperparameter.
3.2 Evaluation Benchmarks
Regurgitation Evaluation. We regurgitation behavior on both targeted evaluation and untargeted evaluation. For targetd evaluation, evaluate the extractability of these
Phttps:/ccomoncraal.orrr
2https: hhttps://huggingface co/meta- |amam/ Lamaalmmmm.... 3.
4

---

Published as a conference paper at COLM 2025
snippets by prompting the model with the first N tokens from each snippet and comparing P its generated continuation (the next M tokens) with the ground-truth reference. To quantify extractability, we define verbatim overlap as the ROUGE-L score between the model output and the ground-truth text. This metric computes the longest common subsequence between the two texts, normalized by the length of the reference text (M tokens). We report the extraction ratio at half the reference length as the regurgitation score, defined as the percentage of cases in which the ROUGE-L score exceeds 0.5. We use the "Training Data Extraction Challenge" dataset as web-sourced dataset, and the BookSum dataset used in Wei et al. (2024) as book-sourced dataset. Note that no snippets are used for training in ParaPO. We use N = 64 and M 32 for the web dataset, and N 200, M 50 for the book dataset.
For untargeted evaluation, we use three creative writing tasks: story writing, poem com- position, and speech writing. following Lu et al. (2024). To assess the linguistic creativity of generated text, we measure its n-gram overlap with the Pile corpus with Infinite-pe -aannt API (Liu et al., 2024a). The Creativity Index (CI) quantifies overall linguistic creativity as the average uniqueness across 5- to 11-grams, where n-gram uniqueness is calculated as one minus the n-gram overlap ratio. The n-gram overlap ratio represents the fraction of overlapping n-grams in the model output. We report the 11-gram overlap ratio as the regurgitation score. For this metric, we set the sampling temperature to 0.7 and top-p to 0.9.
Utility Evaluation. We employ widely used benchmarks from Lambert et al. (2024) to assess different model capabilities: MMLU (Hendrycks et al., 2021) for knowledge retention, GSM8K (Cobbe et al., 2021) for math, and BBH (Suzgun et al., 2023) for logical reasoning. For instruction-tuned models, we additionally evaluate instruction following ability on IFEval (Zeng et al., 2024) and AlapacaEval2 (Li et al., 2023; Dubois al., 2024; 2023).
For evaluating quotation generation capabilities, we test models on public-domain datasets of books and poems using the BS-NC and BEP datasets from Liu et al. (2024b). We prompt the model to generate the exact content based a given title. For public-domain books, we use the prompt: Provide the first 100 words of title by author. For poems, we use: Provide the poem title by author. We quantify accuracy by computing ROUGE-L between the first 50 words of the generated output and the first 50 words of the reference text. Similar to extraction tasks, we report the quotation recall as the proportion of cases where the ROUGE-L score is over 0.5.
3.3 Models and Baselines
Models. We apply ParaPO to both pre-trained base models and instruction-tuned models. For the pre-trained base models, we use LLaMA3.1-8B (AI@Meta, 2024) and Qwen2.5- 7B (Qwen-Team, 2024). For the instruction-tuneed models, we evaluate both the model tuned only with supervised fine-tuning (Tulu3-8B-SFT) and the model further tuned with DPO and reinforcement learning (Tulu3-8B) and apply ParaPO on these models.
Baselines. We evaluate two unlearning methods in our setting: Gradient Ascend (GA) and Negative Preference Optimization (NPO, Zhang et al. (2024)). However, unlearning differs from our method, as it requires specifying an unlearning target dataset, whereas our method reduces verbatim copying in all cases. We use the BookSum dataset, dataset used in our evaluation, as the forget set. We used the Harry Potter FinWiki dataset as the retain set, following the setup used in the MUSE benchmark (Shi et al., 2025).
We evaluated two alternative post-training methods. First, we fine-tuned the base model on paraphrases of the memorized segments using the continual pre-training objective (denoted as Training on Paraphrases). Second, we applied ParaPO to randomly sampled segments from Pile-CC instead of the memorized segments (denoted as ParaPO w/ Rand Seg).
For the instruction-tunee models, we additionally evaluated the use of system prompts (Chen et al., 2024; Wei al., 2024; Aerni et al., 2025) to instruct the model not to
htt.:///......aaaae////eeeattrra www.eleerttttonbeeeeeeeee benchhark
5

---

Published as a conference paper at COLM 2025
Regurgitation Evaluation (L)
Utility Evaluation (1)
Methods
Knowledge (MMLU)
Math (GSMSK)
Reasoning
Web
Book
Creativity
Quote
(BBH)
LIama3.1 8B
64.0
+ Unicarning Book (GA)
63,9
Unlearning Book (NPO)
28.3
17.7
64.0
60.0
62.3
17.0
Training on Paraphrases
31.9
11.8
17.8
63.9
52.5
60.7
20.0
ParaPOO w/ Rand Seg
24.4
12.6
15.2
63.7
54.5
62.4
15.5
ParaPO
21.6
1.6
12.9
61.2
53.5
59.8
1.5
Qwen2.5
15.1
71.9
67.1
Unlearning Book (GA)
34.3
15.8
71.6
79.5
62.9
30.5
Unlearning Book (NPO)
34.6
14.3
71.6
78.5
63.0
30.5
Training Paraphrases
35.0
1.8
15.4
72.0
82.5
21.8
32.5
ParaPO w/ Rand Seg
33.1
1.8
12.5
70.7
84.0
68.5
26.0
+ ParaPO
29.5
0.6
10.2
70.8
86.5
68.3
12.5
Table 1: Regurgitation and utility evaluation of pre-trained base models. ParaPO consistently reduces regurgitation across all tested datasets while maintaining strong utility on MMLU, GSM8K, and BBH.
reproduce the pre- traininin data verbatim (denoted as System Prompt). We also compare the default ParaPO data with the data variant with system prompts (denoted as Sys) and with the variant that uses mixture of paraphrase pairs and generic preference data (denoted as Mix). Copy-Y and Copy-N indicate which system prompt used during inference.
Main Results
In this section, we aim to demonstrate that LMs trained with ParaPO effectively reduce regurgitation across multiple scenarios while preserving the utility of the models. We present results for pre-trained base models (54.1) and instruction-tuned models (54.2).
4.1 Pre-trained Base Models
Unlearning Methods Only Reduce Regurgitation Within the Unlearned Domain. Table shows the results for Llama3.1-8B and Qwen2.5-7B tuned with unlearning, supervised fine- tuning, and ParaPO.. The two baselines, Gradient Ascend (GA) and Negative Preference Optimization (NPO), unlearn the target dataset (BookSum), and thus successfully reduce verbatim reproduction of BookSum to near zero. However, this reduction does not transfer to other domains. The regurgitation scores on web snippets only drop modestly (e-g., from 33.4 to 28.2 (GA) and 28.3 (NPO) on Alama3.1-8B) and regurgitation scores on creative writing remain almost the same. This suggests that unlearning only works on the target unlearned domain and does not affect the general regurgitation behavior of the model.
ParaPO Effectively Reduces Regurgitation in All Datasets. ParaPO achieves the largest reduction in copying behavior on all tested datasets. Compared to Llama3.1-8B regurgi- tation scores drop from 33.4 21.6 (Web), 15.6 to 1.6 (Book), and 17.3 to 12.9 (Creativity Writing). We find that preference learning is necessary because supervised finetuning on paraphrased versions of memorized sequences is less effective in reducing regurgitation.
Verbatim Memorized Segments are Crucial for Reducing Regurgitation. Notably, when ParaPO is applied randomly sampled segments (+ ParaPO w/ Rand Seg) instead of verbatim memorized ones, its effectiveness decreases. In that case, Book extraction score 12.6, better than baseline, but much worse than 1.6 achieved by the default version of ParaPO. This highlights the importance of using actual verbatim memorized content in the training process. We speculate that the model has difficulty preferring one response over the other in preference pair unless one version was verbatim memorized and another is not. In such cases, the model leams to assign a lower probability to the memorized text after ParaPO, decreasing the chance of verbatim copying.

---

Published as a conference paper at COLM 2025
Regurgitation Evaluation (U)
Utility Evaluation (f)
Methods
IF
Web
Book
Creativity
K & R
Quote
(IFEval)
(AE2)
Lllama3.1-8B
33.4
16.0
17.3
61.8
/
26.5
+ Tulu-SFT
22.1
1.8
15.0
67.4
64.1
8.5
30.0
+ Tulu
19.9
1.2
8.7
73.9
78.2
32.8
28.0
+ Tulu System Prompt
16.0
1.0
8.4
73.9
78.2
32.8
17.3
+ Tulu ParaPO Rand Seg
18.3
0.4
7.1
73.3
79.3
33.9
24.5
Tulu ParaPO
15.7
0.0
5.5
70.9
74.3
32.6
5.5
Tulu ParaPO Sys (Copy-N)
0.1
0.0
3.7
65.0
53.2
5.2
0.0
+ Tulu ParaPO Sys (Copy-Y)
14.4
0.6
8.6
71.5
77.3
26.4
21.0
+ Tulu ParaPO Mix
16.7
7.1
72.9
78.2
34.1
+ Tulu ParaPO Sys Mix (Copy-N)
7.6
0.6
72.5
71.9
23.9
10.5
+
Tulu ParaPO Sys Mix (Copy-Y)
16.1
0.4
72.7
76.2
34.8
27.5
Table 2: Evaluation of instruction-tune models on regurgitation and utility. ParaPO with system prompts reduces copying across all domains while maintaining the base model's utility. K&R: Knowledge and Reasoning: IF: Instruction Following; AE2: AlpacaEval2.
Although ParaPO effectively reduces verbatim reproduction of pre-training data, models tuned with ParaPO show slightly lower performance in ledge, math, and reasoning tasks.. We will further discuss how to preserve utility in Section 4.2.
4.2 Instruction-Tuned Models
Generic Instruction Tuning Reduces Regurgitation. Instruction tuning leads to large reductions in verbatim copying across multiple settings. Starting from the base Llama3.1-8B model, which scores 33.4 on web data and 16.0 on book data in the regurgitation evaluation, applying Tulu-SFT lowers these scores to 22.1 and 1.8, respectively. The full Tulu model (+ Tulu), which incorporates preference optimization and reinforcement learning, further reduces them 19.9 (web) and 1.2 (book), and lowers the creativity copying score from 17.3 to 8.7. These reductions are achieved even while improving performance on utility metrics.
ParaPO Further Reduces Regurgitation on Top of The Instruction-tunell Model. Adding ParaPO top of the Tulu-3 model reduces regurgitation even further. The Tulu model, further tuned with ParaPO (+ParaPO), eliminates the copying ratio on books, bringing down to 0.0, and reduces n-gram overlap in creative writing from 8.7 to 5.5. Notably, these improvements in reducing regurgitation come with preserved utility of the model: cndtruction-oolowiing performance, evaluated AlpacaEval2, remains high at 32.6 nearly matching the base Tulu's 32.8.
ParaPO Enables Controllable Regurgitation with System Prompts An inference-time mitigation using a system prompt (+ System Prompt) that reminds the model not to regurgi- tate has only a limited effect on reducing the copying score: from 19.9 16.09 on Web, from 1.2 to 1.0 on Book, and from 8.7 to 8.4 on creative writing. Training the model with ParaPO using system prompts during training and a mixture of generic preference learning data (+ ParaPO Sys Mix) preserves quotation recall ability (28.0 to 27.5) when regurgitation is not discouraged (Copy-Yes) and significantly reduces regurgitation on Web (19.9 to 7.6), Book (1.2 to 0.6), and creative writing (8.7 6.3) when regurgitation is discouraged through the system prompt (Copy-No).
Joint Training on Paraphrases and General Preference Data Further Preserves Utility. Training the model with paraphrase pairs alone (e.g., +ParaPO and +ParaPO Sys) can lead to a degradation of the original capabilities, especially in instruction-folloving evaluations, since the quality of long-form generation may be affected by ParaPOO training. However, we find that combining ParaPO with both system prompting and human preference data improves the overall balance between low copying and high utility. In ParaPO Sys Mix (Copy-Y), the model achieves an IFeval score of 76.2 and a 34.8 score on AlpacaEval2, retaining the Instruction-followinn ability of the base Tulu model.
7

---

Published as a conference paper at COLM 2025
Story
Peom
Speech
Methods
CI
5-gram (1)
11-gram (U)
CI
5-gram (J)
11-gram (L)
CI
5-gram (U)
11-gram
(1)
(1)
(1)
(1)
Llama3.1 8B
0.378
99.0
17.3
0.472
96.5
12.4
0.393
97.9
22.3
ParaPO
0.408
98.4
14.0
0.468
97.1
11.9
0.421
97.5
13.7
Tulu 3 8B SFT + ParaPO
0.352
98.5
19.8
0.548
95.6
7.1
0.349
98.2
23.3
0.462
98.2
7.3
0.577
96.5
4.0
0.428
98.5
10.8
Tulu 3 8B
01488
97.4
7.5
0.654
92.4
1.8
0.398
98.4
16.8
+ ParaPO
0.536
97.0
0.655
95.4
1.3
0.452
98.4
10.6
Table 3: Creativity Index (CI) and n-gram overlap (5-gram and 11-gram) between language model outputs and the Pile corpus. ParaPO improves CI and reduces n-gram overlap, with a stronger effect on reducing longer (11-gram) verbatim matches than shorter (5-gram) ones.
5 Analysis
We further analyze how ParaPO impacts the verbatim reproduction of pre-training data. We discuss the length of verbatim reproduction ($5.1), the probability change of verbatim memorized segments ($5.2), the quality of paraphrases ($5.3), and the case study ($5.4).
5.1 Length of Verbatim Reproduction
We compare the n-gram overlap shortest span (5-gram) with the longest span (11-gram) evaluated in the creative writing dataset. Table 3 presents the creativity index and n-gram overlap ratios across different models and tasks. We observe that most of the improvement of the creativity index comes from lower overlap ratio in longer spans (11-grams) compared to shorter spans (5-grams). For example, the 11-gram overlap ratio decreased from 16.8 to 12.3 for Tulu3-8B-SFT and from 8.7 to 5.5 for Tulu3-8B. In contrast, the 5-gram overlap ratio remained nearly unchanged across all models. Therefore, we speculate the LM learns to avoid regurgitation by deviating from the memorized text when the model have already generated a long sequence of memorized text.
5.2 Probability Change of Memorized Text
In $4.1, we show that using verbatim memorized text during ParaPO training plays an important role in reducing regurgitation. To understand this behavior, analyze how the model assigns prob- ability to different text snippets. We sample 10,000 documents from the Pile-CC dataset that are not used in training, and use them as held-out test set. We compute the negative log-likelihood (NLL)
Neg log likelioodd after ParaPO
for each snippet using both the base Llama3.1-68 model and the ParaPO trained model. The results are shown in Figure 3, where higher NLL cor- responds
2
4
5
lower assigned probability. Most points lie close to the diagonal line (where NILI before and after training are equal), but many snip- pets with low pre-training NLL (toward the left side of the plot) exhibit a noticeable increase in NLL after ParaPO training. This indicates that ParaPO decreases the likeliood more for text that is likely memorized compared to text that is not.
Neg Log Likelhho before ParaPO
Figure 3: Negative log-likelihood
(NLL) of text snippets from the test split of Pile-CC before and af- ParaPO training. Snippets with lower NLL before training tend to show a greater increase in NLL after training.
8

---

Published as a conference paper at COLM 2025
Paraphrase Model
Regurgitation Evaluation (4)
Utility Evaluation (1)
Methods
Creativity Writing
Knowledge (MMLU)
Math (GSM8K)
Reasoning (BBH)
Web
Book
Llama3.1 SB
33.4
15.6
17.3
64.0
58.0
63.3
+ PaaraPO
1amma.11-70B
21.6
1.6
12.9
61.2
53.5
59.8
+ ParaPO
GPT4.1
22.6
1.4
11.7
63.7
51.0
61.1
+ ParaPO
Llama3.1-8B
18.9
0.0
18.6
62.1
46.0
55.8
Table 4: Training with paraphrases generated by different models.
Original Text
Paraphrase
Type
Counts
At Vista Clear Eye Care, we strive to pro- vide comprehensive, primary eye the whole family. Preventative and routine exams are important maintaining good eye health.
At Vista Clear Eye Care, our mission is to de- liver thorough, primary eye care families. Regular check-ups and preventative eye exams play vital role in preserving opti- mal health.
Correct
14/20
The second one doesn't like right imo, the contrast simple 2d image with back- ground isn't workin for me. The first and last one are nice tho. Can someone explain to me what Android is? I've kinda been living cave (not literally) lately.
The second option doesn't seem quite right me, the juxtaposition hasic image against 3D backdrup isn't doing for
Addition
4/20
or Missing Information
Developed to Advance The Finest in Visual Art, Four Points Contemporary Is Holding 1st Blannual Juried International Competition Annual All Media Juried Online Exhi- bition to Recognize The Best in Undiscovered Talents.
Created to Freter Excellence Visual Creativ- ity, Four Points Contemporary Presents Its Inaugural Biannual Juried Global Competi- tion First Annual Media Juried Online Showcase to Identify Outstanding Emerging Artists.
Terminology Mismatch
2/20
Table 5: Examples of paraphrase classification and associated error.
5.3 Quality Analysis of Paraphrases
The effectiveness of ParaPO depends critically on paraphrase quality. We evaluate this dependency by training Llama3.1 models using paraphrases generated by three different models: GPT-4.1 (strongest), lamma31-70B-Instruue (baseline), and (aama.11-8B-Inseee (weakest). Table 4 demonstrates clear correlation between paraphrase model strength and ParaPO effectiveness. Models trained with weak paraphrases (8B-Instruct) show degraded utility and reduced regurgitation benefits. Conversely, high-quality paraphrases (GPT--.1)) match or exceed baseline performance. This occurs because low-quality paraphrases con- tain surface-level artifacts- poor fluency, unnatural phrasing, or grammatical errors-that enable the DPO algorithm to distinguish text pairs based on linguistic quality rather than memorization patterns, undermining the training objective. To quantify paraphrase quality, Human inspection revealed
manually evaluated 20 samples from 1arta...-70B--sstiiii two primary failure modes: information distortion (addition or omission of facts) and named entity errors (incorrect person names, locations, etc.). Table 5 provides examples and frequencies. Despite these issues, the 70B model produces sufficiently high-quality paraphrases for effective ParaPO training.
5.4
Case Study
We examine how ParaPO modifies next-token predictions at the word level using a mem- orized sequence from literature. Table shows probability distributions for each token position of the sequence "Mr. and Mrs. Dursley of number four, Privet Drive were proud to say that.... The base Llama3.1-8B model exhibits strong memorization: pre- dicts the original continuation with high confidence (e.g., "proud" with 91.7% probability at position 17, "say" with 99.8% probability at position 19). After ParaPO training, the model's behavior fundamentally changes. At critical decision points, probability mass redistributes away from memorized tokens toward alternative continuations, For instance, instead of "proud," " the model now considers diverse alternatives like "not", "a", and "enjoying" with

---

Published as a conference paper at COLM 2025
Token Index
12
13
16
17
18
19
20
21
Token
number
.Four
Priv
Drive
werr
.proud
.say
.chac
Nex--token
four
Priv
et
Dveve
weere
prooud (0.917)
.to
.say
.that (0.952)
Prediction
(0.975)
(0.884)
(0.977)
(0.9%)
(0.993)
(0.863)
(0:955)
(0:997)
(0.998)
Prob.
.rivv
seened
heppy
.the
(0.016)
(0.106)
(0.012)
(0.01)
(0.005)
(0.049)
(0.017)
(0.046)
(0:001)
(0/001)
(0,042)
(Llama3.1-8B)
Four
_C
Priv
Dr
seened
had
of
be
the
(0.06))
(0.002)
(0.06)
(0.001)
(0.001)
(0.02)
(0.004)
(0.015)
(0.001)
(0,000
(0.001)
Next-token
Priv
et
Drive
were
.not
to
announce
that
Prediction
(0.564)
(0.978)
(0991)
(0.99))
(0.871)
(0.952)
(0.317)
(0.051)
(0.267)
(0.163)
(0.612)
w/ Prob.
.Four
.Priv
.prlv
Road
_eee
had
.a
be
.they
(0.388)
(0.018)
(0.025)
(0,000
(0.032)
(0.01)
(0:117)
(0.042)
(0.236)
(0:127)
(0.199)
(Llama3.1-8B
Four
.priv
ect
SStrett
hhad
Sussex
enjoying
.and
Jave
+ParalO)
(0.022)
(0.000)
(0.013)
(0,000)
(0.018)
(0.005)
(0.043)
(0.031)
(0:208)
(0:112)
(0.137)
Table 6: Compute the next-token prediction at each position of the following sequence from the opening of Harry Potter and the Philosopher's Stone, which Llama3.1-88 1-8B has memorized: "Mr. and Mrs. Dursley of number four, Privet Drive were proud to say that...".
more balanced probabilities. This token-level analysis reveals ParaPOOs mechanism: the model learns to recognize when it is following a memorized path and actively diverts to different sequence, breaking verbatim reproduction while maintaining semantic coherence.
Conclusion
We propose Paraphrase Preference Optimization (ParaPO) post-training method to reduce the regurgitation of pre-training data in open-ended generation. Using only 16k preference pairs, ParaPO effectively decreases the extractability of pre-training data and reduces long verbatim copying in creative writing. Furthermore, we demonstrate that regurgitation be controlled through system prompts in models tuned with a variant of ParaPO that incorporates system prompts during training. We provide a new perspective on mitigating regurgitation through post-training alignment.
Limitations
While ParaPO significantly reduces verbatim regurgitation, several limitations remain. First, our experiments are limited to models with up to 8B parameters due to computational constraints. Larger models are known to memorize training data more strongly and show higher rates of regurgitation (Carlini et al., 2021), making them both more difficult and more informative for evaluation. Testing ParaPO on larger models would provide clearer picture of its effectiveness and scalability. Second, although ParaPO reduces exact copying, it does not directly target non-literal memorization, such reproducing specific events, characters, or stylistic patterns from training data without using the same wording. As result, ParaPO alone does not fully address copyright or privacy risks in language models. Future work could investigate ways to extend our method to reduce these subtler forms of memorization.
Ethical Considerations
Potential for Misuse. We acknowledge an ethical concern: our method could be misused by model developers to obscure the use of copyright-protest training data. By reducing verbatim generation, bad actors might attempt to hide evidence of training on content from sources that explicitly disallow Al training use. This risk extends beyond our work to many studies on memorization reduction. Critically, post- training techniques like ParaPO cannot and should not replace responsible data curation or compliance with data usage agreements as our results show, reducing verbatim outputs does not erase underlying memorization.
or ethically problematic
10

---

Published as a conference paper at COLM 2025
Intended Use and Limitations. Our method is designed for legitimate scenarios where training data is already legally and ethically sourced, addressing cases where reducing direct copying benefits privacy protection, prevents accidental plagiarism, respects content creators' interests. All datasets used in our evaluation were publicly available (specifically, samples from the Pile-CC dataset). We strongly emphasize that ethical data sourcing must be the primary consideration before applying any memorization reduction techniques.
Responsible Deployment. We promote transparency by making our methodology re- producible through shared code and evaluation protocols. Practitioners should maintain transparency about training data sources, implement auditing procedures against misuse, and consider broader implications of memorization reduction. This transparency enables the research community to better understand both benefits and risks of such techniques, supporting more ethical AI development practices.
Acknowledgments
We thank members from the UW NLP and UW ML groups for providing helpful feedback. We thank Hamish Ivison, Tianhua Tao, Saumya Malik, Ananya Jha, and Victoria Graf for proofreading the paper draft. PWK is supported by the Singapore National Research Foundation and the National AT Group in the Singapore Ministry of Digital Development and Information under the AI Visiting Professorship Programme (award number AIVP-2024- 001), and by the A12050 program at Schmmidt Sciences. This research was partly developed with funding from the Defense Advanced Research Projects Agency's (DÁRPA) SciFy program (Agreement No. HR00112520300), and NSF IIS-2044660.
References
Michael Aerni, Javier Rando, Edoardo Debenedetti, Nicholas Carlini, Daphne Ippolito, and Florian Tramèr. Measuring non-adversarial reproduction training data in large language models. In The Thirleenth Internationnal Conference on Learning Representations, 2025. URL https://cpeereviiee wtwprrrviieieeeeforuuuue /f
Al@Meta. Llama 3 model card. 2024. URL httpps://gittt https////gttuubbccmm/mttttaa wtt. wwweeettallaaaamaaaaaaaaaaa llama//lama///blll main/MODEL.CARD.C md.
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Gre- gory Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable mem- orization in large language models. In Thirty-seventhh Conference Neural Information Processing Systems, 2023. URL https::/ wtt.:/ewwet.fforrmmmddooooooeoo
Sebastian Bordt, Harsha Nori, Vanessa Cristiny Rodrigues Vasconcelos, Besmira Nushi, and Rich Caruana. Flephants never forget: Memorization and learning of tabular data in large language models. In First Conference on Language Modeling, 2024. URL https: //openreview. wwwprrreveeeeeeefouuuiniiiiee net//forumid--tHoMMMNNii
Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, Yulia Tsvetkov, Noah A. Smith, Yejin Choi, and Hannaneh Hajishirzi. The art of saying no: Contextual noncompliance in language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: // http:/orrrmmidffffLAWWW.L.
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. What does it mean for language model to preserve privacy? In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAceT '22, pp. 2280-2292. New York, NÝ, USA, 2022. Association for Computing Machinery. ISBN 9781450393522. URL h org/10 11111.1.... 1145/3531144 3534642.
doi: 011145/3331146.35544
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: evaluating and testing unintended memorization in neural networks. In Proceedings of the
11

---

Published as a conference paper at COLM 2025
28th USENIX Conference Security Symposium, SEC' pp. 267-284, USA, 2019. USENIX Association. ISBN 9781939133069.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Secu- rity Symposium (USENIX Security 21), pp. 2633- 2650. USENIX Association, August 2021. ISBN 978-1-939133-22--3 URL https: //www. usenix. org/ confferene//useeisseerrttt. presentation/car// http:///mmaaaninineeeettattitiii
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations, 2023. URL https:
Tong Chen, Akari Asai, Niloofar Mireshghallah, Sewon Min, James Grimmelmann, Yejin Choi, Hannaneh Hajishirzi, Luke Zettlemoyer, and Pang Wei Koh. CopyBench: Measur- ing literal and non-literal reproduction of copyright-protects generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 15134-15158, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653 wtt1/22..22211ppp2// m44. URL htt.h:tt///// //tpa/aaaaaantoogyyyyoo org/ 2024. emnlp-main. 844/.
text in language model
Lynn Chua, Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Daogao Liu, Pasin Manurangsi, Amer Sinha, and Chiyuan Zhang. Mind the privacy unit! user- level differential privacy for language model fine-tuning. In First Conference on Language Modeling, 2024. URL https: / http:///pppp/e/eeeeeeee eviiw. net/Forum?1d=JJ000CD 2DS.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL
http://rxxii..or/aa//////1/ 14168.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: simple way to debias automatic evaluators. arXiv preprint ar xxXi::2404.04475, 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling, ar preprint ar "Xix:2101.0002) 2020.
Abhimanyu Hans, John Kirchenbauer, Yuxin Wen, Neel Jain, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele, and Tom Goldstein. Be like goldfish, don't memorize! mitigating memorization in generative LLMss. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL http://pppeervieeeeeeeeooooroiie
www.nee/ooormmiddyyyyyisss..s fmWW.
Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. Foundation models and fair use. Journal of Machine Learning Research, 24 (400):1-79, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https: http://////////////////// wtwpeeewwwwwwweefrrreeeeee net/forummid jmIBmmImmm.
12

---

Published as a conference paper at COLM 2025
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022. URL https: http://////peeeeew.ewntteet Forum?id=nZeWKert
Jing Huang, Diyi Yang, and Christopher Potts. Demystifying verbatim memorization in large language models, 2024. URL htttp:///axxiv http:/v...r//b/////////// 17817.
Martin Kuo, Jingyang Zhang, Jianyi Zhang, Minxue Tang, Louis DiValentin, Aolin Ding, Jingwei Sun, William Chen, Amin Hass, Tianlong Chen, Yiran Chen, and Hai Li. Proactive privacy amnesia for large language models: Safeguarding Plll with negligible impact on model utility. In The Thirteenth International Conference on Learning Representations, 2025. URL /ottpee//ppenreveew http://ewweeeeommniiioooooooooee
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Tvison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. 2024.
Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, WWW '23, pp. 3637-3647, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394161. doi: 10.1145/ 0113507.3583199 URL https://ddot org/0. wtt.//////...//.//... 1145/3543507 3583199.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of 2023.
instruction-following models. https://github. cot.ouuboooo/ttttttlaaaaaoa haapaappaaa.eee/e
Jiacheng Liu, Sewon Min, Luke Zettlemoyer, Yejin Choi, and Hannaneh Hajishirzi. Infini- gram: Scaling unbounded n-gram language models to a trillion tokens. In First Conference on Language Modeling, 2024a. URL
Xiaoze Liu, Ting Sun, Tianyang Xu, Feijie Wu, Cunxiang Wang, Xiaoqian Wang, and Jing Gao. SHIELD: Evaluation and defense strategies for copyright compliance in LLM text generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1640-1670, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10.18653/v1 00/5533v1/222444 222//222..mmllmmaai 98/.
URL https: //taa/aaaaaaaaooyyycycoo. org.o/gg//22 2024. 2024.emnlp--eeemmmmmmmmmmm1.mmp1 emnlp- paimn..
Ximing Lu, Melanie Sclar, Skyler Hallinan, Niloofar Mireshghallah, Jiacheng Liu, Seungju Han, Allyson Ettinger, Liwei Jiang, Khyathi Chandu, Nouha Dziri, et al. Ai as humanity's salieri: Quantifying linguistic creativity of language models via systematic attribution of machine text against web text. Xiv preprint ar arXio:2410.042255, 2024.
Jakub Lucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, and Javier Rando. An adversarial perspective on machine unlearning for Al safety. Transactions Machine Learning Research, 2025. ISSN 2835-8856. URL https: http////ppeeeeeee/////eee net/forum? id=J5IRyTKZ9s.
William Merrill, Noah A. Smith, and Yanai Elazar. Evaluating n-gram novelty of language models using rusty-DAWG. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 14459-14473, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18653//e 2tt5//112222..enppmmaaa 800. URL http:////o/claaologgooy org/ 2024. wee..emmlppmnnnn 800/-
Sewon Min, Suchin Gururangan, Eric Wallace, Weijia Shi, Hannaneh Hajishirzi, Noah A. Smith, and Luke Zettlemoyer. SILO language models: Isolating legal risk in nonpara- metric datastore. In The Twelfth International Conference on Learning Representations, 2024. URL https::/openree: /ttp:://///eeeeeee.eeee wtwp:r/wwww..tfffooooooodooooooo
13

---

Published as a conference paper at COLM 2025
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Hen- derson.
aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, 2024. 2024. URL
https://openreviie www...nefforrrue dd Fedz.
Qwen-Team. Qwen2.5: A party of foundation models, September 2024. URL https:
//qwenIm. github. wttpii////ll/aeee hwwp////weene.mm 5/.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language
is secretly a reward model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 53728-53741. Curran Associates, Inc., 2023. URL https: neurips cc/paper beaaperrfllls/ppeepareeeeepeeeee files//aper/292222 paw.eeeeeeee............
Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. The language barrier: Dissecting safety challenges of LLMs in multilingual contexts. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 2668- 2680, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653/ 202///24. atn.24.innings-.....55 URL httpp://aat https://aclanthoooogy anthology org/2024, findings-acl. 156/.
Taiwei Shi, Kai Chen, and Jieyu Zhao Safer-instruct: Aligning language models with automated preference data. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Compu- fational Linguistics: Human Language Technologies (Volume Long Papers), pp. 7636-7651, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/ v1/2024.naacl-Be maw.222naacl-ooo....o 422. URL httpss://allanthology/ org/2024. naacl-long 422/-
Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A. Smith, and Chiyuan Zhang. MUSE: Machine un- learning six-way evaluation for language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /twp///wppeeeeeeee//e/ppe...e..p net/ www.:/ne/ffrrrnet forum??d::Trrmme33338
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhi- lasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Belt- agy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: open corpus of three trillion tokens for language model pretraining research. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), PP- 15725-15788, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ 20.1663/v1/20244 v1/2024.acl-longg anthology. .rg. 2.0..acll11..... 840/.
URL https: / /acl//ccclccc /ttpss//aclantholoogy
Manan Suri, Nishit Anand, and Amisha Bhaskar. Mitigating memorization in Ilms using 06040.
activation steering, 2025. URL
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed Chi, Denny Zhou, and Jason Wei. Challeng- ing BIG-bench tasks and whether chain-of-though can solve them. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Com- pulational Linguistics: ACL, 2023, pp. 13003-13051, Toronto, Canada, July 2023. Asso- ciation for Computational Linguistics. doi: 10. (0.118653/vi 220......niings...... .c..82. URL https://aclanthoooyy.or///// http://2222.....nnssssseee findings-ann 824/.
Xinyi Wang, Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. Generalization v.s. memorization: Tracing language
14

---

Published as a conference paper at COLM 2025
models' capabilitiis back to pretraining data. In The Thirteenth International Conference wttp://www...oooo......T//O////
Learning Representations, 2025. URL
Boyi Wei, Weijia Shi, Yangsibo Huang, Noah A. Smith, Chiyuan Zhang, Luke Zettlemoyer, Kai Li, and Peter Henderson. Evaluating copyright takedown methods for language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: wwwpppeeeevveeee
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. DEPN: Detecting and editing privacy neurons in pretrained language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Pp. 2875-2886, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ 11166///v1/222.emmnmpmpp/ em///23eemllmmmin..... URL https:://allaattlll/ll//
org /rgg/02.... www.lmmnaaleeeeeeee p-main..... 174/.
Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning, 2025. URL https://arxiv.. wttplii/.rgg//////////// http//bb//211 23123.
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations, 2024. URL https: trØKidwPLc.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. In First Conference on Language Modeling, 2024. URL https: /ttp///reeie.e..tt/ooo/m///////a
15

---

Published as a conference paper at COLM 2025
A Related Works
A.1 Language Model Memorization
Language model memorization (Carlini et al., 2019) occurs when neural networks store and reproduce exact near-identical sequences from their training data, with verbatim regurgitation manifesting as direct output these memorized fragments (Carlini et al., 2021). This behavior emerges through standard training procedures, influenced by model scale (larger models exhibit greater memorization), data repetition (frequently encountered sequences are
readily memorized), and task type (factual tasks demonstrate stronger memorization than reasoning tasks) (Carlini et al., 2023; Wang et al., 2025). Training dynam- ics reveal that improved language modeling capabilities correlate with both enhanced task performance and increased verbatim memorization (Huang et al., 2024; Xie et al., 2025).
Memorization creates several safety concerns: privacy risks when models store personal or sensitive data (Brown et al., 2022), copyright violations from reproducing protected content (Henderson et al., 2023; Chen al., 2024; Wei et al., 2024), and compromised evaluation validity when models regurgitate test data encountered during pre-training (Bordt et al., 2024). The distributed encoding of high-risk information across model parameters rather than isolated components complicates targeted removal (Huang et al., 2024).
Mitigation strategies include training techniques (e.g,, differential privacy that introduces noise during training) (Chua et al., 2024; Hans et al., 2024), data curation approaches that minimize repetition and filter sensitive content (Min et al., 2024), and machine unlearning methods for selective memorization removal (Shi et al., 2025). Prompt engineering and output filtering provide temporary safeguards but fail to address fundamental causes (Chen et al., 2024; Wei et al., 2024; Liu et al., 2024b).
A.2 Safety Fine-tuning
Safety fine-tuning adapts LMs to align with human values and reduce harmful outputs (Qi et al,, 2024). Recent studies identify challenges in balancing safety with utility, addressing cross-lingual disparities, and preventing adversarial misuse recovery of hazardous capabilities (Shi et al., 2024; Shen et al., 2024; Lucki et al., 2025).
Safety Fine-tuning research on regurgitation reduction has primarily targeted training LMs to refuse harmful or malicious prompts through supervised fine-tuning or alignment techniques (Liu et al., 2024b; Brahman et al., 2024). Our work focuses instead on reducing unintentional regurgitation in benign contexts, such as extending user-provided prefixes or creating original text.
16

---

Published as a conference paper at COLM 2025
B Data Processing Details
Synthetic Paraphrases. To generate paraphrases of memorized segments, we use Llama3.1 1- 70B-Instruct with a temperature of 0.6 and top-p of 0.9. The model is prompted with the following template:
Paraphrasing given snippet
Your task is to rewrite the given text code, maintaining the same meaning while using different words. Follow these guidelines:
Preserve the original length closely as possible.
Ensure the rewritten version is clear and grammatically (and syntactically) correct.
The text to rewrite enclosed below:
{text]
Please provide only the rewritten version without any additional comments or explanations.
Data Examples Some chosen and rejected snippets from the training data of the LLaMA 3.1-8B model are shown in Table
Chosen
Rejected
Question: Is manual intervention required for all users to implement the fix, must they delete the file themselves? Alterna- tively, will this issue be resolved automat- ically for the majority of users-if not, may deter people from using this product.
Question:
Do users have to apply the fix specifically, delete the file manually? Or this going to sorted out automatically for most people-if not is true that it will put people off this product.
Situated in Nashville, Tennessee, and po- sitioned at the intersection of education, innovation, and healthcare, VUMC is vi- brant community of professionals united by shared goal of making profound impact. This is an environment where your skills will be recognized, your under- standing broadened, and your capabilities pushed to new heights. This is an envi- ronment where your unique perspective encompassing diverse backgrounds, ideas, experiences, and leadership styles is ac- tively sought and honored. This is an envi- ronment where team members are aware they are integral to something bigger than themselves.
Located in Nashville, Tennessee, and oper- ating at a global crossroads of teaching, dis- covery and patient care, VUMC munity of individuals who come to work each day with the simple aim of chang- ing the world. It is place where your expertise will be valued, your knowledge expanded and your abilities challenged. It is place where your diversity - of cul- ture, thinking, learning and leading is sought and celebrated. It is a place where employees know they are part of
com-
Table 7: Examples of chosen rejected snippets
C Model Training Details
We fine-tuned the Llama-3.1-4 3.1-BBBBBBBBBB------ and Qwen-2.5-7B model using Direct Preference Optimiza- tion (DPO) for 2 epochs with global batch size of 64, distributed across 4 NVIDIA A100. All models were trained with DeepSpeed Stage and bfloatl6 precision. The default learning rate was 5e-7 unless otherwise specified. We used linear learning rate scheduler with
17

---

Published as a conference paper at COLM 2025
warmup ratio of 0.1 and no weight decay. We experimented with different mixture ratios between the paraphrase pairs and generic DPO data by adjusting the proportions to 1.0:0.0, 0.9:0.1, and 0.5:0.5. These ratios were treated as hyperparameters to assess their impact performance. In our experiments, we used the 0.9:0.1 ratio for the ParaPO Mix variant and the 0.5:0.5 ratio for the ParaPO Sys Mix variant. All training runs were conducted with 0.1.
D Baseline Methods
Supervised Fine-Tuning with Paraphrases. We fine-tune the model using the chosen paraphrases from the ParaPO dataset. The model is trained with learning rate of le-4 for two epochs with LoRA (rank 256).
System Prompts. As an alternative mitigation method, we evaluate the effectiveness of system prompts in reducing memorization. The following system prompt is used:
Your primary goal is to produce content that is completely original and free from any existing text on the Internet. This means that if someone were take few words from your explanation and search for them online, they can not find any matching results.
Unlearning Methods. We train models using unlearning algorithms under the same setup as in MUSE (Shi al., 2025), modifying only the model architectures and hyperparameters as needed. Specifically, we evaluate two methods: Gradient Ascent with KL Regularization (GA-KLR) and Negative Preference Optimization with KL Regularization (NPŌ-KLR). For each method, we run experiments on two language models: LLaMA-3.1 1-BB and Qwen-2.5- 7B. We use learning rate of 1e-5 and a global batch size of 32.
E Evaluation Details
Metrics. We use the ROUGE-L score, which considers both precision and recall. Precision defined as the ratio of the longest common subsequence (LCS) length to the output length, while recall is the ratio of the LCS length to the ground truth length. The final ROUGE-L score computed as the geometric mean precision and recall. For word-level ROUGE-L, use word tokenize from the nltk library to tokenize sentences into words.
To assess model memorization in copyright-related evaluations, we compute the QA accu- racy using exact string match between the gold answer and the model output.
Dataset. The datasets used in our experiments licensed under different terms. The Training Data Extraction Challenge is licensed under Apache-2.0, BookMIA under MIT, BookSum under BSD-3-Clause, and the SHIELD code repository under MIT. For the Training Data Extraction Challenge, we tokenize texts using the GPT--eo 1.3B tokenizer. In copyright evaluation experiments with BookSum and BookMIA, reference text to compute the ROUGE-L score.
the first 50 words of the
Additional Experimental Results
Utility Preservation To preserve the model's utility, we train the model with a mixture of ParaPO paraphrase pairs and generic preference optimization data. Moreover, we also tune the model with LoRA (Hu al., 2022), parameter-efficient finetuning method. The trade-off between utility preservation and regurgitation reduction is shown in Figure 4. We observe that for both full-parameter finetuning and LoRA finetuning, using 90% of paraphrase pairs and 10% of the generic preference optimization data is sufficient to retain most the model's utility while still yielding a noticeable reduction in regurgitation. We also find that, under the same hyperparameters, LoRA fine-tuning leads to greater
18

---

Published as a conference paper at COLM 2025
Fraction ParaPO
LoRA
Full parameter 0.5
Data in Training Set
(IFE)
78
0.5
0.5
76
75
Utilityy
74
73
72
5
10
15
20
Regurgitation Evaluation (Web)
Figure 4: Trade-off between utility and regurgitation when mixing ParaPO with conventional DPO data. Lower paraphrase fractions reduce copying less effectively, while higher fractions preserve utility.
Events (Non-literal) (>4, %, L)
Events (Non-literal) (Max, 4)
Characters (Non-literal) (>2, %, J)
Characters (Non-literal) (Max, L)
Literal
Literal (Max, )
Model
Methods
(>0.5, %, L)
Default ParaPO
1.00 0.46
2.034
5.876 6,384
Tulu 3
0.0
1.243
9
Table 8: Literal and non-literal copying results on CopyBench.
reduction in regurgitation. We speculate that full-parameter fine-tuning may reduce the training loss more easily, resulting in smaller overall changes to the model compared to LoRA fine-tuning.
Full Evaluation on CopyBench. We evaluate both literal and non-literal copying using CopyBench (Chen al., 2024), comparing the Tulu-3-8B model with its ParaPO-tuned version. For non-literal copying, we prompt the model with story-writing prompts and measure event and character overlap with reference texts. The results, shown in Table 8, indicate that our method is effective in reducing literal copying but has limited impact non-literal copying.
Evaluation on Enron Email Dataset Our regurgitation evaluations on web and book snippets directly measure how extractable the training data are from the model, which relates to privacy concerns. We further evaluated the Enron Email dataset (which contains personal information from Enron Corporation employees) and observed the same trend. Using the same setup as the Pile-CC subset, we report the percentage regurgitated cases in Table 9, ParaP'O shows a significant reduction. Manual inspection found that most remaining cases involve website URLs and file paths, which were not changed during paraphrasing in ParaPO.
Web (Pile-CC)
Email (Pile-Enron)
I.lmm3.1-8B
33.4
21.6 (-35%)
3.0 (-32%)
Table 9: Extraction Evaluation Enron Email.
19


Reviews:
[{"review_id": "4JpWSQ56B4", "reviewer": "Reviewer_nEDD", "summary": "This propose ParaPO to reduce the regurgitation, which is a interesting realistic problem. Specifically, authors build their method on DPO. They first perference data needed by DPO. And the \"perference\" of ParaPO is the degree of regurgitation. Then, they use these samples to fine-tune LLMs to reduce the regurgitation.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Clear writing and Detailed presentations: The paper is clearly structured and provides thorough descriptions of both the methodology and evaluation pipelines.\n- Controllable regurgitation via prompts: The idea of using system prompts to flexibly control whether regurgitation is desired is simple and practical, enabling LLMs to recall exact quotations when explicitly required.\n- New and realistic problems: The authors demonstrate that ParaPO reduces regurgitation metrics across domains (books, web, creative writing) more effectively than baselines like unlearning or simple paraphrase pretraining.", "reasons_to_reject": "- The contribution is marginal: While the idea is practically useful, the core contribution is essentially applying DPO to a paraphrase dataset constructed for a specific goal (reducing regurgitation). This approach, while effective, does not introduce fundamentally new techniques in optimization or model training.\n- The comparison of methods lacks baseline experiments. A significant omission is the lack of comparisons to simple supervised fine-tuning (SFT) baselines trained on the same paraphrase data. Although one such baseline is briefly evaluated (“Training on Paraphrases”), a more rigorous exploration—e.g., combining SFT with system prompts, or training on paraphrase data using next-token prediction instead of DPO—would make the evaluation more complete.\n- Insufficient discussion of reinforcement learning (RL) alternatives. Since ParaPO is positioned as a fine-tuning strategy, it would be helpful to compare it against on-policy RL methods (e.g., PPO or GRPO), which are also commonly used to shape model behaviors and could in principle reduce regurgitation. This would strengthen the paper’s claim that ParaPO is superior or more practical.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CSRneYGEOx", "reviewer": "Reviewer_Jibe", "summary": "This paper introduces ParaPO, a post-training method designed to reduce verbatim reproduction (regurgitation) of pre-training data in language models while preserving their utility. The authors address a key challenge in modern LLMs: their tendency to memorize and regurgitate content from training data, which raises concerns about copyright, plagiarism, privacy, and reduced creativity. Empirical results demonstrate that ParaPO effectively reduces regurgitation across multiple datasets without significantly compromising model utility, outperforming previous unlearning methods that only work on specific target domains. The authors evaluate their approach on both pre-trained base models (Llama3.1-8B, Qwen2.5-7B) and instruction-tuned models (Tulu3 variants), using multiple evaluation metrics. Results show that ParaPO reduces regurgitation across the evaluated datasets while maintaining strong performance on utility benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- ParaPO introduces a creative application of preference learning techniques to reduce verbatim copying. While preference optimization has been widely used for aligning models with human preferences, applying it specifically to combat regurgitation is novel.\n- Verbatim reproduction is a major concern for language model deployment, with legal, ethical, and practical implications. ParaPO addresses this challenge directly with a practical solution.\n- Unlike unlearning methods that only work on specific target domains, ParaPO provides a generalizable approach to reducing regurgitation across different contexts.", "reasons_to_reject": "- The authors use Llama3.1-70B-Instruct to generate paraphrases. However, there is no analysis of how paraphrase quality affects ParaPO's performance. Understanding whether high-quality paraphrases are necessary or whether simpler rule-based approaches might work would provide valuable insights for practitioners with limited computational resources.\n- While the paper shows that ParaPO reduces verbatim copying, the underlying causal mechanism isn't thoroughly explored. For instance, does the model truly learn to paraphrase, or is it simply avoiding certain high-probability continuations? More causal analysis would deepen understanding of how ParaPO works.", "questions_to_authors": "1. it's not entirely clear how paraphrase quality impacts the effectiveness of ParaPO. Have you observed cases where poor paraphrases degrade alignment performance? Did you manually or automatically filter any paraphrases for quality?\n2. Even though the issue mentioned in the limitation section, since your experiments are conducted on models up to 8B parameters, which are known to memorize less than larger ones, can you speculate on how ParaPO might scale? Is the method expected to remain stable or effective in the context of stronger memorization in larger LLMs?\n3. From Table-1, in Math and Reasoning evaluation, applying ParaPo decreases Llama3.1 8B score while increasing the Qwen2.5 7B score. What might be the reason behind this phenomenon?\n4. Given the model is explicitly trained to reject verbatim segments and prefer paraphrases, is there a measurable or perceived impact on coherence or discourse structure in long generations? Could this hurt performance on tasks requiring multi-turn or long-span dependencies?", "extra_scores": {}}, {"review_id": "ROU1Ec6vjY", "reviewer": "Reviewer_6ZUs", "summary": "This submission describes an attempt to reduce verbatim memorization of language models via preference optimization (DPO) over paraphrased variants (generated from a better language model). The method can be extended to a more controllable variant by using \"Copying: Yes\" or \"Copying: No\" as the system prompt.\n\nMany useful experiments have been conducted and the reported results show that the presented methods are in general working as intended.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The studied topic (verbatim memorization) might be interesting to many conference attendees.\n- The empirical performance seems promising.\n- Rich experimental comparisons and analysis have been conducted.", "reasons_to_reject": "- The presented method is a very straightforward application of DPO on a specific type of data, while the current presentation may leave an impression that ParaPO is a new post-training method.\n- Paraphrasing is a key step in this work, while this work currently only includes relevant details in the Appendix without referring in the main text. Current results are based on paraphrases generated by Llama-3.1-70B-Instruct alone, without verification on validity, diversity, or other statistical characteristics of the rephrased texts.\n- There has been no discussion at all on a potential misuse: model developers may hide their unethical use of copyright-protected data in pre-training via discouraging verbatim generation. This is a serious concern on many related studies as well, given that many websites have explicitly indicate that they disallow data to be used for training AI models while there's no effective way to technically stop data crawling.\n- Doubts in some key results (see questions below)", "questions_to_authors": "1. Why using Llama-3.1 as the paraphrasing model given that many proprietary LLMs might produce better and more diverse paraphrases? Given that the prompt requires to \"Preserve the original length as closely as possible\", to what extent the rephrasing relies on lexical/phrasal paraphrases versus richer syntactic paraphrases? In what proportion of the paraphrases the semantic meaning was not identical to the original?\n2. Any idea on why ParaPO improves Qwen2.5-7B on math and reasoning benchmarks as shown in Table 1?\n3. One can read from Table 2 that the controllable variant that discourage verbatim generation (ParaPO + \"Copying: No\" system prompt) seems to have a negative impact on general utility, especially instruction following (IFEval and AlpacaEval2). What might cause this difference from that without the system prompt?\n4. One slightly more convincing reason to disallow verbatim memorization is an intention for privacy preservation. Is there any part of experiments that may reflect such an intention?", "extra_scores": {}}]

Author rebuttal:

Meta-review: Mostly positive reviews on this interesting and timely work. I recommend we accept this paper.

The reviewers arguments not to accept the paper include:
- Marginal contribution
- Missing baselines

The reasons to accept include:
- Clear writing
- Important and realistic problem
- Rich experimental findings

Overall, the issue of verbatim regurgitation in LLMs is important and timely. The particular method has several advantages over existing unlearning methods and the empirical results are compelling.

**This paper went through ethics reviewing. Please review the ethics decision and details below.**  
Decision: Acceptance (if this paper is accepted) is conditioned on addressing the following in the camera-ready version  
Details: A discussion on the potential for dual use should be included. 
Decision: Accept