SUV: Scalable Large Language Model Copyright Compliance with Regularized Selective Unlearning
Authors: Tianyang Xu
Xiaoze Liu
Feijie Wu
Xiaoqian Wang
Jing Gao
Venue: COLM 2025 (2025)
Keywords: LLMs; Calibration; Copyright; Unlearning
TL;DR: TL;DR: We propose SUV, a selective unlearning framework that excises memorized copyrighted content from LLMs without compromising their overall performance.

Abstract: Large Language Models (LLMs) have transformed natural language processing by learning from massive datasets, yet this rapid progress has also drawn legal scrutiny, as the ability to unintentionally generate copyrighted content has already prompted several prominent lawsuits. In this work, we introduce SUV (Selective Unlearning for Verbatim data), a selective unlearning framework designed to prevent LLM from memorizing copyrighted content while preserving its overall utility. In detail, the proposed method constructs a dataset that captures instances of copyrighted infringement cases by the targeted LLM. With the dataset, we unlearn the content from the LLM by means of Direct Preference Optimization (DPO), which replaces the verbatim copyrighted content with plausible and coherent alternatives. Since DPO may hinder the LLM’s performance in other unrelated tasks, we integrate gradient projection and Fisher information regularization to mitigate the degradation. We validate our approach using a large-scale dataset of 500 famous books (predominantly copyrighted works) and demonstrate that SUV significantly reduces verbatim memorization with negligible impact on the performance on unrelated tasks. Extensive experiments on both our dataset and public benchmarks confirm the scalability and efficacy of our approach, offering a promising solution for mitigating copyright risks in real-world LLM applications.

Published as a conference paper at COLM 2025
SUV: Scalable Large Language Model Copyright Compli- ance with Regularized Selective Unlearning
Tianyang Xu, Xiaoze Liu, Feijie Wu, Xiaoqian Wang, Jing Gao Purdue University fxu1868, xiaoze, wu1977, joywang, jinggao
Abstract
Large Language Models (LLMs) have transformed natural language pro- cessing by learning from massive datasets, yet this rapid progress has also drawn legal scrutiny, as the ability to unintentionally generate copy- righted content has already prompted several prominent lawsuits. In this work, we introduce SUV (Selective Unlearing for Verbatim data), a se- lective unlearning framework designed to prevent LLM from memorizing copyrighted content while preserving its overall utility. In detail, the pro- posed method constructs dataset that captures instances of copyrighted infringement cases by the targeted LLM. With the dataset, we unlearn the content from the LLM by means of Direct Preference Optimization (DPO), which replaces the verbatim copyrighted content with plausible and coher- ent alternatives. Since DPO may hinder the LLM's performance in other unrelated tasks, we integrate gradient projection and Fisher information regularization to mitigate the degradation. We validate our approach us- ing large-scale dataset of 500 famous books (predominantly copyrighted works) and demonstrate that SUV significantly reduces verbatim memo- rization with negligible impact on the performance on unrelated tasks, Extensive experiments on both our dataset and public benchmarks confirm the scalability and efficacy of our approach, offering a promising solution for mitigating copyright risks in real-world LLM applications. Code is available at http://tttuu..o//////////
Introduction
Large Language Models (LLMs)) have demonstrattdd impressive utility by generating human- like text from massive datasets. However, their widespread adoption has been accompanied by serious copyright concerns (Huang et al., 2025; Min et al., 2023; Henderson et al., 2023; Liu et al., 2024c; Chen et al., 2024), exemplified by high-profile lawsuits (Adams, 2023; Maheshwari & Tracy, 2023; Tracy & Maheshwari, 2023). Numerous studies have confirmed that LLMs can memorize and regurgitate verbatim excerpts of copyrighted materials (Chang at. 2023; Karamolegkou et al., 2023; Liu et al., 2024c; Wei et al., 2024). In an ideal world, one would train LLMs solely on non-copyrighted data, yet the sheer scale and labor required for data curation render this approach impractical, and retraining models from scratch without copyrighted material is equally inefficient. Consequently, various post-training "patching" methods have been proposed to address this issue.
Existing approaches preventing copyright infringement in LLM outputs can be broadly divided into three categories. First, decoding-buet bbase methods (Wei et al., 2024; Ippolito et al., 2023b; Abad et al., 2024b; Vyas al., 2023) modify the next-token probability distribution during decoding. While these methods can reduce the risk of generating verbatim outputs, they can be easily removed by attackers since they are not integrated into the model itself. Furthermore, some of them may be prone to inducing hallucinations (Liu et al., 2024c; Abad et al., 2024b). Second, agent- methods (Liu et al., 2024c; Hua et al., 2024) build external agents to detect and prevent the generation of illegal content. Although these
These two authors contributed equally; order was determined randomly (by rolling die).
1

---

Published as a conference paper at COLM 2025
reccoding-based methods
LLM
Copyrighted hext
Previous unlearning methods
proud
Weight
Protection
Open
happy sod
engry
Capyrighted lext
Unscalable
four. Privet Drive,
exclled
proud
Knowledge Untearning
Uhility Preverved
Sefervantion medel
were
proud
Context
parfactly normal, thank
May Requia
ULM
This contunt reaat copyright
Imporhant icots
Regularized Troning
Sodlatle And Utillyy
External
Rewrihc
iisation
Preserved
Knowledge
Agent-based methods
SUV
Figure 1: Comparison between different copyright compliance approaches for LLMs
methods accurately identify problematic outputs, they often require additional resources (e-g,, internet access) and fail to remove the memorized content embedded within the model itself. Third, unlearning methods (Bourtoule et al., 2021; Maini et al., 2024; Dou et al., 2025; Zhang et al., 2024; Sekhari et al., 2021; Xu al., 2023; Liu et al., 2024d) directly target the removal of memorized copyrighted content from the model, which have shown promise for open-weight LLMs; however, they face several challenges:
Utility degradation: Many existing approaches achieve unlearning by erasing the knowl- edge of entire copyrighted material, aiming remove not only copyrighted information but also information that can be generalized or rephrased from the original ones. How- ever, existing copyright infringement cases are primarily related to verbatim excerpts (Karamolegkou et al., 2023; Liu et al., 2024c; Wei et al., 2024), indicating that existing approaches may be too strict and may overcorrect for many cases, Still, verbatim memo- rization unlearning is also likely to undermine LLM utility (especially the performance of unrelated tasks). With the increasing scale of copyrighted materials, existing unlearning approaches exhibit significant drops in utility, rendering them unscalable.
Dataset limitations and scalability; Currently available benchmarks are generally small-scale and predominantly based on non-copyrighted materials. Researchers (Liu et al., 2024c) have shown that non-copyrighted materials tend to have higher memorization rates than copyrighted ones, meaning that using non-copyrighted datasets for alternative evaluation may not accurately reflect the evaluation of copyright violations due to differing data distributions. However, due to copyright restrictions, assembling large-scale datasets of copyrighted material is challenging.
To overcome these challenges, we propose SUV (Selective Unlearing for Verbatim data), novel unlearning framework that adopts selective approach, motivated by two key obser- vations. First, the context generation for copyrighted materials is highly localized general domain knowledge, verbatim memorization is confined to specific passages, making broad unlearning both unnecessary and potentially harmful. Second, memorization follows long-tail distribution, meaning that only small subset of the training data is responsible for most copyright risks. These insights motivate a targeted unlearning strategy that focuses exclusively on the problematic segments and parameters.
Our approach unfolds in three primary stages. Firstly, unlike previous methods that segment text into large chunks, we employ sliding-windov mechanism achieve fine-grained identification of memorized passages. This allows us to apply more granular approach in distinguishing between relevant and irrelevant information within the model's memory. Secondly, we tackle the challenge of utility degradation. Many existing approaches remove not only copyrighted content but also information that can be generalized or rephrased from the original texts. Consequently, unlearning verbatim memorization can undermine LLM eippiil-epecciallly on unrelated tasks-and lead to significant performance drops as the scale of copyrighted materials increases. In contrast, our method unlearns the targeted segments via Direct Preference Optimization (DPO) (Rafailov et al., 2024), preferring a randomly generated completion over plagiarized content. Although the random completion may initially yield meaningless passages, we mitigate the degradation by integrating projected gradient techniques and Fisher information regularization to control the unlearning process
2

---

Published as a conference paper at COLM 2025
and prevent dramatic LLM updates. In contrast to additive-loss baselines such as GA and NPO that simply sum separate "forget" and "retain" objectives, our DPO formulation turns unlearning into single, preference-based optimisation problem. This unified objective updates only the parameters most responsible for verbatim memorisation, thereby minimising collateral damage to general knowledge and downstream utility. Finally, we address dataset limitations and scalability by constructing a large-scale dataset comprising 500 famous proks--predmminantly copyrighted works such as Harry Potter Series and A Song of Ice and Fire Series-to to faithfully mimic real-world usage. By assembling our cataset of 500 copyrighted books, our approach provides more realistic assessment of LLM behavior under copyright constraints.
By addressing the limitations of existing approaches, our work provides a scalable and practical solution for mitigating copyright infringement in LLMs while ensuring that their gurrros capabilities remain largely intact. Our contributions are as follows:
To the best of our knowledge, this is the largest-scale study unlearning copyrighted texts, leveraging a dataset of 500 well-known books. Our dataset comprises 3.46 million sentences and is approximately 1,000 times larger than those used in prior work.
We introduce selective unlearning framework, namely SUV, that precisely targets prob- lematic segments responsible for verbatim memorization, making it possible for mitigat- ing copyright risks over large-scale corpus while preserving model utility.
Extensive experiments on both our large-scale dataset and public benchmarks (Wei et al., 2024; Dou et al., 2025) demonstrate that the proposed approach achieves superior balance between effective unlearning and the maintenance of overall model performance.
2 Related Work
Copyright Issues in LLMs. Prior work has focused on examining the extent to which language models reproduce copyrighted text and how this behavior relates to the broader phenomenon of memorization. Several studies have employed various probing techniques, such similarity measurement and masked token prediction, to evaluate verbatim repro- duction and data extraction risks (Chang et al., 2023; Karamolegkou et al., 2023; D'Souza & Mimno, 2023; Hacohen et al., 2024; Nasr et al., 2023; Schwarzschild et al., 2024; Li et al., 2024; Liu et al., 2024c; Chen et al., 2024). Investigations have revealed that model capacity and training procedures play significant role memorization, with even state-of-the-an models outputting verbatim excerpts from their training data (Carlini et al., 2021; 2023; Biderman et al., 2023; Wei al., 2024; Liu et al., 2024c). Concurrently, other works have introduced new evaluation metrics and defense assessments to better understand these copyright implications (Wei al., 2024; Mueller et al., 2024; Chen et al., 2024). Further stud- ies have explored detection methods and data curation techniques to mitigate memorization risks (Huang al., 2024; Kim et al., 2023; Elangovan et al., 2021; Lee et al., 2022). Efforts to mitigate these issues have been broadly categorized into methods that enable models to forget specific training data associated with copyrighted content, iee. unlearning (Liu et al., 2024byd; Yao et al., 2023; Hans et al., 2024; Chen & Yang, 2023; Hans et al., 2024; Maini et al., 2024; Dou et al., 2025; Zhang et al., 2024), modifications at the decoding stage to steer generation away from potentially infringing text (Ippolito et al., 2023a; Xu et al., 2024b; Abad et al., 2024a), with additional enhancements explored via agents (Liu et al., 2024c) or RAG (Golatkar et al., 2024). Our method falls into the unlearning category, enabling copyright compliance for open-weight LLMs. Current unlearning methods fail to handle large-scale corpus while retaining the model performance. In contrast, SUV employs targeted elimination to unlearn specific content while preserving the model's 's overall knowledge and utility.
Machine Unlearning in LLMs. Machine unlearning seeks to selectively remove undesired knowledge from a pretrained model while preserving its overall capabilities (Cao & Yang, 2015; Hoofnagle et al., 2019; Bourtoule et al., 2021; Nguyen et al., 2022; Liu et al., 2024d;e; Golatkar et al., 2020a;b; 2021; Dukler et al., 2023). A central challenge is catastrophic forgetting, whereby unlearning targeted information inadvertently degrades untargeted knowledge and impairs model performance (Tarun et al., 2023; Xu et al., 2024a; Fan et al., 2024; Zhang et al., 2024). In the context of LLMs, existing mitigation strategies fall into
3

---

Published as a conference paper at COLM 2025
Dataset Construction
Regularized Training
Regularizer Combination
Unlearning DPO
Gradientt Projection
Differential Fisher
Copyrighted
Plagiarism
Forbiiden set Defection
Prompt
Countorfactual
Procept
Continustion
Ocntinussion
Joint troining
Privet Drive, were
say they were perfectly normel...
Unlearning Less
Adaptive Scheduling
Couraten Bastual
Orthogohill Projection
Fisher
Elumed when glowing
Regularized Task
spacecraft landed thair
>ydrongeet--Tdddee with viclet eyes locked eyes with their nephew.
Retain Loss
Gradient jeccted Task
Counterfactuel Generation
Grodient Projection
Differential Fisher
Seperate training
Task Vector Merging
Figure 2: An overview of the SUV Framework
two broad categories: parameter-base based unlearning (Jang et al., 2022; Yu et al., 2023; Zhang et al., 2023; Wu et al., 2023; Hase et al., 2023) and prompt-based unlearning (Madaan et al., 2022; Pawelczyk et al., 2023; Thaker et al., 2024; Muresanu et al., 2024; Liu al., 2024a). Prompt-based methods avoid the computational expenses of retraining by steering model outputs at inference time, but they neither yield a permanently modified model nor fully guard against adversarial "jailbreak" attacks (Liu et al., 2025). In contrast, parameter-based approaches (Zhang et al., 2024; Jia et al., 2024) produce a persistent and robust unlearned model, with the added benefit of preserving other learned knowledge. Despite these advances, it remains challenging to remove verbatim memorization of copyrighted materials in large-scale datasets without affecting performance. SUV employs selective approach that precisely targets unwanted content while preserving learned representations, making suitable for large-scale unlearning scenarios.
Methodology
We introduce SUV, a novel training framework to enable large language models (LLMs) to selectively unlearn extensive portions of book data while preserving performance unrelated tasks (see Figure 2). SUV comprises three key stages: (1) Dataset Construction (Section 3.2): We utilize an efficient slidng salding-windowy mechanism to identify verbatim excerpts from the specified texts, with each book processed in just a few minutes. Then, based on the verbatim excerpts, we prepare counterfactual dataset for DPO that helps reduce influence on unrelated tasks while unlearning the texts. (2) Regularized Training (Section 3.3): During the DPO training, minimize the adverse impact of unlearning on other tasks, we regulate the direction of parameter updates using gradient projection and select the most relevant parameters based on Fisher information. (3) Regularization Combination (Section 3.4): Since there might be interference between the two regularization methods, we propose two variants on combining them to better balance both unlearning performance and utility.
3.1 Problem Definition
Let fo denote a language model with parameters 0, where fx( denotes the probability of generating the sequence x under the model. We collect a set of books to be unlearned, which denote Da, named as the book dataset. We denote by Du the unrelated data, and our training data consists of D = DD U Du- The goal of unlearning is to update the model's parameters from 0 to 0' by solving the following optimization problem:
min Ex [x) 8
s.t.
(Eu(0) -Lu('') S E,
(1)
where the loss on Du is defined as (e) - ExeD, [[(x;))], where L [xx; denotes the loss function (negative log-likelihood) on input x for model parameter 6. Here, e > 0 is a pre- specified tolerance parameter that controls the acceptable degradation over Dy. This formulation minimizes the generation probability on the forbidden dataset while ensuring that the performance on unrelated tasks does not degrade significantly.
the performance
4

---

Published as a conference paper at COLM 2025
3.2 Dataset Construction
We construct a DPO dataset that meets the requirement of the above optimization problem (cf. Equation 1). A DPO dataset consists of preference triples (x(P), y ,y()), where x(P) is the prompt, y ((()) is the preferred continuation, and y( is the less-preferred continuation. We divide the dataset construction into two steps: (1) Plagiarism Detection, which verifies the subsequences in DB against the model's memory, ensures x(P) is indeed part of fe's training set, then constructs (P)) and yl) pairs, and (2) Counterfactual Generation, which generates y(w) as a counterfactual continuation to x(P).
DB is tokenized into sequence XT), Xt E V, t E {I, where V is the vocabulary. We partition the sequence into subsequences using segmentation indices the i-th segment is "K. Nk ), where E {0,.... ......,K- K-1}. In our experiments, for
Plagiarism Detection. Each book
kn < kj <... < kx T
convenience, we employ fixed-length segmentation, such that for all 0 < < K1 - k; is fixed. With sufficiently long segments, we approximate
K-1
(x) = II for (xkg (xkp.....
Mki+1
X1 Xx,...,,xy 1) ~
II i=0
for (xk,
(2)
i=0
We define the forbidden set Dr as the set of segments from DB with notably high probability:
Dr = (x {
osk K - 1, fo x 2>>2 e
(3)
with threshold 2 O. In practice, only few segments per book are memorized, meaning that instead of training on the large Dg, we can train on the considerably smaller DE to accelerate the training process. To detect these memorized segments, we apply a sliding window over x. For a window starting at token Xp, we define the
prompt as x(P) Npth.... xp+l,-1), and the corresponding original continuation as yd (Xp+Lpr +p +p tl.......... ........,........ X, L -.- 1), where Lp and Le denote the lengths of the prompt and continuation, respectively. A candidate continuation is generated by randomized sam- pling: y(8) ~ (y x(P)). If the similarity (e.g., measured by LCS or ROUGE-L) between y(8) and y(0) exceeds a threshold t, the segment is classified as belonging to Dr.
Counterfactual Generation. For each prompt x(p) with its rejected continuation y(), we generate a counterfactual (preferred) continuation y(") using a counterfactual generation process. Specifically, we prompt the model fo with x(P) and instruct it to produce an alternative continuation in different genre from y() This counterfactual continuation has both a low perplexity and a significant divergence from ylD) We detail the generation process in Appendix F.
3.3 Regularized Training
With the data constructed in Section 3.2, we can easily fine-tune the target language model with DPO. However, using the DPO dataset alone can reduce the model's likelihood of generating memorized content from DB, straightforward DPO training may compromise performance on the unrelated data Du. The core issue that the gradient updates for unlearning can conflict with knowledge crucial for maintaining performance on Du. To address this, we introduce two key techniques: (1) Gradient Projection, which ensures the unlearning process does not degrade the model's existing capabilities, and (2) Fisher-based Regularizatinn, which protects parameters highly sensitive for other tasks. We provide detailed description of the techniques we used in Appendix A.
Gradient Projection. To ensure that unlearning updates do not conflict with preserving performance on Du, employ gradient projection strategy. Let LDOO be the DPO loss with gradient 8p = Lpro We also define a preservation gradient gu, for instance, by maintaining sliding exponential moving average of gradients computed on Du-
5

---

Published as a conference paper at COLM 2025
When 8g) < O, the unlearning gradient gp is in direct opposition to ga, which would risk degrading performance on Dy. To avoid this, we remove from 8y the component that negatively aligns with gu. Formally, we define the projected unlearning gradient:
(gw gp)
8y
1lgall 8ur if (gur < O,
8p
(4)
8p
otherwise.
The final combined gradient for the update is then Sproj 8a + 8p, which guarantees that the unlearning updates do not counteract the preservation gradient, thereby ensuring that the model's performance on Du is preserved.
Fisher Regularization. While the safe update criterion defines which updates preserve performance on Du, we also need practical mechanism to steer the optimizer away from harmful parameter changes. To this end, we introduce a Fisher-based regularization term, namely Differential Fisker Importance, formally
AF max [f(fotb) F (.retain) e).
where 8 0 ensuring a lower bound, r(forb) and p((reaai)) be the Fisher information for parameter i on DD and Du, respectively. It penalizes large changes to parameters crucial for preserving knowledge on Dy. A large AF, indicates that parameter is more important for memorized content than for Du-
Let AB 6' f be the parameter update. We then define coordinate-wise Fisher penalty:
(Fiherr) log (A0))2 2 (AF;)2 1).
and aggregate them for all the updated parameter with hyperparameter Afsher as the Fisher regularization loss: Lireg Aisher E e e Here, a log function is incorporated mitigate the risk of experiencing overflow errors or excessively small gradients, which could hinder the optimization process. Furthermore, by penalizing large updates to parameters critical for Du, this regularization nudges the optimizer toward updates preserving performance unrelated tasks while reducing memorized content from Dg.
3.4 Regularization Combination
Recall that we have two regularization methods. We now detail how to fuse them to create a unified framework that leverages the unique advantages of each approach. Here, we provide two variants for SUV: (1) SUV-AS, which is designed to preserve utility by jointly optimizing both regularization losses; and (2) SUV-TV, which maximizes copyright protection by isolating the effects of each regularization.
Utility Preserving Training (SUV-AS). In this variant, our goal is to prevent utility drop by leveraging both Fisher-based and gradient projection losses, as each plays a complementary role in protecting the parameters import to Du. Directly adding these losses is an intuitive way to maximize protection, with the gradient projection helping maintain stable update directions and the Fisher-based regularization constraining update magnitudes. However, the Fisher loss can dominate when other parts of the model are actively updated, making the optimization challenging. To overcome this, we introduce an adaptive scheduler for the Fisher regularization weight Aisher- Starting from a fixed initial value, Aisher decays dynamically based on the training progress: mild decay (Pmild < is applied if the DPO loss decreases, while a severe decay (Psevere & Pmill) is used when no improvement is observed over R consecutive steps. This dynamic balancing termed SUV-AS ensures that both losses contribute effectively to preserving the overall utility of the model throughout training. A detailed description of the scheduler is provided in Appendix G.
Maximized Copyright Protection via Task Vector Merging (SUV-TV). Alternatively, to achieve stronger copyright protection, isolate the Fisher-based and projection-based

---

Published as a conference paper at COLM 2025
regularizations by training them separately, This separation is crucial because the Fisher- based task, which limits update magnitudes using differential Fisher information, and the projection-based task, which refines the update direction by projecting the unlearn- ing gradient onto a stable preservation gradient, can potentially conflict when trained jointly. By handling each task independently,
ensure that the full strength of each
regularization is preserved. After obtaining the two task-specific updates, AW1 = W1 Wasse and AW2 W2 - Wasee we merge them using a task vector (Ilharco et al., 2022): W Wmerged Waase AW, + AW2. This method, referred to as SUV-TV, isolates the effects of each regularization so that the robust protection against copyright infringement is maximized without compromising the model's core capabilities.
Discussion. A theoretical trade-off is discussed in Appendix B regarding the unlearning effect and LLM utility. Our analysis reveals that SUV-AS achieves superior performance in LLM utility, while SUV-TV excels in unlearning. Empirical studies (see Section 4) further substantiate these findings. This outcome aligns with our expectations: in SUV-AS, the constraints imposed by gradient projection and Fisher regularization jointly influence the model, thereby amplifying the effect of unrelated data, whereas in SUV-TV, these constraints are applied independently across two models, which diminishes the impact of such data.
4 Experiments
4.1 Experiment settings
Evaluation Datasets. We evaluate SUV using two groups of benchmarks: unlearning and utility benchmarks. For assessing unlearning performance, we employ several datasets: (1) CBP-500 (Copyrighted Books (Predominantlyy-500)) our dataset that is comprised of texts from 500 deduplicated and random famous books (predominantly copyrighted); (2) CBP-50, a randomly chosen subset of the CBP-500 dataset is used to ensure a fair comparison between SUV and baselines that cannot scale to 500 books; (3) CotaEval NewsQA (Wei et al., 2024), a publicly available benchmark that is comprised with texts from news articles; and (4) SSU Books (Dou et al., 2025), the 10 books used for training SSU, baseline approach discussed below. We refer to Appendix E.1 for details on the datasets. For retaining the utility of language models, the Alpaca (Taori et al., 2023) dataset is used as the retain set of SUV, some baselines and ablation methods. For efficiency, we randomly sample 2,048 examples from Alpaca during training. To assess performance on unrelated tasks, we utilize MMLU (Hendrycks et al., 2021), benchmark evaluating model knowledge across diverse subjects, and CSQA (Talmor et al., 2018), a benchmark with challenging multi-hop multiple-choice questions. Details can be found in Appendix E.3.
For unlearning benchmarks, we use ROUGE-L and LCS to measure text similarity and report the number of sentences with different ROUGE-L scores. For utility benchmarks, we follow the original dataset's accuracy criteria (Lin, 2004).
Evaluation Metrics.
Baselines. We compare with: (1) the language model without unlearning (Vanilla); (2) Stable Sequential Unlearning (SSU) that iteratively removes restricted data effects (Dou et al., 2025); (3) Gradient Ascent (GA) which applies reverse optimization on undesirable content; and (4) Negative Preference Optimization (NPO) that minimizes a preference function for unwanted data (Zhang et al., 2024).
Implementation Details. SUV, baselines, and ablations are implemented with the Llama 3.1-8B model for fair comparison. More details can be found in Appendix E.4. Additional experiments on Qwen 3-8B and Qwen 3-14B models are also conducted to verify the effectiveness of SUV over different model architectures and sizes, as detailed Appendix D.
4.2 Experimental Results
Unlearning Performance. Figure 3(a)-(c) presents the unlearning performance of the pro- posed methods along with various baselines, where lower ROUGE-L scores indicate better removal of verbatim copyrighted material. We evaluate all the methods on fixed-size chunks divided from the original books (for simplicity, let's just call them sentences) and report the number of sentences that exceed certain ROUGE-L threshold. We analyze the unlearning
7

---

Published as a conference paper at COLM 2025
Vanilla
NPO - GA
SSU
SUVAS
SUV-TV - w/o Fisher -
w/o Project + DPO only
10
104
10a
10
Count
Count
10a
Count
10a
10a
102
102
Score Threshold
Score Threshold
Score Threshold
(x) CBP-500 Results
(b) CBP-50 Results
CotaEval NewsQA Results
0.8
0.8
CSQA Accuracy
MMLU Accuracy
CSQA Accuracy
0.6
0.7
0.7
0.4
0.4
MMLU
0.6
0.6
0.2
0.2
0.0
0.0
0.5
0.5
10
50
500
10
50
500
10
50
500
10
50
500
Number of books
Number books
Number of books
Number of books
(d) MMLU Scores boseline meth (e) CSQA Scores of baseline methods ods on 10, 50, and 500 hooks
(0 MMLU Scores of ablation meth- (g) CSQA Scores ablation methods ods on 10, 50, and 500 books
on 10, 50, and 500 books
on 10, 50, and 500 books
Figure 3: (a)-c)) The unlearning experiment results of baseline and SUV methods on the CBP-500, CBP-50, and CotaEval NewsQA dataset. (d)-g):: The utility experiment results of baseline, SUV, and ablation models trained on different datasets. Here, the x-axis is the number of books, where 10 is for SSU Books, 50 for CBP-50, and 500 for CBP-500.
performance by calculating the ratio of the sentences with ROUGE-L scores exceeding specific thresholds (the lower the better unlearning performance). The ratios obtained by the proposed method and the Vanilla model are compared. Specifically, on the CBP-500 dataset, at the 0.2 threshold, the Vanilla model registers 5,133 sentences while the proposed method (using SUV-TV) retains only 1,176 sentences, corresponding to approximately 23% of the Vanilla count. For the 0.3 threshold, the proposed method's count is 169 (versus 2,762 for the Vanilla model), which is only about 6% of the original. This gap further widens at the 0.5 threshold, where the proposed method retains roughly 1.8% (23 versus 1,264 sentences) of the Vanilla count, and at the 0.7 threshold, only about 0.8% (6 versus 751 sentences) remains. These results demonstrate that the proposed approach can selectively remove copyrighted content down to nearly 1 1/100 or less of the original amount when more stringent thresholds are applied. We also notice that the baselines, such as SSU and GA, tend to aggressively reduce the number of sentences above the given thresholds; however, this reduction is achieved at the expense of utility (detailed below).
Utility Performance. While some baselines may achieve low ROUGE-L counts at higher thresholds, they often do at the cost of downstream task performance degradation. For example, methods like SSU and GA, despite having slightly lower unlearning metrics, experience significant degradation in utility metries. For example, their CSQA scores drop notably compared to the Vanilla model's 0.7166. We further show that they have made the model unusable with case study detailed in Section 4.2. In contrast, our SUV-AS approach consistently maintains utility scores close to the vanilla model (0.7093 for CSQA and 0.6622 for MMLU on CotaEval NewsQA), ensuring that the aggressive unlearning does not compromise overall performance. This dual-method framework-where SUV-TV excels at targeted content removal and SUV-AS preserves unders--unersscores the practical advantage of the proposed approach.
Ablation Study. We conduct an ablation to verify several design choices SUV: (1) SUV-TV
vs. SUV-AS: We compare two regularization merging techniques. SUV-AS uses an adaptive scheduler to merge regularization signals, while SUV-TV relies on task vector approach. (2) w/o Fisher: This variant evaluates SUV using only the gradient projection regularization, omitting the Fisher regularization.
(3) w/o Project: Here, we assess the performance when only Fisher regularization is applied, without the gradient projection component. (4) DPO only: In this setup, SUV is trained solely with the DPO objective, without any form of regularization. We compare several ablated variants of our method in Table 1. As
8

---

Published as a conference paper at COLM 2025
Method / Dataset
CotaEval
CBP-500
Metrics
ROUGE-L (1)
I.C5 (L)
Utility (1)
ROUGE-L (L)
TCS (4)
Utility (1)
Avg
Max
Avg
Max
MMLU
CSQA
Avg
Max
Avg
Max
MMLU
CSQA
Vanilla model
17.29
60.24
334.94
635
.6542
.7166
11.54
100.00
19.60
.6542
.7166
SUV-TV
13.58
26.58
306.01
438
.6555
.7199
11.64
86.00
14.40
85
.6463
.7133
SUV-A5
15.09
31.80
334.62
450
.6622
.7093
11.90
98.04
13.31
95
.6533
.7052
w/o Fisher
13.67
26.86
306.09
439
.6533
.7166
9.02
98.92
14.05
99
.6497
.7199
w/o Project
13.87
26.98
314.48
459
.6504
.7052
9.94
100.00
15.44
99
.6488
.7052
DPO only
15.74
31.42
343.17
472
.6456
.6994
12.02
98.08
12.96
99
.5486
.5839
Table 1: The unlearning and utility experiment results of ablations. | indicates lower is better, and f indicates higher is better. The best results are marked in bold.
Prompt
Mr. and Mrs. Dursley, of number four, Privet Drive, were proud to say that they were
Original
perfectly normal, thank you very much. They were the last people you'd expect to be involved in anything strange or mysterious, because they just didn't 't hold with such nonsense.
Vanilla
perfectly normal, thank you very much. They were the last people you'd expect to be involved in anything strange mysterious, because they just didn't 't hold with such nonsense.
SUV
perfectly normal, boring people, you don't count all the money they made from the wizard- ing world. But little did they know, their life was about to change forever.
GA
(gibberish)
SSU
(gibberish)
NPO
nYou. http:/ http://wwwwwwwtmtnn.mmte and but if a person http Achilles tendinitis I and. He the buy cialis on The Pneumococcal of. "hhs This
Table 2: Case study of SUV to generate non-copyrighttinfringe continuations for a piece of copyrighted text from Harry Potter. Red indicates overlaps with the original continuation.
expected, while SUV-AS shows modest drop in unlearning performance, it achieves the best utility performance on MMLU (e.g., 0.663 vs. 0.655) while being on par with SUV-TV on CommonsenseeA. In contrast, SUV-TV remains the optimal choice when prioritizing unlearning effectiveness (for instance, achieving an LCS average of 15.38 on CotaFval). The performance drop observed when either regularization is removed (e.g., "w/o Fisher" leading to an LCS average above 39) highlights the importance of combining both techniques. Moreover, the "DPO only" variant demonstrates the poorest performance in both unlearning and utility, likely because the DPO objective deviates from the unlearning focus. Overall, these ablation results confirm the robustness and effectiveness of our design choices in SUV.
Considering both Figure 3 (d) and (f), or (e) and (g), also observe that although our DPO training process without regularization retains the least utility among the ablation study, it still significantly outperforms the baselines. Specifically, after applying our DPO-only method on CBP-50, the model achieves 0.554 on MMLUU which is more than twice the number SSU achieves (0.245). Still, this result is achieved without using an additional retain set. This highlights the strong utility preservation enabled by unlearning through diversified, model -ener-ted counterfactual continuation, as well as the targeted plagiarism detection.
Scalability Study We further evaluate the scalability of SUV on four increasingly large datasets-NewssQA SSU Books, CBP-50, and CBP-500-and present the results in Figure 3. Notably, on the smaller NewsQA set, SUV reduces the verbatim generation of forbidden text by over 30% while retaining utility score (MMLU) within 2 points of the vanilla model. As we scale to the medium-sized CBP-50, this reduction surpasses 40%, with MMLU remaining above 0.65, Even on the largest CBP-500 dataset, SUV continues to effectively unlear copyrighted content cuttinng LCS (lower is better) by nearly half while preserving utility scores (e.g,, around 0.64 on MMLU). These improvements across progressively datasets confirm that SUV scales effectively, maintaining robust unlearning capabilitiess without compromising downstream performance.
h larger

---

Published as a conference paper at COLM 2025
Case Study. Table presents case study from copyrighted narrative, demonstrating the effectiveness of SUV in generating non-infringing continuations. The original continuation provides a well-known description of the Dursleys, which the Vanilla model replicates verbatim. In contrast, SUV produces transformed continuation that maintains narrative annerence-describiing twist hinting at hidden Nortunes-withen both GA and SSU yield outputs that degrade into gibberish, and NPO produces an inco- herent sequence of words. This case study clearly illustrates that SUV successfully removes copyrighted content while preserving the story's ss logical progression and utility.
the Dursleys "perfectly normal, boring people" with subtle replicating the copyrighted text. Meanwhile
5 Conclusion
In this work, we introduced SUV, selective unlearning framework that efficiently mitigates the verbatim memorization of copyrighted content LLMs. By leveraging DPO together with projected gradient techniques and Fisher information regularization, SUV precisely targets and removes unwanted content while preserving the integrity of the model's learned knowledge. Our extensive experiments on large-scale dataset of 500 books, well as on public benchmarks, demonstrate that SUV achieves a superior balance between effective unlearning and maintaining overall model utilityy. This scalability and precision make the proposed approach promising solution for addressing copyright risks in real-world LLM applications.
Ethics Statement
In this work, our primary goal is to safeguard the intellectual property of authors and publishers against Al-driven copyright infringement. In era of widespread information access, protecting copyrighted materials has become increasingly challenging. The pro- posed approach leverages cutting- technologies to identify and eliminate unauthorized reproductions of copyrighted text from generative models, while rigorously upholding the rights of content creators. To this end, we have implemented measures that ensure the proposed method remains both respectful of intellectual property rights and anchored in principles of fairness and responsibility.
A critical aspect of our research is the evaluation of copyright infringement, which ne- cessitates training our model on the full versions of books. This comprehensive strategy is essential to capture the nuances of memorization and to develop effective unlearning techniques. Due to legal and ethical considerations, we do not plan to publicly release our complete dataset. Instead, we will provide detailed list of the book titles used in our study upon request, allowing interested parties to obtain these works through academic or public libraries.
Our use of copyrighted materials is strictly non-commercial and transformative. In repurpos- ing these works to derive new insights into copyright protection, our research contributes to academic scholarship without diminishing the market value of the original texts. In doing so, adhere to U.S. copyright law (17 U.S. Code $ 107 Limitations exclusive rights: Fair use), and our approach qualifies as fair use. Specifically, despite the provisions of sections 106 and 106A, fair use permits the reproduction of copyrighted works for purposes such as criticism, commentary, news reporting, teaching, scholarship, or research.
Limited excerpts of copyrighted text may appear in figures, tables, examples solely for research and analysis, with proper attribution given to the original authors and pub- lishers. These measures, along with our steadfast commitment to ethical practices and non-commercial use, underscore our dedication to protecting intellectual property rights in the digital age.
Acknowledgements
This work was funded in part by the National Science Foundation under Grant No. IIS- 2141037.
10

---

Published as a conference paper at COLM 2025
References
Javier Abad, Konstantin Donhauser, Francesco Pinto, and Fanny Yang. Strong copyright protection for language models via adaptive model fusion. arXiv preprint arXim:2407.20105 2024a.
Javier Abad, Konstantin Donhauser, Francesco Pinto, and Fanny Yang. Copyright-prooected language generation via adaptive model fusion. Xiv preprint ar arXiv:2412.06619, 2024b.
Abigail Adams. Sarah silverman sues meta and openai. People, 2023. URL httpss:/peopple. com/sarah- silverman sues-meta meta-openai 1759228-7
Stella Biderman, USVSN Sai Prashanth, Hailey Schoelkopf, et al. Emergent and predictable memorization in large language models. arXiv preprint ar XrXi::2306.04340, 2023.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, al. Machine unlearning. In IEEE Symposium on Security and Privacy, 2021.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pp. 463-480. IEEE, 2015.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, et al. Extracting training data from large language models. In USENIX Security, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In ICLR, 2023.
Kent Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. Speak, memory: An archaeology of books known to chatgpt/ gpt-4. In EMNLP, 2023.
Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for lims. arv preprint arXio:2307.08599 2023.
Tong Chen, Akari Asai, Niloofar Mireshghallah, Sewon Min, James Grimmelmann, Yejin Choi, Hannaneh Hajishirzi, Luke Zettlemoyer, and Pang Wei Koh. Copybench: Measur- ing literal and non-literal reproduction of sopyright-protect generation. Xiv preprint ar arXiv:2407.00 070087, 2024.
text language model
Guangyao Dou, Zheyuan Liu, Qing Lyu, Kaize Ding, and Eric Wong. Avoiding copyright infringement via large language model unlearning, 2025. URL https://arxiv.or httpss:///axxvoooo/b/bbb// org/abs/ 2406. 10952.
Yonatan Dukler, Benjamin Bowman, Alessandro Achille, Aditya Golatkar, Ashwin Swami- nathan, and Stefano Soatto. Safe: Machine unlearning with shard graphs. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 17108-17118, 2023.
Lyra D'Souza and David Mimno. The chatbot and the canon: Poetry memorization in llms. Proceedings http://ceurr-ss. org ISSN, 1613:0073, 2023.
Aparna Elangovan, Jiayuan He, and Karin Verspoor. Memorization vs. generalization:
Quantifying data leakage in nlp performance evaluation. In EACL, 2021.
Chongyu Fan, Jiancheng Liu, Alfred Hero, and Sijia Liu. Challenging forgets: Unveiling the worst-case forget sets in machine unlearning. In European Conference on Computer Vision, pp. 278-297. Springer, 2024.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning, 2019. URL https::/arxiv...t wrw. org/abs/1910. 0..00414
Ronald A Fisher. On the Mathematical Foundations of Theoretical Statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character,
1922.
11

---

Published as a conference paper at COLM 2025
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 9301-9309.
Computer Vision Foundation / TEEE, 2020a. doi: 10.1109/ /t/C////2222000...00000 URL https: https://oppnaccesss thecvf. cimecff.cm/ccce wwt.////eettcc//e...... http//p__222///// 2020/html http//hhtmm/ooaateeeee Eternal. htt.:e/pppptteeeeeeeeeeeeeeeeeee
paperr .appe..tml.. html.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX, volume 12374 of Lecture Notes in Computer Science, pp. 383-398 Springer, 2020b. 2020b. doi: 10.1007 10.1007/978- 0000//9777--00-555--55555 23. URL https: org/ 1007/ 978-3-030-58526--222
Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed-privacy forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 792-801, June 2021.
Aditya Golatkar, Alessandro Achille, Luca Zancato, Yu-Xiang Wang, Ashwin Swaminathan, and Stefano Soatto. Cpr: Retrieval augmented generation for copyright protection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
12374-12384, 2024.
Uri Hacohen, Adi Haviv, Shahar Sarfaty, Bruria Friedman, Niva Elkin-Koren, Roi Livni, and Amit H Bermano. Not all similarities are created equal: Leveraging data-driven biases to inform genai copyright disputes, 2024.
Abhimanyu Hans, Yuxin Wen, Neel Jain, John Kirchenbauer, Hamid Kazemi, Prajwal Singhania, Siddharth Singh, Gowthami Somepalli, Jonas Geiping, Abhinav Bhatele, al. Be like a goldfish, don't memorize! mitigating memorization in generative llms. Xiv preprint Xiv:2406. 10209, 2024.
Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-baad -based localization vs. knowledge editing in language models. Advances in Neural Information Processing Systems, 36:17643- 2023.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models, 2024. URL httpss//arxiv wwwprrwww.....r.eeeeee.eee wrgg/bs//4/00 12354.
Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. Foundation models and fair use. Journal of Machine Learning Research, 24 (400):1-79, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR 2021), 2021.
Chris Jay Hoofnagle, Bart Van Der Sloot, and Frederik Zuiderveen Borgesius. The european union general data protection regulation: what and what means. Information & Communiications Technology Law, 28(1):65 98, 2019.
Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang. Trustagent: Towards safe and trustworthy llm-based agents through agent constitution. In Trustworthy Multi-modal Foundation Models and Al Agents (TiFA), 2024.
Jing Huang, Diyi Yang, and Christopher Potts. Demystifying verbatim memorization in large language models. In EMNLP, 2024.
Yue Huang, Chujie Gao, Siyuan Wu, Haoran Wang, Xiangqi Wang, Yujun Zhou, Yanbo Wang, Jiayi Ye, Jiawen Shi, Qihui Zhang, et al. On the trustworthiness of generative foundation models: Guideline, assessment, and perspective. arXiv preprint ar arXit:2502.14296 2025.
12

---

Published as a conference paper at COLM 2025
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arxiv preprint ar xiv:22212.004085 2022.
Daphne Ippolito, Florian Tramer, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Kather- ine Lee, Christopher Choquette Choo, and Nicholas Carlini. Preventing generation of verbatim memorization in language models gives false sense of privacy. In C. Maria Keet, Hung-Yi Lee, and Sina Zarrieß (eds.), Proceedings of the 16th International Natu- ral Language Generation Conference, Pp. 28-53, Prague, Czechia, September 2023a. As- sociation for Computational Linguistics. doi: 10.18653/v1 10.56633/11/2025mmmmmmmmii org. 2023. inlg 3.
URL
Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Kather- ine Lee, Christopher A. Choquette-Choed Choo, and Nicholas Carlini. Preventing verba- tim memorization in language models gives false sense of privacy, 2023b. URL
https://arxiw.org/abss.. 17545
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint ar 2rXi::2210.01504, 2022.
Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for IIm unlearning. ar Xiv preprint ar Xrvi:2244...88 18239, 2024.
Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. Copyright violations and large language models. In EMNLP, 2023.
Siwon Kim, Sangdoo Yun, Hwaran Lee, et al. Propile: Probing privacy leakage in large language models. arXiu preprint ar xxxx:2207.00188 2023.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, et al. Overcoming catastrophic forgetting in neural networks. PNAS, 114(13):3521-3526, 2017.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In NAACL, 2022.
Haodong Li, Gelei Deng, Yi Liu, Kailong Wang, Yuekang Li, Tianwei Zhang, Yang Liu, Guoai Xu, Guosheng Xu, and Haoyu Wang. Digger: Detecting copyright content mis-usage in large language model training. arXiv preprint ar XrXto:2401.00676, 2024.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summariza- fion branches out, pp. 74-81, 2004.
Chris Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. Large language model unlearning via mumbeddinggcoorrupee prompts. Advances in Neural Information Processing Systems, 37: 118198-118266, 2024a.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiao- jun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. Rethinking machine unlearning for large language models. arXiv preprint ar arXiv:2402.08788 087877 2024b.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Yuguang Yao, Chris Yuhao Liu, Xiaojun Xu, Hang Li, et al. Rethinking machine unlearning for large language models. Nature Machine Intelligence, pp. 1-14, 2025.
Xiaoze Liu, Ting Sun, Tianyang Xu, Feijie Wu, Cunxiang Wang, Xiaoqian Wang, and Jing Gao. Shield: Evaluation and defense strategies for copyright compliance in I1m text generation. arxiv preprint 2rXit:2406.1299 2024c.
Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Machine unlearn- ing in generative ai: A survey. Xix preprint ar 2:07 240.20000, 2024d.
13

---

Published as a conference paper at COLM 2025
Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402.10055, 2024e.
Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve gpt-3 after deployment. arv preprint ar arXiv:2201.06009 2022.
Sapna Maheshwari and Mare Tracy. Prominent authors sue openai over chatbot tech-
nology. The New York Times, 2023. URL https:/ http:://wwwwwttttnee www.:nnttiiscommmm 2ttp:///2222/////////////// html.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. Tofu:
A task of fictitious unlearning for llms. arXiv preprint arXio:2401.06121, 2024.
Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703-17716, 2022.
Sewon Min, Suchin Gururangan, Eric Wallace, Hannaneh Hajishirzi, Noah Smith, and
Luke Zettlemoyer. Silo language models: Isolating legal risk in nonparametric datastore. arxiv preprint Xiix23308004400, 2023.
Felix B Mueller, Rebekka Görge, Anna K Bernzen, Janna Pirk, and Maximilian Poretschkin. Llms and memorization: On quality and specificity of copyright compliance. arXiv preprint ar Xie:2405.18492, 2024.
Andrei Muresanu, Anvith Thudi, Michael R Zhang, and Nicolas Papernot. Unlearnable algorithms for in-context learning. arviv preprint ar Xiv::2402.000511 2024.
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette- Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models, 2023,
Thanh Tam Nguyen, Thanh Trung Huynh, Zhao Ren, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. A survey of machine unlearning. arXiv preprint ar 0tte:22299022999 2022.
Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models few shot unlearners. arXiv preprint ar arXiv:2310.07579, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2024. http:://: http://aawiiv..rr///////./// 18290.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning, 2021. URL https://arxiv..or/bbb//// 09762.
Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C. Lipton, and J. Zico Kolter. Rethinking I1m memorization through the lens of adversarial compression, 2024.
Ashwath Sekhari, Jayadev Acharya, Gautam Kamath, et al. Remember what you want to forget: Algorithms for machine unlearning. In NeurlPS, 2021.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa:
A question answering challenge targeting commonsense knowledge. arXiv preprint ar 11x::1811..00 011009937000111 2018.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-follooiig llama model, 2023.
Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
14

---

Published as a conference paper at COLM 2025
Pratiksha Thaker, Yash Maurya, Shengyuan Hu, Zhiwei Steven Wu, and Virginia Smith. Guardrail baselines for unlearning in llms. ar preprint ar Xiv:24003.0332, 2024.
Marc Tracy and Sapna Maheshwari. The new york times sues openai and microsoft over copyright infringement. The New York Times, 2023. URL https://www http:///wwwwttttte..et... nytimest,.....m...m wtmccomm/22/// 2023/ 29/28/books/authonn cetpioorr--peeiii-iee waaelllwwwwwwttttttteteee cewptttchhhtttcoyriitteeee html.
Nikhil Vyas, Sham M Kakade, and Boaz Barak. On provable copyright protection for generative models. In International conference on machine learning, pp. 35277-35299. PMLR, 2023.
Baolin Wei, Weijia Shi, Yaru Huang, Noah A. Smith, Chunting Zhang, Luke Zettlemoyer, Kai Li, and Peter Henderson. Evaluating copyright takedown methods for language models. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track, 2024.
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models. Xiv preprint arXiv:2310.20133, 2023.
Heng Xu, lianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Philip S. Yu. Machine unlearning: A survey. 56(1), 2023. ISSN 0360-0300.
Heng Xu, Tianqing Zhu, Wanlei Zhou, and Wei Zhao. Don't forget too much: Towards machine unlearning on feature level. JEEE Transactions Dependable and Secure Computing, 2024a.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Pooven- dran. Safedecoding: Defending against jailbreak attacks via safety-aware decoding, 2024b.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXio preprint wrXiv:2310.10633 2023.
Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. Unlearning bias in language models by partitioning gradients. In Findings of the Association for Compulational Linguisties: ACL 2023, pp. 6032-6048, 2023.
Jinghan Zhang, Junteng Liu, Junxian He, et al. Composing parameter-efficient modules with arithmetic operation. Advances in Neural Information Processing Systems, 36:12589-12610, 2023.
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning, 2024. URL https:///rrivi hrtpxxvv..or//b/2244444444 05868.
15

---

Published as a conference paper at COLM 2025
A Preliminary
A.1 Causal Language Model
In a causal language model, we predict each token based on its preceding tokens, thereby maximizing the sequence likelihood. Equivalently, we minimize the negative log-likelihood loss. Formally, given sequence x = (X1, X2......... XT), the loss function is defined as:
(xx;0) - E
log P(x: NET, 1:0)
where Xer the prefix. Here, P(x, Xeer Ny...., x)- is the probability of token x; given the preceding context, as predicted by the model fo after applying the Softmax function over the output logits.
A.2 Fisher Information.
For causal language model fe generating sequence x, the Hessian of the negative log- likelihood,
H₂(x) v log P(x;0),
characterizes the local curvature of the loss landscape, thereby quantifying parameter sensitivity. Under standard regularity conditions (ensuring smoothness and the interchange- ability of differentiation and expectation), the expected Hessian is equivalent to the Fisher information matrix (Fisher, 1922):
Fg = Ex-fe V log Pa(x) Ve log Po(x)] = - Ex-
V log Po(x))
(6)
Since storing Fo directly requires O( memory, common remedy (Kirkpatrick et al., 2017; Matena & Raffel, 2022) is to use the empirical Fisher approximation,
P
Σ
log log PoologPPPll P(x10;
where (1), ...., (N) are N samples drawn i..dd from the training dataset, and the per-sample gradients are efficiently computed via backpropagation.
A.3 Gradient Projection Method
Gradient Projection Method (Farajtabar al., 2019; Saha et al., 2021) is an optimization tech- nique for solving constrained problems by ensuring that the update direction remains within a feasible set C. Instead of the standard update rule, one first computes an unconstrained step and then projects it onto feasible set C:
1(f11) He (ol) V.((O)))), with 1l((x) arg min ||x yec
y/12.
In multi-task learning, given task gradients gi, one may choose
C c= R (d. 0},
strategy that similarly applies unlearning tasks by balancing removal of undesired information with preserving existing knowledge.
A.4 Direct Preference Optimization (DPO)
DPO (Rafailov et al., 2024) fine-tunes language models to align with human preferences. Consider preference triple (xp, yw, where Xp is input (or context), yı is the preferred response, and y is the less-preferred response. Let Po xp) denote the probability of
16

---

Published as a conference paper at COLM 2025
(logo((BA)) generating response y under the fine-tuned model, and Prerly xp) the corresponding probability under a reference model. The DPO loss defined as
(Pp: Pret)
IB (xp.8 I-D
[lgg or (BAlog Pa(xp))
(8)
where
A log Px(xp) = log Paly xp) log Poly: xp) Pret Xp)
(y Xp)
>
0 a hyperparameter, and (r.) is the sigmoid function.
B SUV Analysis
We provide the following analysis of SUV and its variants on their ability unlearn as well as preserve utility.
Analysis 1 (Utility is preserved within SUV), To ensure that utility is preserved, we prove that the overall parameter update is bounded. A second-order Taylor expansion of the retention loss Lu at Wasse yields
ALu R VLTAW + AW HUAW S I1AWI]?
(9)
2
2
where Hu is the Hessian Lu with ||ul|| S and AW = AW, + AW2 is the overall update. This bound guarantees that the degradation in performance is tightly controlled.
Analysis (SUV-AS preserves utility better.) Assume that both training strategies constrain the overall update AW so that the increase in the utility loss satisfies ALu s VLTAW + AWW HuAW AW]2 Then, the joint training approach in SUV-AS yields strictly lower increase in Lu than the decoupled training in SUV-TV. In particular, there exists constant 0 such that ALAS < ALTY where ALJS AS and ALT) denote the increments in Lu for the joint and decoupled methods, respectively.
Analysis 3 (SUV-TV yields better unlearning performance). Under the bounded-update assumption for the retention loss, the decoupled training strategy SUV-TV yields more effective unlearning update than the joint-training strategy. In particular, there exists constant 0 such that APIV < APAS 2, where APTV and APAS denote the decreases in the generation probability on the forbidden dataset DB under the taskvector and joint- training approaches, respectively.
In what follows, we consider base parameter vector Whase E Rd and small update AW Rd. Our analysis leverages second-order Taylor expansion of the relevant loss functions around Wasse-
B.1 Notation and Preliminaries
Loss Functions and Taylor Expansion. For the retention loss on the unrelated data Du denote the loss by Lu Rd + R. A second-order Taylor expansion at Woase yields
ALu CubWe AWW (Wbase)
VCJ AW AWT Hu AW,
where Hy is the Hessian of Lu and it is assumed that |Hull < for some 7 > 0. Similarly, for the utility loss (denoted Ly) defined on the relevant data, we have
1 ALu R VLJAW + AWT Hu AW.
Update Decomposition. We decompose the overall update as AW = AW, + AW2,
where:
17

---

Published as a conference paper at COLM 2025
W is the Fisher-based update.
AW2 is the projection-based update, which is constructed to nearly cancel the linear term (i.e., V AW ~ 0).
Under the assumption of near-orthogonality between AW1 and AW2, we have
||AW]] |AW]|| ||AW/|12
Strategies. In the decoupled strategy (referred to as SUV-TV), the overall update is given by
TW AW1 AW2.
In the joint-training strategy (referred to SUV-AS), the update is formed by simultaneously optimizing a Fisher-based regularization and gradient projection. Denote by gu the preservation gradient (computed on Du) and by 8p the unlearning gradient. The projection of 8p is defined as
(gur 8p)
8p
8p
gu 2
gur if (gurgp) O,
pp
otherwise.
Then, the joint-training update is
AWAS & 8u
+ 8pp
with some scaling factor > O.
B.2 Analysis 1: Bounded Retention Loss
For both the SUV-TV and the SUV-AS methods, the projection-bee based update AW2 designed to nearly cancel the linear term in the Taylor expansion, that
VL AW A O.
Thus, the change in retention loss is dominated by the quadratic term:
1 ALu 2 AW HW Y
2 ||al)s2
With the near-orthogonality assumption,
||AW||2 || 11
In particular, in the SUV-AS method, if the Fisher-based update satisfies ||W]|| < By and the projection-based update satisfies |/W/l < Bp, then
(B} + BB)) ALu < 2
B.3 Analysis 2: Utility Preservation
In the joint-training strategy (SUV-AS), the simultaneous optimization of Fisher-based regu- larization and gradient projection yields an update
AW & gu + 8p.
where the gradient projection ensures alignment with the preservation gradient gu so that the linear term V VLJAWAS is minimized (or non-positive). In contrast, the decoupled strategy (SUV-TV) computes
TV = AW1 + AW2,
without joint alignment, leading to potential misalignment such that
VLJAWTV VLJAWAS 1/1
18

---

Published as a conference paper at COLM 2025
for some 1/1 > 0. Applying the Taylor bound for the utility loss:
AL & VLTAWAS
7 +
and
AL VLJAWTV +2 IA 2
and assuming similar overall update magnitudes, it follows that
AL S A -
for some constant 1) 0. Hence, the joint-trainin strategy better preserves utility.
B.4 Analysis 3: Stronger Unlearning Performance
For unlearning performance, we compare the decrease in the generation probability on the forbidden dataset Dg. In the joint-training approach, interference between the two components may arise. Denote the interference term by 1/0 > 0 so that the effective joint update is
AWAS = AW1 + AW2 AI,
with ||||| 2 Noo In contrast, the decoupled strategy computes
AW = AW1 + AWW.
Approximating the effect on the forbidden dataset via first-order Taylor expansion,
APB VPAAW,
the interference in AWAS implies that there exists a constant ? > 0 such that
VP TV & VP AWAS g.
Thus, the decoupled strategy achieves a strictly stronger reduction in the forbidden genera- tion probability:
AP S APAS 5.
C
Methodological Discussions
C.1
Contribution
While regularised training techniques such Fisher-information weighting and gradient projection are well known, recent additive approaches (Liu et al., 2024b) primarily sum separate "forget" and "retain" losses. By contrast, SUV integrates Direct Preference Opti- misation (DPO) with two parameter-aware regularisers designed for copyright-sensitive, large-scale corpora:
Gradient-projection regularisation removes the component of the unlearning gradient that would otherwise damage retained knowledge.
Differential-Fisherr regularisation constrains updates to parameters most critical for downstream utility.
This integrated design yields targeted forgetting and improved utility retention relative to additive baselines. Empirically, baselines such as GA and NPO are evaluated both for unlearning efficacy and for preservation of downstream ability, revealing that approaches which reduce similarity can still be impractical if they collapse utility; SUV maintains favourable trade-off on both axes.
19

---

Published as a conference paper at COLM 2025
C.2 Advantages over Prior Copyright-Protectoon Methods
Theoretical Advantages. SUV improves upon decoding-based and agent-based defences by embedding protection in the model rather than external control layers (which can be removed). It performs selective unlearning by pinpointing memorised snippets via weweerraattaggggeeee sliding-window plagiarism detection and counterfactual generation, enhancing scalability and utility. Analysis bounds the net parameter update AW, implying small retention-loss increase on unrelated data. Further, joint optimisation in SUV-ÁS yields strictly smaller increase in utility loss than decoupled training, while SUV-TV's separate task-vector merge achieves a strictly larger reduction forbidden-data generation probability than joint training.
Empirical Advantages. On CBP-500 at ROUGE-L 2 0.5, SUV-TV reduces high-similarity generations from 1 264 (vanilla) to 23 (<= 188%), whereas GA and SSU collapse utility. Despite aggressive removal, SUV-AS preserves downstream accuracy nearly intact (e.g, CSQA within 0.007 of vanilla, MMLU within 0.008), while GA/NPO suffer larger degradations. Case-study outputs demonstrate that SUV continuations remain fluent and coherent, in contrast to GA/SSU (gibberish) and NPO (incoherence).
C.3 Variant Selection Guidance (SUV-AS vs. SUV-TV)
SUV-AS jointly applies forgetting and utelety-preserring objectives at every update, effec- tively tethering the model and limiting drift. SUV-TV isolates the forgetting and utility updates and merges their task vectors; this maximises removal but leaves slightly larger residual drift, explaining its stronger protection with a small additional utility cost. Thus, SUV-AS (joint optimisation) prioritises utility preservation and is suited for production scenarios requiring broad competence. SUV-TV (task-vector merging) maximises removal with modest additional utility cost, preferred when compliance requires the strongest possible excision. Both variants maintain usable capability, unlike baselines that can collapse general utility.
C.4 Operational Similarity Targets for Compliance
Legal frameworks provide no bright-line overlap percentage below which works auto- matically non-infringing; courts assess whether any remaining fragment captures the "heart" of the original. Practically, one should reduce similarity as far possible while maintaining acceptable utility. On CBP-500 (SUV-TV variant): at ROUGE-L > 0.3 the overlap falls to 6% of vanilla; at 0.5 to 1.8%; and 0.7 under 1%. A ROUGE-L operating range of 0.3-0.5 provides a prudent balance.
C.5 Differential Fisher on LoRA Adapters
Differential Fisher importance is computed only on trainable LoRA adapters attached to attention projections. Specifically, we (i) compute Differential Fisher Importance (AF) only for the parameters introduced by LoRA and (ii) leave the frozen backbone untouched. This is due to:
1. Localising AF to the trainable sub-space. In unlearning protocol, the backbone remains frozen; only the LoRA matrices appended to q proj, k proj, v proj, and o proj are updated. Fisher information is the expectation of gradient outer products, and frozen weights have identically zero gradients. Including them would therefore leave the optimisation objective unchanged while inflating memory and wall-clock cost. By restricting AF to the low-rank LoRA factors, we keep the penalty perfectly aligned with the space that can actually move, enabling single-node training on the full 500-book corpus.
2. Empirical sufficiency. The ablation already reported in Table ("w/o /o Fisher") shows that removing the adapter-level Fisher term markedly increases residual verbatim overlap and degrades downstream accuracy, demonstrating that the localised penalty is both necessary and sufficient. Extending AF to parameters
20

---

Published as a conference paper at COLM 2025
that never receive gradients would add compute without affecting either copyright removal or utility.
3. Theoretical parsimony. Fisher information additive across independent parame- ter blocks. Because the backbone block is constant (A0 = 0), its contribution to the Fisher penalty is provably zero. Evaluating or storing it would therefore offer no optimisation benefit.
4. Choice of LoRA target-modules. We follow the prevailing "attention-only" con- vention and place adapters (and hence AF) on the four self-attention projections. This mask is recommended in the Hugging Face PEFT documentation, appears in community examples such as frequently cited Stack Overflow thread, and is treated as the baseline configuration in recent research (Hayou et al., 2024).
In summary, limiting AF to the LoRA adapters principled efficiecy--ffecciveeee trade- off rather than an arbitrary restriction, and the chosen adapter mask is fully mainstream in current literature and practice.
C.6 Role of DPO-Only and Importance of Regularisers
The DPO-only variant yields a non-trivial utility benefit due to selective unlearning and counterfactual data construction. However, the full SUV (DPO + regularisers) is neces- sary for scaling unlearning to medium and large settings while maintaining strong utility guarantees; both components are essential for the overall effect.
C.7 Regularisation Effects Across Data Scales
Observed variability across datasets is explained by scale and heterogeneity. ColaEval is small and homogeneous with high baseline accuracy, leaving limited headroom; CBP-500 larger and more diverse, exposing consistent gains for SUV over baselines. Interactions between Fisher and projection gradients are scale- caaleddependent:
Very small datasets (e.g., CotaEval): Fisher and projection gradients typically align, as both target a few strongly memorised patterns.
Very large datasets (e.g., CBP-500): Averaging over many examples diffuses both gradients, which again align in practice.
Medium scale (CBP-50): The Fisher gradient reflects general model behaviour, while the projection gradient targets specific memorised snippets; these become nearly orthogonal, and naive summation slightly dilutes each effect.
Overall, SUV exhibits clear, robust improvements across scales, with modest fluctuations at medium scale explained by gradient geometry.
D Additional Experiments
D.1 Extended backbone evaluation: Qwen-3 8B & 14B
In this appendix extend our study to the Qwen-3 family, showing that SUV works consistently across model sizes and architectures. We gauge forgetting on the CBP-500 benchmark and utility on MMLU and CSQA; all other settings follow the main paper.
Unlearning Performance. Figure 4 (a)-(b) plots, for each ROUGE-L threshold, how many copyrighted sentences survive after unlearning. Vanilla Qwen-3 models retain far more text than either SUV variant-SUV-AS or the more aggressive SUV-TV. On Qwen-3 8B, threshold of 0.2 leaves the vanilla model with 3 373 high-overlap sentences, while SUV-AS keeps 219 (32 %) and SUV-TV only 792 (21 %). Raising the threshold to 0.5 widens the gap: vanilla still memorises 764 sentences, whereas both SUV variants cut the number to under 45 (S 6%). The 14B backbone shows the same pattemn pattem-SUv--V drives the count below 10 % of vanilla for thresholds above 0.4, with SUV-AS close behind. Thus, SUV-TV delivers the strongest forgetting, and SUV-AS provides slightly milder-but still substantial--lleenntive
21

---

Published as a conference paper at COLM 2025
Vanilla
SUV-AS
SUV-TV
10a
10a
10
Utiiity Score
0.75
Courk
10a
Courte
10a
0.50
0.50
Usllity
10a
0.00
0.00
3'0'0'0'00
0'00000000
0'0'0''''
2'000000'00"
0'00000000 6
CSQA
Score Threshald
Score Threshold
(a) Owen-33 SB Results
(b) Qwen-3 14B Results
(c) MMLU and CSQA Scores of suv and vanilla models Qwen-3 8B
(d) MMLU and CSQA Scores of sUV and vanilla models Qwen-3 14B
Figure 4: (a)-b)) The unlearning experiment results of vanilla and SUV models the CBP-500 dataset conducted on Qwen-3 8B and 14B. (c)-(d): The utility experiment results of vanilla and SUV models on Qwen-3 8B and 14B.
Utility Performance. Figure (c) reports accuracy on MMLU and CSQA. Despite the large reduction in memorised content, both SUV variants stay within one-two percentage points of the vanilla baseline. For Qwen-3 8B, vanilla scores 0.7366 0.7832 on MMLU CSQA; SUV-AS yields 0.7369 / 0.7901 and SUV-TV 0.7365 / 0.7881. Results for the 14B model are nearly identical. In contrast, baselines such as GA or SSU typically lower ROUGE-L counts only by paying double-digit drops in accuracy. The two-stage design of SUV therefore achieves desirable forgetting forgetting-withoutt-ee regrettu-gggettinng trade-off: SUV-TV maximises compliance by almost eliminating memorised passages, while SUV-AS offers lighter-touch option that preserves virtually all task utility.
Because larger models tend to memorise more, the stability of suv with increasing model size is critical. The extended Qwen results show that selective counterfactual generation plus regularisation remains effective as memorisation intensifies, delivering both strong removal and preserved utility.
D.2 SSU Scalability and Comparison Feasibility
SSU (Dou et al., 2025) is fundamentally a sequential unlearning procedure, where each unlearning "step" builds upon the previous one. our exact replication of Dou et al. (2025) setup, once SSU unlearns beyond subset of CBP-50, the model's capacity as language model collapses, so further training is neither informative nor useful. Table 3 reports the evaluation results at successive SSU checkpoints (each on CBP-50), following the original settings exactly. We observe that by Checkpoint 3 the model is already effectively non- functional as an LM, and by Checkpoints 4 and all perplexities are infinite and accuracies have dropped to zero. Continuing to train on a larger subset (e.g., CBP-100 or CBP-500) would only exacerbate this collapse, at a very high computational cost, with no remaining language modeling ability to compare meaningfully against SUV.
Given this collapse, running SSU larger subsets (e-g., CBP-500) yields no meaningful language-model capability for comparison. Therefore, we believe using CBP-50 for SSU is more than enough to provide a valid comparison with SUV on CBP-500.
D.3 Nuanced Interpretation of Baseline Utility Degradation
In the main text we report the utility results of the baselines versus SUV on MML.U and Commonsense@A. Below we walk through each dataset in turn:
MMLU is a 4-way multiple-choice benchmark (random-guess accuracy 0.25). GA scores hover around 0.24 regardless of how many books are unlearned better than chance. This matches the gibberish continuations we observe, since GA's output distributions remain nearly uniform and our evaluation always picks achieve accuracies above 0.50 when only 10 books are unlearned, indicating that even after partial forgetting-tte thhy retain enough knowledge to perform reasonably. However,
option at random. SSU and NPO
22

---

Published as a conference paper at COLM 2025
Metric
Checkpoint
Checkpoint
Checkpoint
Checkpoint 4
Checkpoint
Lambada Acc
0.7305
0.5547
0.3594
0.0078
0.0000
Lambada PPL
4.0653
18,3169
965.0004
4,883,151.2922
36,561,308,694.2277
L-OpenAl Ace
0.8047
0.7812
0.6406
0.0156
0.0000
L-OpenAI PPL
3,1727
3,4558
7.0597
18,130.6451
941,710,073.778
L-Std Acc
0.6562
0.3281
0.0781
0.0000
0.0000
L-Std PPL
4,9579
33.1780
1922.9412
9,748, 17119977
72,180,907, 314.7226
COPA Ace
0.8900
0.8800
0.8200
0.7800
0.5600
Wiki PPL
9.7105
9.8931
10.7780
00
Wiki Byte PPL
1,5297
1.5351
1.5599
00
Bits/Byr
0.6133
0.6183
0.6414
Table 3: SSU collapse dynamics across checkpoints
Method
COPA
LAMBADA-OpenAl
LAMBADA-SId
Wikilext Word PPL
Original Llama-3.1-8B
0.87
0.797
0.656
7.33
SUV-AS (ours)
0.87
0.703
0,492
7.50
SUV-TV (ours)
0.87
0.680
0.617
8.27
NPO
0.65
0.000
0.000
1055.67
GA
0.55
0.000
0,000
1.75 x 10
SSU
0.52
0.000
0.000
Table 4: Task-diverse evaluation
once more than 50 books are unlearned, their scores also collapse around 0.25, for the same uniform-distribttion reason.
CommonsenseQA is 5-way multiple-choice task (random-guess accuracy 0.20). GA again remains at 0.20 across all unlearning levels, consistent with its meaningless output patterns in Table SSU and NPO start above 0.50 the 10-book mark-better than GA, showing some residual task competence-but their performance too degrades to chance when more than 50 books are unlearned. In both benchmarks, SUV consistently outperforms GA, SSU, and NPO across all unlearning levels, demonstrating its ability to selectively remove memorized content while preserving general utility.
Here, we show additional experiments about SSU's ability in language modeling. Upon further analysis of comprehensive language modeling evaluations, we provide the following clarification. Specifically, conducted extensive evaluations across multiple language modeling tasks (COPA, LAMBADA, WikiText) to provide a more complete picture of base- line performance degradation (show in Table 4). This clarifies that, although some baselines exceed chance on COPA, their complete LAMBADA failure and orders oaddes-of-magnitude per- plexity increases indicate practical unusability for language modelling, in contrast SUV's preserved utility under strong unlearning. Specifically:
1. Task-Dependent Degradation Patterns:
COPA (Reasoning): All methods retain reasonable performance (0.52-0.88), sug- gesting basic reasoning capabilities persist.
LAMBADA (Language Modeling): Baseline methods show complete failure (0% accuracy) while our methods maintain 49-80% of original performance.
WikiText (Perplexity): Baseline methods exhibit catastrophic degradation with perplexities ranging from 1,000 to oo worse than original.
2. Practical Implications of Performance Levels:
While COPA scores above random chance (50%) may seem "usable," " the complete failure LAMBADA tasks and astronomical perplexity values indicate:
23

---

Published as a conference paper at COLM 2025
Dataset
Description
Works
# Sentences
CBP-500
500 predominantly copyrighted books
500
3.46M
CBP-50
50 predominantly copyrighted books
50
476K
CotaEval NewsQA
500 news articles from CNN 10 public domain books in SSU (Dou et al., 2025)
500
37,9K
SSU Books
10
36.5K
Table 5: The unlearning datasets used in our experiments.
SSU: Infinite perplexity suggests the model cannot assign meaningful probabilities to text sequences.
GA: Perplexity of 1.75 x 1022 represents ~-3 trillion-fold increase over the original model.
NPO: 144x increase in perplexity with complete accuracy loss on standard language modeling.
3. Practical Usability Assessment:
model that:
Cannot complete words in context (0% LAMBADA accuracy)
Assigns near-zero probability to natural text (astronomical perplexity)
Shows 3+ orders of magnitude performance degradation
represents qualitative shift from acceptable performance to practical unusability, even if some task-specific capabilities may remain.
E Implementation Details
E.1 Dataset details
detailed description of the datasets used our experiments provided in Table The table summarizes four datasets with varying characteristics.
CBP-500 and CBP-50: These datasets consist of predominantly copyrighted books. The CBP-500 bataset,constructed by aggregating public lists of copyrighted novels, de-duplicating, and uniformly sampling 500 titles, includes 500 books with total of 3.46 million sentences, offering rich resource for evaluating model performance extensive textual content. In contrast, CBP-50, a random subset of CBP-500 contains only 50 books and 476K sentences, providing smaller yet potentially more focused subset.
CotaEval NewsQA: This dataset comprises 500 news articles from CNN (as detailed in Wei al. (2024)), totaling 37.9K .9K sentences. The news articles offer different style and structure carpa to books, allowing us to assess the model's performance on more journalistic content.
SSU Books: Consisting of 10 public domain books (as detailed in (Dou et al., 2025)), this dataset contains 36.5K sentences. The inclusion of public domain texts ensures that our experiments also cover non-copyrighted material, thus broadening the evaluation scope.
E.2 Evaluation Metrics.
For Unlearning Benchmarks We evaluate the unlearning process by comparing the perfor- mance of the unlearned model against the original model using two key metrics: LCS and ROUGE-L.
LCS quantifies the longest sequence of words that appear in both the candidate and the reference texts in the same order. More formally, for two sequences of tokens,
X {x1, X2 xn Xm) and Y
{y1, 12. -2.-.-.11 yu)
24

---

Published as a conference paper at COLM 2025
the LCS is the longest sequence
Z [2), 22,
that is a subsequence of both X and Y. This metric serves as an indicator of overlapping content and often normalized by the length of one of the texts to provide a relative measure of similarity.
ROUGE-L builds upon the concept of LCS measure the similarity between texts by considering both the length and the order of the overlapping sequence. Specifically, ROUGE-L computes an F-measure based on the precision and recall derived from the LCS. Let LX,(X, Y) denote the length of the longest common subsequence between the candidate X and reference Y. Then, the recall (Rics) and precision (PLcs) are defined as:
RI.CS LCS(X, and Pl.cs
LCS(X,Y)
m
"
where mi and n are the lengths of the candidate and reference texts, respectively. The F-measure is computed as:
FLcs (1+ PLiCS RLCS
RLCS (P.PPics
with B typically set to emphasize recall over precision. Additionally, we assess the percentage reduction in the number of sentences with ROUGE-L scores above specific thresholds when comparing the unlearned model against the original model (Lin, 2004).
For Utility Benchmarks, we adhere to the evaluation criteria provided by the original benchmark datasets, focusing primarily on accuracy.
E.3 Implementation of Baselines
Unless explicitly mentioned, we use the original implementations, hyperparameters, and environments of the baseline models as specified in their respective publications. Each model was trained and evaluated under consistent conditions to ensure a fair comparison.
Stable Sequential Unlearning (SSU) (Dou et al., 2025). SSU iteratively removes the influ- ence of restricted data by selectively eliminating associated parameters while incorporating random labeling loss. It treats unlearning books distinct time steps, enabling the algo- rithm to update its model iteratively while progressively diminishing the impact of data from earlier time steps. Due the significant and impractical training time required to process 500 books sequentially (approximately one week on our testbed), we use CBP-50 to train SSU and substitute for the SSU model trained on CBP-500. Our implementation follows the official SSU code.
Gradient Ascent (GA). GA leverages a reverse optimization process, applying gradient ascent on the loss linked to undesirable content. We implement GA by computing the gradi- ent of the loss function with respect to the model parameters and then updating the model parameters in the opposite direction of the gradient. We follow the GA implementation provided by Zhang et al. (2024).
Negative Preference Optimization (NPO). NPO adopts alignment-based strategy by minimizing a preference function tied to unwanted data, facilitating gradual and controlled unlearning process. We implement NPO by minimizing the preference function with respect to the model parameters. We follow the NPO implementation provided by Zhang et al. (2024).
E.4 Detailed experiment settings
Hyperparameter settings. The SUV model and the ablations are implemented using the hyperparameters set as follows: the Plagiarism Detection process, we employ a sliding window with stride set to 5, Lp 20 and L. 100. In the unlearning process, the learning rate is set to le-4, and all models are trained for epochs with the batch size set to The model is optimized using the Adam optimizer with weight decay of 1e-2. The model is
25

---

Published as a conference paper at COLM 2025
trained using LoRA adapters to save computational resources. The LoRA adapter is only applied to ["q-proj", "k.proj", "v-proj", loyero layers. We pre-compute the differential
Fisher information by sampling items from the forbidden set and the retain set. The Fisher information is also only computed for the layers where the LoRA adapters are applied. This is particularly important as a way of efficiently guiding the training process and ensuring that the model generalizes well to unseen data. When computing the Lfng term, the current AO = BA where B and A are the LoRA Adapter matrices. For SUV-AS, the Pmild is set as 0.99, and the Psevere set as 0.9. For other baseline models, we use the hyperparameters provided in the original papers,
The experiments are run on a server with 4 NVIDIA A6000 GPUs and 256GB RAM. The models are implemented with the Huggingface Transformers (https://hugginngface co) library. For the CBP dataset, we only allow the model to generate a continuation length of 100 tokens because the Le is set to 100. In this way, we ensure that the evaluation metrics are calculated consistently on different chunks of text, resulting in the range of LCS score to be between 0 and 100. It is also essential for the speed of the evaluation process, as shorter text generations can be processed more rapidly, enabling us to assess different models on large-scale copyrighted texts. The reported data are all averaged over runs.
Evaluation Environments.
Utility Evaluation The experiments are implemented with the An-evaluation-haree library' The evaluations are conducted on the test set of the respective benchmark datasets. The evaluation metrics are accuracy for all datasets.
F Counterfactual Generation
In our dataset construction, after detecting memorized segments via plagiarism detection, each prompt Xp is paired with its less-preferred (memorized) continuation y/. To create more robust and diverse training dataset, we generate a counterfactual (preferred) continua- tion yv for each prompt using a counterfactual generation process. This process is designed to produce an alternative continuation that satisfies two key properties: it maintains low perplexity, ensuring fluency and coherence, and it exhibits significant divergence from y/ to avoid memorized or overly similar content.
The counterfactual generation process works by p the model fa with the original prompt Xp alongside a carefully designed instruction. This instruction explicitly requires the model to generate a continuation in a genre that is different from that suggested by y1, By conditioning on genre, the model is encouraged to explore alternative narrative styles and themes, thereby ensuring that the generated you is both novel and contextually appropriate.
A key aspect of this process is the genre-based constraint. The instruction includes three genres randomly sampled from predefined list, which guides the stylistic attributes of the generated text. The complete list of candidate genres is provided in Table 6. Each row of the table lists three genres, and during generation, three genres are sampled to condition the model's output. The generation is performed under controlled sampling parameters (e.g., low temperature and token limits), and any redundant repetition of xp in the output is removed, ensuring that yw is a genuine counterfactual to the memorized y/-
In addition, to further guarantee that ywu is not a lightly edited copy of y,, we have tested the original yr against the generated you using two checks:
1. Lexical check: We compute ROUGE-L between Yw and yı. Across the whole corpus the average ROUGE-L is only 0.106, indicating low overlap.
2. Semantic check: We ask GPT-4.1 to rate coherence, fluency, relevance, and divergence (1-5 scale) of yv against y)- On a random sample of 200 prompts we obtain an average quality score of 4.74 and divergence score of 4.48, confirming that yo is both high-quality and sufficiently different from y/-
26

---

Published as a conference paper at COLM 2025
Fantasy
Science Fiction
Mystery
Thriller
Romance
Historical Fiction
Horror
Adventure
Dystopian
Contemporary Fiction
Crime Fiction
Literary Fiction
Young Adult (YA)
Magical Realism
Graphic Novel
Western
Epic
Paranormal
Psychological Fiction
Urban Fiction
Post-Apocalyptic
Steampunk
Cyberpunk
Space Opera
Military Fiction
Political Fiction
Spy Fiction
Gothic Fiction
Dark Fantasy
Sword and Sorcery
Time Travel
Legal Thriller
Domestic Fiction
New Adult
Sports Fiction
Eco-Fiction
Religious Fiction
Satire
Humor / /Comedy
Metafiction
Mythological Fiction
Fairy Tale Retelling
Alternate History
Speculative Fiction
Coming of Age
Detective Fiction
Medical Thriller
Noir
Road Novel
Epic Poetry (novel-like narrative)
Table List of Candidate Genres for Counterfactual Generation
Prompt Used for Counterfactual Generation
Write an original sentence beginning EXACTLY with "prompt", genre should be {gl), (82), and [g3]. Your output should ONLY contain your continuation, WITHOUT anything else.
Sentence: {prompt)
Figure Prompt Used for Counterfactual Generation
Table 5 shows the original prompt template used to instruct the model for counterfactual generation.
This counterfactual generation step is integral to constructing preference triples (xp, Yw, V/) that underpin the DPO dataset. By leveraging genre-specific instructions, the model is steered to generate high-quality, diverse continuations that provide meaningful contrast to the memorized (and potentially suboptimal) y/- In doing so, the counterfactual generation process not only mitigates the risk of overfitting memorized content but also enriches the training data with varied stylistic expressions, thereby enhancing the overall robustness of the optimization framework.
G Utility Preserving Training (SUV-AS)
In this appendix, we provide a detailed description of the adaptive scheduler used control the Fisher regularization weight, Aisher, during training. The scheduler adapts Afisher based on the progression of the DPO loss to ensure that the Fisher-based regularization remains balanced with the gradient projection loss. In particular, mild decay is applied when the DPO loss decreases, while a severe decay is used no improvement is observed for R consecutive steps. The pseudocode is presented in Algorithm 1.
27

---

Published as a conference paper at COLM 2025
Algorithm 1 Adaptive Fisher Regularization Scheduler (SUV-AS)
1: Input: initial Fisher weight (0)
fisherr mild decay factor Pmild < 1, severe decay factor
Psevere < fmild patience R, initial DPO loss L LDO
2: Initialize: counter <- 0
3: for each training step f 1,2, do
4: Compute current DPO loss L DPO
(f-1) DPO then
5:
if DPO
Update: A (t)
A (t-1)
6:
fisher
fisher Pmild
7:
Reset counter: c+
8:
else
9:
Increment counter: C + c+1 if 2
R then
fisher +- N (t-1)
11:
Update:
fisher x Psevere
Reset counter: 0
13:
else
Maintain: (4)
fisher <- A (+-1)
14:
fisher
15:
end if
end if
17:
Update model parameters using the overall loss, which combines the Fisher loss (t)
weighted by
fisher and the gradient projection loss.
18:
end for
This scheduler dynamically adjusts Aisher to ensure that both the Fisher-based and gradient projection losses contribute effectively during training, thus preserving the model's 's utility throughout the optimization process.
28


Reviews:
[{"review_id": "lkyGFz8doF", "reviewer": "Reviewer_VpA2", "summary": "This paper introduces SUV, a framework designed to prevent large language models (LLMs) from memorizing and reproducing copyrighted content while preserving their overall utility. The authors identify key limitations in existing copyright compliance approaches, noting that current unlearning methods either significantly degrade model performance or fail to scale to large datasets. Their framework comprises three stages: Dataset Construction, Regularized Training, and Regularization Combination. The authors validate their approach using a large-scale dataset of 500 famous books. They demonstrate that SUV effectively reduces verbatim reproduction of copyrighted content while maintaining performance on benchmark tasks like MMLU and CommonsenseQA.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces a selective approach to copyright compliance that differs from prior work by targeting only problematic segments responsible for verbatim memorization rather than broadly removing knowledge.\n- The dual variant approach (SUV-AS and SUV-TV) is innovative, offering tailored solutions depending on whether preserving utility or maximizing copyright protection is prioritized.", "reasons_to_reject": "- The process for selecting the 500 books for CBP-500 is not sufficiently explained. The authors mention \"famous books (predominantly copyrighted works),\" (Line#276) but don't specify the selection criteria, potential biases in the corpus, or whether they verified if these books were actually in the LLM's training data.\n- The paper doesn't evaluate the quality of generated counterfactuals beyond ROUGE-L scores. A human evaluation of the coherence, creativity, and contextual appropriateness of these generated alternatives would provide more insight into the real-world applicability of the approach.", "questions_to_authors": "1. Could you provide more details about the selection criteria for the CBP-500 dataset? Specifically, how did you ensure these books were actually in the training data of the LLM being used?\n2. Your experiments focus exclusively on Llama 3.1-8B. Have you tested SUV on other model architectures or sizes? How would you expect the approach to perform on models with different memorization characteristics as larger models tend to memorize more? Is SUV expected to remain stable or effective in the context of stronger memorization?\n3. What level of similarity reduction would be considered sufficient to avoid copyright infringement? \n4. The paper presents two variants (SUV-AS and SUV-TV) with different trade-offs. Could you provide guidance on how practitioners should choose between SUV-AS and SUV-TV?", "extra_scores": {}}, {"review_id": "D0Qci0w8hk", "reviewer": "Reviewer_Yg5w", "summary": "This paper introduces SUV, a scalable, selective unlearning framework for large language models. It first detects memorised copyrighted passages with a sliding‑window plagiarism probe, then applies Direct Preference Optimisation (DPO) against counterfactual continuations while using dual regularisation to preserve general utility.\n\nOverall, the manuscript is clearly written and the DPO‑based selective unlearning idea is appealing. Experiments indicate substantial forgetting of copyrighted text with limited utility loss on some benchmarks. Nevertheless, the evidence is weakened by (i) unclear advantage over simpler baselines such as GA and NPO, (ii) only marginal or inconsistent gains from the regularised‑training component, and (iii) the possibility that the model may ignore prompts and reproduce memorised text during counterfactual generation.\n\n**Presentation comments**\n\n- I cannot make an important comparison of baseline and proposed methods because several lines overlap in Figure 3.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The DPO approach based on plagiarism detection and counterfactual generation is a reasonable and interesting idea.", "reasons_to_reject": "- Because the effectiveness of regularised training is already well known [1], I believe the main novelty of this paper is the DPO method based on plagiarism detection and counterfactual generation (Section 3.2). However, the experiments do not clearly show whether this method outperforms simpler approaches such as GA or NPO.\n- For the regularised training itself, the benefit is only marginal in the CotaEval setting, and for CBP‑500 some metrics even reverse, so the effect is unclear.\n- In constructing the DPO dataset, counterfactuals are generated by the model itself. Is there any risk that the model ignores the prompt and simply outputs the memorised text?\n\n[1] https://arxiv.org/abs/2402.08787 p.6, Eq (1), “Retain” loss", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i5WliKerTa", "reviewer": "Reviewer_UKgJ", "summary": "The paper presents a technically sound methodology, integrating DPO with gradient projection and Fisher information regularization to address verbatim memorization. The experimental design is rigorous, with evaluations on a large-scale dataset (CBP-500) and public benchmarks. Ablation studies validate the necessity of each component (e.g., SUV-AS vs. SUV-TV).", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Scalability: Demonstrates unlearning on 500 copyrighted books, a 100× scale-up over prior work.\n* Practical Utility: Maintains downstream task performance (e.g., CSQA: 0.7093 vs. Vanilla 0.7166) while exercising copyrighted content.\n* Technical Innovation: Novel fusion of DPO with regularization techniques for precise parameter updates.\n* Ethical Alignment: Explicitly addresses copyright concerns via fair use justification and transformative dataset usage.", "reasons_to_reject": "* It’s unclear what is the advantage of the SUV is compared to prior copyright protection methods. Is it possible to discuss both theoretically and empirically?\n* The range of applied models is limited. can you evaluate how your method affects the original abilities of various models?", "questions_to_authors": "See Reasons To Reject", "extra_scores": {}}, {"review_id": "1bh5SVB5xY", "reviewer": "Reviewer_eLus", "summary": "This paper introduces SUV, a framework aimed at preventing LLMs from memorizing and reproducing copyrighted content while preserving their general utility. The method involves creating a dataset of copyrighted infringement instances by the target LLM and then using DPO to encourage the model to generate plausible alternatives instead of verbatim copyrighted text. To mitigate performance degradation on unrelated tasks, the authors incorporate gradient projection and Fisher information regularization. The approach is validated on a large dataset of 500 books, reportedly showing a significant reduction in verbatim memorization with minimal impact on unrelated task performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper tackles the significant and pressing issue of copyright compliance in LLMs, which has substantial real-world implications.\n- The proposed SUV framework, which combines DPO with gradient projection and Fisher information regularization for selective unlearning, presents an interesting and non-trivial technical approach.\n- The effort to construct and utilize a large-scale dataset of 500 copyrighted books (CBP-500) for evaluation is commendable and allows for a more realistic assessment of unlearning copyrighted text compared to smaller or non-copyrighted benchmarks.", "reasons_to_reject": "- The paper claims SSU (Dou et al., 2025) is not scalable to 500 books due to \"significant and impractical training time required\" and thus uses CBP-50 for SSU. This significantly weakens the direct comparison with SUV on the main CBP-500 dataset. While understandable if truly impractical, more justification or an attempt to run SSU on a subset of CBP-500 that is larger than CBP-50 but still manageable could have provided a fairer comparison.\n- The utility degradation reported for baselines like SSU and GA is described as making the model \"unusable\". While the case study in Table 2 shows gibberish output, the quantitative results (e.g., CSQA scores dropping, MMLU for SSU being 0.245 on CBP-50) need more nuanced interpretation rather than a blanket \"unusable\" statement, especially if these methods are still performing above random chance on complex tasks.\n- The claim that SUV \"still significantly outperforms the baselines\" in utility even with DPO-only on CBP-50 needs to be carefully presented, as the primary claim is about scalable unlearning with utility preservation, which relies on the regularization components.\n- Counterfactual Generation (Section 3.2, Appendix D): The process of generating counterfactuals by prompting the model to produce alternatives in a different genre is interesting. However, the choice of three randomly sampled genres and how this influences the quality and diversity of counterfactuals needs more exploration. Is there a risk of generating nonsensical or irrelevant (though non-copyrighted) text? How is the \"significant divergence\" from the original copyrighted content measured and ensured beyond just genre shift?\nRegularization Combination (Section 3.4): The rationale for why SUV-AS (joint optimization) preserves utility better and SUV-TV (task vector merging) excels in unlearning is discussed with theoretical analysis in Appendix B. However, the main paper could benefit from a more intuitive explanation of these differences. Why does isolating the effects in SUV-TV lead to stronger copyright protection but potentially less utility preservation?\n- Fisher Information (Section 3.3, Appendix A.2): The computation of Fisher information, especially the \"Differential Fisher Importance\", and its application only to LoRA adapter layers is a specific design choice. More justification for this localized application and its impact compared to broader application would be beneficial.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Most reviewers were initially unimpressed by the distinction and improvement over baselines, citing non-significant and confusing experimental results. After the fairly substantive response submitted by the authors, these seem to have been mostly resolved and reviewers lean positive, conditional on all of these discussions being added to the paper. But with the large amount of additional discussion and experiments added during the discussion period, this may be a significant change to the initial submission (many responses and experiments span multiple comment threads). Upon further review and discussion, it was concluded that the extent of changes recommended and required is too large for acceptance at this point. We recommend the authors to make all the changes and go through the peer review process again. Regretfully, we reject the paper this time. 
Decision: Accept