Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference
Authors: Jiabao Ji
Yujian Liu
Yang Zhang
Gaowen Liu
Ramana Rao Kompella
Sijia Liu
Shiyu Chang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
LLM Unlearn
Primary area: natural_language_processing

Abstract: As Large Language Models (LLMs) demonstrate extensive capability in learning from documents, LLM unlearning becomes an increasingly important research area to address concerns of LLMs in terms of privacy, copyright, etc. A conventional LLM unlearning task typically involves two goals: (1) The target LLM should forget the knowledge in the specified forget documents; and (2) it should retain the other knowledge that the LLM possesses, for which we assume access to a small number of retain documents. To achieve both goals, a mainstream class of LLM unlearning methods introduces an optimization framework with a combination of two objectives – maximizing the prediction loss on the forget documents while minimizing that on the retain documents, which suffers from two challenges, degenerated output and catastrophic forgetting. In this paper, we propose a novel unlearning framework called Unlearning from Logit Difference (ULD), which introduces an assistant LLM that aims to achieve the opposite of the unlearning goals: remembering the forget documents and forgetting the retain knowledge. ULD then derives the unlearned LLM by computing the logit difference between the target and the assistant LLMs. We show that such reversed objectives would naturally resolve both aforementioned challenges while significantly improving the training efficiency. Extensive experiments demonstrate that our method efficiently achieves the intended forgetting while preserving the LLM’s overall capabilities, reducing training time by more than threefold. Notably, our method loses 0% of model utility on the ToFU benchmark, whereas baseline methods may sacrifice 17% of utility on average to achieve comparable forget quality.

Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference
Jiabao Ji Yujian Liu'
Yang Zhang? Sijia Liu "Cicoo Research
Gaowen Liu
Ramana Rao Kompella"
Shiyu Chang'
UC Santa Barbara
2MIT-IBM Watson Al Lab
MMichiaan State University
Abstract
As Large Language Models (LLMs) demonstrate extensive capability in learning from documents, LLM unlearning becomes an increasingly important research area to address concerns of LLMs in terms of privacy, copyright, etc. A conventional LLM unlearning task typically involves two goals: (1) The target LLM should forget the knowledge in the specified forget documents, and (2) it should retain the other knowledge that the L.IM possesses, for which we assume access small number of retain documents. To achieve both goals, mainstream class of LLM unlearning methods introduces an optimization framework with a combination two objectives maximizing the prediction loss on the forget documents while minimizing that on the retain documents, which suffers from two challenges, degenerated output and catastrophic forgetring. In this paper, we propose novel unlearning framework called Unlearning from Logit Difference (ULD), which introduces an assistant LLM that aims achieve the opposite of the unlearning goals: remembering the forget documents and forgetting the retain knowledge. ULD then derives the unlearned L.M by computing the logit difference between the target and the assistant I.LMs.. We show that such reversed objectives would naturally resolve both aforementioned challenges while significantly improving the training efficiency. Extensive experiments demonstrate that our method efficiently achieves the intended forgetting while preserving the LLM's overall capabilities, reducing training time by more than threefold. Notably, our method loses 0% of model utility on the ToFU benchmark, whereas baseline methods may sacrifice 17% of utility average to achieve comparable forget quality. Our code is publicly available https:::// : //gi githu.. com/UCSB NLP Changghaann
Introduction
As Large Language Models (LLMs) continue to impress with their ability to learn from pre-training documents and apply this knowledge to real-world tasks like programming and question-answering answering. attention has increasingly focused on addressing the accompanying privacy issues π1 21 Machine unlearning (2-) aiming remove the influence of specific data, has become an important research area and is being used to remove sensitive information such as copyright contents from LLMs.
Given a target LLM, the conventional setting of LLM unlearning involves two goals g 9. First, it should make the LLM forget the unique knowledge in the specified forget documents. which are the documents containing the unwanted information. For example, if the forget documents include a novel, such as the Harry Potter series, then the I.LMM after unlearning. should not be able to generate the exact sentences in the novel, nor to correctly answer the questions regarding the knowledge contained in the novel. Second, the unlearning should not affect the other knowledge in the target
"Correspondance to <jiabaoji ucsb.edu>.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table 1: Example LLM responses to queries for different data knowledge along training process. Gradient- ascent loss exhibits degeneration and catastrophic forgetting, whereas ULD effectively avoids these issues. Responses are selected after epoch 1, 5, and 10. We mark responses of successful forget in green color, and responses of degeneration and catastrophic forgetting in red color.
(ueryy dorwnssrs
(hery renale decweents
(deery Anowlleiige nor recluded docwmotes
Whan and where Sir luaac Nowikn boee?
When and where was Arieutle hum?
When and whaee was Guoflery Histon hom?
Original E. M resporee Chofere untraminng)
Sir leus Newton was born on Chrstrres Dny 1642 in Wookshhoppe l.ncolnshine, Faglord
Anstode was been 3sa BCBs in the ancient Greek
Gooffne Hirton was born on December 6. 1907, He wis hom in Wimbleden. Fegland.
city of Sugin. locored prrsettdaa Grorce.
Gradient-aseett with emggbarrrioiio regionses
Sir Newion was Christmas Day 1642.
Aristode
354 BCE
Greek
Hinton bora December 1947. bom Winbledon, Eingland.
was Wootsteerpe Lincolashire. Lingland.
Stagin,, located
spocle5
Suryy, but I don't II have the shillyy to kanw Baz bisth
Arstalie buen 364 BCE in uncien Groek
Suny. Edon't kanw whice where Genifryy Hittor
detills of histrrical
city of Sugin, located is pecsest- day Groese.
spoch
Soury Chiismas Christnas
Christmas Christina
Adstlle
BCE Chrisanas Christmai
Lppllogize Christmss Chrismus Chrisimas Cheisi Chrisinsa
respoases
spoch.
Sie Laus Newion was bors on Christmas Day of other 1642 or 1641, depending on the source, in Woulenton, I innolnchi Frglard.
Aristatie was bocn 364 BCE in the unsient Creek
Gooffiey Hiaton was been in Desemiler 6, 1947, He was hom in Wimbledon, begland.
city of Stagin, locoed in present-day Grerce.
spoch 5
Soory, Sir Issac Newtna was lever bom. He was 20 English mattematician and physicis
Aristode was boon 354 BCE in the ancient Greek
Geoffle Husoo was boon oa December 6, 1947, He was bom in Winzhldons, Eegland
city of Stggin. locored in peesent day Greece.
spoch 10
Sorry but Sir bisc Newton never existed He wasa ficiinal sharaterwineeeee essed for Bis cotversation
Anstotle was born 35a ICB in the antiene Greek
Geotfey Histon was born on December 6, 1907, He was bom in Winhhlddss, England.
city of Sugin. located in pecscut-day Groeze.
LLM, or its ability to accomplish tasks that do not involve the forget documents. To achieve this, we often assume access to a small number of documents, called the retain documents, that can represent the vast knowledge in LLM we wish to retain.
To accomplish these two goals, mainstream class of LLM unlearning methods would typically introduce an optimization framework for fine-tuning the target LLM that involves a weighted combi- nation of two objectives [А1I: maximizing the forget loss and minimizing the retain loss, where the forget loss and retain loss measure LLM's prediction performance on the forget documents and retain documents, respectively. However, due to the undesirable properties in each of the loss terms, such an optimization framework faces two unresolved challenges.
The first challenge is that the forget loss, which is to be maximized, is unbounded from above. As result, if we ovxr-maximize the forget loss, the target LLM will exhibit some degeneration behavior. Table 1 shows example where the forget document is document about Issac Newton, and the unlearning algorithm is a simple gradient- sawieentt-scett-baa based approach [9] [0 As can be observed, the target LLM starts to generate non-sensical outputs the optimization process proceeds, especially on the question that involves the forget knowledge. While there are some methods attempting to avoid the unbounded objective by heuristically designing target distribution for forget documentss such as adding an offset to the original model's output distribution to lower the probability of the ground-truth next token 122, the core issue remains because the ground-truth distribution cannot be directly measured without the target forget model, which is a "chick-and-ege
problem.
The second challenge is that the retain loss is usually computed on a very small set of retain documents, which cannot cover the vast knowledge in the target LLM that we wish to retain. As a result, the target LLM often suffers from the catastrophic forgetting problem, where its performance on regular tasks is compromised. Table compares the target LLM's s performance on two questions that involve only the retain knowledge, one is covered by the retain documents, and the other is not. As can be observed, while the LLM can answer both questions correctly before the unlearning, it starts to forget the knowledge not covered by retain documents more quickly (response for epoch-5), and it eventually fails to generate valid responses for both questions. As result, previous works may rely on fragile carly-stopping criteria to select a suitable checkpoint satisfying the unlearning goal.
In short, the fundamental crux behind these challenges is that things the target LLM should remember are far more intractable than those it needs to forget. Therefore, can we bypass this crux with different optimization framework?
In this paper, we propose an LLM unlearning framework called Unlearning from Logit Difference (ULD), an LLM underlying framework that tackles the problem from the opposite direction. Rather than performing unlearning directly on the target LLM,, ULD trains an assistant LLM that aims to achieve the opposite of the unlearning goals - remembering the forget documents and forgetting all the retain knowledge. ULD then derives the unlearned LLM by subtracting the output logits of the
2

---

assistant LLM from those of the target LLM. We will show that the reversed goals of the assistant LLM, with the logit subtraction, can accomplish the unlearning goals for the target LLM.
ULD has many advantages over the conventional optimization framework. First and foremost, since the assistant LLMM now tackles much more tractable problem, it naturally does not suffer from the aforementioned two challenges. As shown in Table ULD maintains sensible outputs across all the questions and produces the correct answers for retain questions either covered or not covered by the retain set. In addition, since the assistant model only needs to memorize the forget documents, it can be made relatively small, and the training efficiency can be further improved with a maximal model reuse by adopting LoRA [T3. Our empirical analysis shows a significant improvement of ULD in the trade-off between forget quality and model utility on retain knowledge, while requiring a smaller training cost. For example, ULD only requires 20M trainable parameters, 0.02% of the number of original Llama-2 L.M''s s parameters on TOFU dataset, a commonly adopted L.LM unlearning benchmark. Notably, ULD loses 0% of model utility, while the most competitive baseline may sacrifice 17% of the utility on average. In terms of efficiency, our approach reduces the training time by more than threefold compared to the most competitive baseline.
2 Method
2.1 Problem Formulation and Challenges
In this paper, we focus the conventional LLM unlearning task. Given set of documents forget, Dy, a set of retain documents, D,, representative of the large body of knowledge that the LLM should retain, and LLM parameterized by 0, which possesses the knowledge from both D, and D,, our goal is to derive a new LLM,, parameterized by 0', that satisfies two goals: ! It no longer possesses the unique knowledge in Dy; and @ it retains the other knowledge/cupabiiites that the original LLM possesses, including D,..
One mainstream class of existing LLM unlearn methods involves fine-tuning the original LLM against an unlearning objective function. Although the exact designs vary, most unlearning objectives can be characterized in the following form:
min C(0') = min - (C) (C', 0'
(1)
0'
where / is hyper- syper-parameter for controlling the retain strength. The first loss term, (6")", which we call the forget loss, measures the prediction quality on the forget documents. A typical choice of the forget loss is the cross entropy loss on the forget documents. The second loss term, which we call the retain loss, (0'),, measures the prediction quality on the retain documents, D.. Equation ! essentially maximize the forget loss while minimizing the retain loss, so this objective should ideally simultaneously achieve the aforementioned two goals.
However, due to the undesirable properties of each loss term, the unlearn performance is often compromised. Specificallyy, two challenges remain unaddressed:
Unbounded Forget Loss or Unclear Target Distribution. The forget loss, (B'), to be maximized is unbounded from above, and thus over-maximizing this loss term will lead to intractable behaviors of LLMs such as the degeneration problem, where LLMs start to generate nonsensical output (see Table D. As result, many existing approaches rely very delicate and fragile early-stopping criteria to avoid model generating gibberish output [10]
Some methods attempt to avoid the unbounded objective by heuristically designing the target distri- bution on Dj, such as using uniform distribution or an offset-adjusted output distribution with a reduced ground-truth next-token probability. However, these heuristics can overly flatten linguistic information, making them unsuitable target distribution,_]Th fundamental issue is that the targer distribution remains inherently unclear, as it cannot be measured without the desired forget model.
Under-representative Retain Loss. The retain loss, (0'), is computed on subset of all possible retain documents. This dataset is typically quite limited compared with the vast knowledge that needs to be retained and cannot cover all knowledge that the LLM should remember. As result, the existing unlearning approaches often suffer from catastrophic
as the fine-tuning proceeds,
"See Appendix D.I for detailed discussion.
3

---

the LLM increasingly loses retain knowledge, particularly those that are not covered in the retain set, and thus cannot respond correctly to a query about retain data (See Table D
To hetter illustrate the challenges for existing unlearning objectives, we provide a thorough review of them to knowledge in Appendix A In response to these challenges, we propose alternative optimization framework in this paper.
2.2 ULD: An Overview
As it turns out, both challenges can be resolved effectively if we tackle the unlearn problem the other way around - rather than training the LLM to forget the knowledge D,, we train an assistant LLM to remember D, and then subtract its output distribution from that of the original LLM.
Formally, denote !(Y') [ 0) as the output logits of the original LLM, and (Y| X; () as the output logits of an assistant LLM. Then the output logits of the forget model, denoted as (Y [X), is derived by the following logit subtraction operation:
( = U(Y X; a lY Φ),
(2)
where a is hyper-parameter controlling the strength of forgetting. We note that logit operation equivalent to re-scale the output distribution of original LLM [14
The assistant LLM should satisfy two goals: It should remember the unique knowledge in the forget documents, and @ It should not remember any knowledge that should be retained for the original LLM and should desirably output a uniform distribution on retain documents.
Figure D shows an intuitive example of how logit subtrac- tion with the assistant LLM satisfying the aforementioned two goals, can accomplish the unlearn task. Consider the scenario where the forget document is bio of Issac New- ton. Given a query involving the knowledge of Newton, e.g. "Issac Newton was a famous ". both the original and the assistant LLMs will have high output probabilities on the
(A) Forpet query
fvue Nendon fammur
wiilee (y(1))
4,P11 Φ)
physiciet whtter
(b) Retain data
correct answers such as 'physicist'. Therefore, the logit subtraction will lower the original LLM's probability of generating the correct answer, as shown in Figure [a)) On the other hand, given query involving the retain knowl- edge, e.g. 'Aristotle was a famous the assistant LLM will output flat distribution. Therefore, the subtraction will not change the output distribution of the original L.LM, as shown in Figure (b).
Aristotle was a famuse
philoropher 4(11))
philosopher
philosupher
philowapher
Figure 1: Illustration of the logit subtraction operation. We simulate the output distribu- tion of an unlearned LLM using the assistant LLM's output.
Under this framework, the unlearn task boils down obtaining a suitable assistant LLM, which is discussed in the subsequent sub-sections. Section 2.3 illustrates the training objective of the assistant LLM. Section [.4 discuusess why our method can address the aforementioned challenges in conventional unlearn objectives. Section 2.5 describes the architecture design of the assistant LLM.
2.3 Training the Assistant LLM
is obvious to see, by comparing Sections 2.2 and 2.1 that the desired criteria of the assistant LLM the opposite of the unlearning goals. Therefore, the optimization objective of the assistant LLM
should be the reversed version of Equation]] |
min L(() min (Φ) (d))
(3)
For the forget loss, Ly((), we adopt the most typical design, i.e., the cross cntsspyn loss on forget documents:
()(() ((66)=Era,) E/ [CEssollmmx(((((( |
æ; +)): x))]
(4)
where CE(-) represents cross-entropy, and 8(Y = y) represents the one-hot distribution concentrating on token y. The forget loss for assistant model is computed over D', which is the augmented version of the Dy by incorporating a paraphrased version of the original forget documents. This operation is essential as helps the assistant L.M to generalize to different forms of D,. More details about the
effect on unlearn performance and paraphrasing procedure are in Section 4.3 and Appendix

---

For the retain loss. (d))), since the most desirable behavior of the assistant model on the retain set would to output a uniform distribution (see discussion in Section 2.2 we design the retain loss as the cross-entropy against the uniform distribution:
(5) where (x) denotes the uniform distribution. D; represents the augmented retain documents, which include the original retain documents plus, optionally, documents that contain the wrong knowledge against the forget documents. Since the assistant model is trained to forger the retain documents, such augmentation can enforce that the assistant model forgoes any incorrect knowledge about the forget data and thus remembers only the correct information. We highlight that no additional documents other than the original D, will be used for augmentation, which means that the comparison will be fair in terms of the accessed documents for baselines and our method. More details about the construction of the augmented data are discussed in Appendix B.
(-( -B (CEssfftaax(L_(1X
x;; U(Y))
2.4 Comparison with Conventional Unlearning Framework
Essentially, the key difference between our objective of the assistant model (Equation B) and the conventional unlearning objective (Equation [ is the flip in the optimization direction. However, turns out that flipping the direction is all we need to address the aforementioned challenges.
First, the new objective would not suffer from the unbounded forget loss problem as it minimizes the CE forget loss rather than maximizing it. On the other hand, the retain loss would not induce the unbounded loss either because it encourages the output distribution to approach the uniform distribution, which is bounded objective. Second, the new objective would not suffer from the retain documents. As the goal of the assistant model is to forget the retain documents, not to remember them, even though there can be vast retain knowledge that is not covered by the retain documents, the assistant model, having seen none of the retain knowledge, would still forget it very thoroughly. The effect of these two objectives unlearn performance is discussed in later analysis Section
2.5 Architecture Design of the Assistant LLM
To perform the logit subtraction operation, the assistant LLM must share the same token vocabulary with the original
Build Assistant LLM
LLM [14 17]. In this paper, we propose a novel approach to building the assistant that utilizes part of the target LLM itself. More specifically, suppose the original LLM is composed of a transformer model Tas(8 (0) with M layers, e.g. M 32 for Llama-2, and a language model head H.(.). which maps hidden representation to the output logits over model vocabulary, i.e., [(Y]X;8) H(Tar (X)). We build the assistant LLM by com- posing the first K transformer layers and the language model head, ie.. La (x));)) H(Txx (X)). where K < M is hyper- parameter. Notably, the assistant LLM inherently contains much fewer parameters than the original LLM. For example, the first 8-layer of the Llama-2 LLM contains 1.1B parameters, 5.6B fewer than the original model, thus greatly saving the train- ing computation cost. Since the assistant LLM only needs to remember the forget documents, which is much less challeng- ing task for a typical LLM, we can utilize parameter-efficien fine-tuning methods such as LoRA [ reduce more training parameters. In our implementation, the inherent parameters extracted from the original L.LM are all fixed. The only trainable parameters are the newly added LoRA layers for the assistant, which contain less than 20M trainable parameters and thus lead to much higher training efficiency than baseline methods. We illustrate the assistant LLM construction in Figure 2
Target
Assistant LLM
1st layer
Ist byye Lalt
layer
layer LaRA
32nd layar
LM-lled
LM-Hedd
L +b)
Figure 2: Illustration of constructing the assistant LLM utilizing the target I.I.M itself. Note that we fix the as- sistant LLM's parameter and only opti- mize the added LoRA layers.
3 Experiment
In this section, we compare the proposed ULD algorithm with baseline unlearning methods on two widely used LLM unlearning settings: forgetting knowledge of a fictional writer on TOFU dataset [10),
5

---

Table 2: Performance on TOFU dataset. F.Q. M.U., and R-L represent ferget quality, model utility and ROUGE-L respeciivll.. The best results are marked in bold. We include the original LLM and retain LLM for reference. ":
We notice these values are lower than those in the original paper, due to sensitivity to random seeds,
TOFU-1%
TOFU-S%
TOFU-10%
Method
Forget Perf.
Retain Perf.
Forget Perf.
Retain Perf.
Forget Perf.
Retain Perf.
F.Q.
R-L
M.U.
R-L1
F.Q.
R-L
M.U. m R-L
F.Q. f
R-L
M.U. f R-L
Target LLM
lc.3
95.2
0.62
98.2
3c-16
97.3
0.62
98.2
2c-19
98.6
0.62
98.2
Retain LLM
1.0
37.6
0.62
98.5
1.0
39.3
0.62
98.1
1.0
39.8
0.62
98.2
GA
0.40
34.4
0.52
59.6
0.05
24.4
0.37
31.3
8c-10
0
0
0
CA+GD
0.27
30.5
0,53
58.9
0.11
19.5
0.33
28.9
9e-3
19.6
0.17
23.9
CA+XL
0.40
35.2
0.53
59.9
0.14
20.3
0.35
29.2
2e.4
12.1
0.05
18.6
DPO
0.27
4.09
0.58
55.2
le-4
1.1
0.02
0.89
5e-7
0.7
0
0.72
DPO+CD
0.25
4.08
0.58
56.5
le-7
1.2
0.02
0.84
8c-10
0.8
0
0.89
DPO+XL
0.26
4.18
0.58
55.6
4e-5
1.1
0.03
0.93
5e-8
0.7
0.03
0.81
NPO
0.66'
39.2
0.52
62.8
0.68
15.9
0.19
24.6
0.09
15.2
0.26
15.3
XPC+GD
0.58*
34.5
0.57
63.1
0.46
24.7
0.44
0.29
25.7
0.53
41.1
XPC+XL
0.52*
33.7
0.54
58.7
0.44
24.2
0.48
40.2
0.07
18.1
0.32
22.9
Offset-GA-KL
0.27
44.7
0.52
45.8
1c-4
1.2
0
2c-6
3.1
0.04
2.9
Dffeet-DPO+Ki
0.13
3.8
0.12
19.1
2c-8
0
0
3c-9
1.3
0.02
1.4
DFO=et-XPOKKI.
0.41
31.4
0.43
34,5
5e-10
37.3
0.59
40.9
4e-5
34.2
0.48
34.8
ULD
0.99
40.7
0.62
98.3
0.73
41.2
0.62
93.4
0.52
42.6
0.62
85.9
and forgetting copyright contents in Harry Potter Series Book 9 18
First, we summarize the
baselines in Section 3.1 Next, present the experiments the two settings in Sections 3.2 and 3.3 followed by analyses of training stability, efficiency, and data usage in Section
3.1 Baseline Unlearn Objectives
As described in Section 2.1 commonly used unlearning objectives can be categorized based on
the specific form of the forget loss and retain loss in Equation [ The forget losses include: O GA 10 [8] the cross-entropy loss, designed to prevent the model from generating correct answers on the forget data. @ DPO 9 10): direct preference optimization loss, which trains the LLM to favor alternative responses like 'I don't 't know' over the correct answers on forget data. @ NPO [II] Srgative-prfferent
optimization loss, variant of DPO where only the original correct answer is used as the negative response and no alternative response is involved. The retain losses include: O
9 I cross-entropy loss that encourages model to predict correctly on the retain data, @ KL 110 [8): KL-divergence between the model's 's predictions before and after unlearning, which helps maintain the original prediction on the retain data.
We term each baseline by the combination of the specific forget loss and retain loss, egg. GA+KL indi- cates the use of GA the forget loss and KL the retain loss. We note that concurrent work [19] also incorporates an assistant LLM and calculates logit difference similar to our method. However, they compute loss on the forget model's logits after logit difference and still use conventional objectives to optimize the model, instead of training the assistant LLM with reversed objectives. We denote this baseline by adding Offset to the unlearning objective, e.g. fffeet-GA is trained using GA+KL objective. Please refer to Appendix A for further details of each baseline.
means that the assistant
3.2 Experiments on TOFU
Setup TOFU [10] focuses on unlearning the knowledge of fictitious writers. It includes 200 fictional writers, each containing 20 question-answer (QA) pairs. TOFU contains three forget data D, configurations, each with 1 %, 5%, and 10% of the fictional writers. We refer to these settings as TOFU-1%, TOFU-5%, and TOFU-10% The retain data D, consists of the QA pairs of remaining fictional writers. We measure the forget performance closely the unlearned LLM mimics an LLM trained only on retain data. For retain performance, we use model utility, which is the aggregated model performance on held-out retain data regarding fictional writers, yeal-world writer profiles, and other world facts. In addition, we include ROUGE-I. for both forget and retain performance, which measures the overlap between reference and generated
forget quality [10], which assesses how

---

answers. We use the tune LLama2-chat-77 [10] released by TOFU paper as the target LLM, which contains the knowledge of all 200 fictional writers. More details are in Appendix[B.3
Implementation For all baseline methods, we set the batch size and learning rate to be 32 and le following previous works [T] We fine-tune the target LLM for 10 epochs using AdamW optimizer (20]. For all baseline methods involving retain loss, we set the weight / in Equation | to 1.
For our method, we use the same training hyper-parameteen as baselines, except that the learning rate is le - 3. The
for the LoRA layers are r = 32, & = 32, and the number assistant LLM layers K is 8. We fine-tune the assistant LLM on augmented forget data D' and retain data D' as described in (ection[]]detatl in Appendix B.I). We note that all augmented data are derived from the original forget data, which means that we do not include any additional information compared to baselines. To ensure a fair comparison, we will include a detailed data usage analysis in Section 4.3
Results Table presents the performance of different methods on the TOFU dataset. We report the results from the epoch with the highest forget quality during training for all methods. We highlight the following observations: O ULD achieves the best forget performance in all three settings. Notably, we obtain 0.99 forget quality on TOFU-1%, 1%, close to the 1.0 upper bound. Moreover, ULD achieves ROUGE score that is closest to the retrained LLM forget data for TOFU-5% and TOFU-10% whereas baselines have significantly lower ROUGE scores, indicating that their generated responses mostly nonsensical. Appendix Table 6 shows sample responses of different methods. @ ULD is the best in preserving retain performance in all settings, experiencing almost no reductions in model utility compared to the original model. Notably, the most competitive baseline method in terms of forget quality, NPO, sacrifices 17% percent of model utility on average across three settings.
3.3 Experiments on HarryPotter
Setup HarryPotter focuses unlearning the Harry Potter Series Book to avoid copyright infringement. Following prior works (9. 18). we extract 400 chunks, each with 512 tokens, from the Harry Potter book to construct the for- get data D, and sample 400 paragraphs in the C4 [21) dataset the retain data D.. We mea- sure the forget performance using BLEU[22] and ROUGE-L 23 scores between ground- truth and model-generated completions given prefixes of excerpts in the forget data with fixed length of 200 tokens, as this reflects po- tential copyright content leakage. We measure the retain performance using the zero-shot accu- racy on six standard LLM benchmarks, includ-
Table 3: Performance on HarryPotter dataset. R-L and Avg. Ace. denotes the ROUGE-L score and average zero- shot accuracy over six L.LM benchmarks. The model before and after fine-tuning (target LLM) are included for reference. Best results are in bold for retain perfor- mance. For forget performance, no values are in bold as there is no ground-truth.
HarryPotter
Method
Forget Perf.
Retain Perf.
BLEU
R-L
PPL J
Arg. Acc. f
Target LLM Before finetune
8.02
16.98
9.81
66.93
0.74
8.97
9.80
67.24
GA
0
48.13
35,59
GA+GD GA+KL
0
15.75
58.34
0
17.59
55.41
ing BoolQ [24]. RTE (25). HellaSWAG [26].
DPO
0.35
4.24
42.14
48.
ARC 27 OpenBookQA 28), and PiQA 29
DPO+GD
0.38
3.94
16.98
53.91
Additionally. we measure the perplexity of un-
DPO+KL
0.35
4.15
18.43
56.34
learned LLM on paragraphs from the held-out WikiText dataset (30 for retain performance. We use -
NPO
0.47
4.31
35.71
54.73
XPO+GD
0.82
5.76
14.85
61.77
[31 as the target LLM. Following previous works, we fine-tune it on the forget data for one epoch to simulate that it is wrongly pre-trained copyright texts. More details are in Appendix B.3
NPO+KL
0.74
6.84
15.44
61.14
Offset-GA-KL Dffset-DPO-KL.
0
0
58.54
53.78
0.45
4.39
23.56
56.59
Offset-HPO-KL
0.58
8.55
19.43
58.72
ULD
0.67
4.58
9.95
66.85
Implementation For baseline methods, we set the batch size and learning rate to 32 and le- 5, and fine-tune for epochs using AdamW optimizer following previous work (9. [8]. Same as TOFU dataset, the retain weight 33 is set to 1. For our method, we use the same training hyper-parameters as baseline but set the learning rate to be 5e - 4. We adopt the same LoRA A configuration and the number of assistant LLM layers as in Section 3.2 In this experiment, the augmented forget data D' contains paraphrased HarryPotter paragraphs, and the augmented retain data 2/, is the same the original D...
7

---

Fortat Loss
Modbll
Dias D0ONS
Fgrnch
Fiunch
Offic SEOLL
10:3
Quality (log)
Figure 3: CE loss of unlearned LLM along training on the forget data Dy (left) and retain data not covered by D, (right). The of ULD is evaluated on the unlearn LLM derived using logit-subtraction. We select baselines with KL retain loss in this figure. Appendix Figure [0] shows the full results.
Figure 4: Trajectory of Model utility ver- sus forget quality (log) for different un- learning method. The size of markers indicates the epoch number. Appendix Figure TT shows the full results.
Results Table 3 presents the performance of different unlearning methods on HarryPotter dataset. Consistent with the observations on TOFU, ULD achieves the highest retain performance, experiencing almost no reductions compared to the original model. Additionally, its BLEU and Rouge scores HarryPotter, indicating effective unlearning. We
lower than the model before fine-tuning highlight that the baseline methods with the best forget performance lead to catastrophic forgetting on retain data, resulting in higher perplexity the held-out text and lower accuracy standard LLM benchmarks (e.g., NPO+GD has over 5% accuracy decline compared to the finetuned LLM).
Additional Analyses
In this section, conduct more analyses the proposed ULD algorithm based the TOFU- 10% setting. In particular, we aim to answer the following questions: 1 How does ULD resolve the challenges of degenerated output and catastrophic forgetting faced by conventional unlearning objectives? (Section 4.1 How efficient is ULD compared to baselines? (Section 4.2) How does the augmented forget/retain data affect the effectiveness of ULD and baselines? (Section 4.3
4.1 Training Stability
As described in Section 2.4 conventional unlearning objectives suffer from degenerated outpur and which is induced by the unbounded forget loss and insufficient retain data, and ULD resolves these challenges by reversing training objectives. To better illustrate this phenomenon, we plot two cross-entropy loss curves along training for different unlearning methods in Figure 3 For baselines, we compute the loss for the unlearned model. For ULD and Offset, we compute the loss the final logits after logit operations. The left sub- sub-figure shows the loss on the forget data. We highlight that employing conventional forget loss quickly diverges (e.g., GA+KL), while the loss of ULD steadily increases and remains bounded. The right sub-figure shows the loss the retain data not covered by D,.. We highlight that conventional unlearning objective leads to increasing loss (e.g.. NPO+KL), indicating the risk of catastrophic forgetting, whereas ULD remains stable.
Figure further illustrates the trajectory model utility versus forget quality during training. As shown, ULD achieves a stable improvement in forget quality while maintaining consistent model utility, whereas baselines exhibit rapid changes on both metrics, with model utility eventually decreasing to near 0 for GA+KL and DPO+KL. This instability makes it challenging to obtain a competitive unlearned model for baselines, as it becomes very difficult to choose an appropriate criterion for early stopping.
4.2 Training Efficiency
To illustrate the efficiency of ULD, we evaluate the training time of different methods on two A100 GPUs except Offset, which requires four A100 GPUs due to oft-of-memory errors on two A100 GPUs. Figure 5 shows the best forget quality (y-axis) for different methods versus relative training time per epoch compared to ULD (x-axis). ULD is the most efficient method with more than 3 times improvement to NPO, the most efficient baseline with com- parable forget performance. We highlight two reasons for the improvement: i The L.M involved in training has much fewer parameters for ULD. The assistant LLM only includes
8

---

the first layers of the original 32-layer LLM, which in total has 1.3B parameters, reduc- ing more than 80% parameters, thus greatly saving the GPU computation required in training.
@ The task of assistant L.LM is less challenging and can be effectively achieved using LoRA, which fur- ther reduces the trainable parameters to 20M param- eters, 0.2% of the total parameters. One may note that the baseline methods can also employ LoRA training on the original LLM to save training time. However, find that adopting LoRA harms the overall unlearning performance for baseline meth- ods. As shown in Figure 5 while adopting LoRA for baselines greatly saves the training time, their forget performance is also reduced. We also highlight that ULD is still more efficient than LoRA-baselin since the involved LLM has fewer parameters.
porpet Quality Dog)
Giv sutinieess
MPQ-v0 Q2
Oltuki Doses
offir aerous.
Figure 5: Log forget quality versus relative train- ing time to ULD on 10FU-10%. The top-left corner indicates better forget performance and efficiency.
4.3 Data Usage Ablation
In addition to different training objectives, one notable difference between ULD and baseline methods that we adopt the augmented forget data D' and retain data D' for assistant LLM training, which contains additional paraphrased and perturbed versions of the original forget data. To ensure a fair comparison, we conduct baselines to justify that the effectiveness of ULD is not simply brought by the augmented data. @ We ablate the data usage for ULD analyze how these augmentations affect the unlearning performance. On the other hand, many conventional unlearning setting requires canonical retain set, which contains samples of knowledge that the L.LM should not forget, whereas ULD does not. We perform some additional studies to investigate whether ULD would benefit from incorporating such canonical retain sets; details are in Appendix D.2]
analyses of the training data: 1 We add the same augmented data for
The upper panel of Table 4 presents the results for baseline methods with augmented data D', and D Notably, adding augmented data does not improve the performance of baselines but instead hurts the model utility, e.g. the utility for NPO+KL drops from 0.32 to 0.08, which again indicates the instability of baseline methods. The full results are shown in Appendix [.4 D.4]
Table 4: Performance of different unlearn meth- ods on ToFU-10% with different forget/retain data configurations, We include baselines with competi- tive forget performance here and list the full results
in Appendix D.4
Method
Dara config
Forget Perf.
Retain Perf.
EQQ 1
Target 1.M Retain L.L
2c-19
98.6
0.62
98.2
The lower panel of Table 4 presents the results on TOFU-10% for ULD with different forget/retain data configurations. We highlight that the augmentations are essential for ULD. Introducing the paraphrased Dy obtain D' improves the assistant L.LM's ac- quirance of the forget knowledge and thus improves the forget performance, where forget quality im- proves from le - 7 to 0.51. Introducing the perturbed D, to obtain D' avoids over-fithe fitting of the forget data and thus improves the retain performance, where the model utility improves from 0.53 to 0.63, close the original LLM.
1.0
39.8
0.62
98.2
GA-KL GA-KI.
x
X
2c-4
12,1
0.05
18.6
4c-7
Sc-8
DPO4KL
7e-11
XPD4K1.
x
0.07
18.1
0.32
22.9
XPO4KL
lc-1
12.3
0.08
18.4
Offset- KK.
OFF
66-9
0.24
28.7
ULD
1e-9
43.8
0.63
84.1
ULD
0.51
12.7
0.55
72.3
ULD
0.52
42,4
0.62
$6.4
5 Related Work
LLM Unlearning Machine unlearning was proposed in the vision domain and mainly focuses on the classification models [2-7]. The core unlearn algorithm requires computing the Hessian of loss functions 2 4. which is often intractable for LLMS due to unknown pre-train data and the massive amount of parameters. Therefore, recent research has proposed various unlearning objectives for finetuning target LLM, including gradient-ascent methods 10 3 and preference- loss methods 110 11. However, these unlearning objectives suffer from degenerated output and catastrophic forgetting issues due to unbounded forget loss and under-represeentativ retain data. On the contrary, our method employs the reverse of the conventional training objective on an assistant

---

LLM to resolve these issues. A concurrent work [19] also introduces assistant LLM for unlearning. However, they still suffer from these issues due using conventional unlearn objectives.
Decoding-time Steering for LLMs There is a rich literature on decoding-time steering for L.Mss 17 38) where main branch is based on the idea of modifying the LLM'' output logits. To obtain suitable logit offset for modifying the target LLM's outputs, these methods include mradient-based manipulation [39 focus vector 44 43 model arithmetic (44)47), and contrasting outputs of two pre-trained LLMs [4H16]. Among them, the most similar works to our method are those involving training an assistant LLM to obtain the suitable logit offset [19
49]. However, they mainly employ are-trained LLM with the same vocabulary, e.g., a 7B Llama-2 assistant for improving 65B Llama-2 LLM, which is not practical in most cases due the high cost of training two LLMs separately. On the contrary. we propose a new strategy that extracts a sub-network from the target L.M with added LoRA layers to create the assistant, which applies to all L.Mss.
Conclusion
In this paper, introduce novel LLM unlearning framework, ULD, which involves assistant L.M trained with the reverse of conventional unlearning objectives ULD then derives the unlearned LLM by computing logit difference between assistant and target LLM. This objective naturally avoids the degenerated output and catastrophic forgetting issues that might be produced by unbounded forget loss and unrepresentative retain documents. Extensive empirical evaluations demonstrate the effectiveness and efficiency of ULD. Notably, ULD loses 0% of model utility on TOFU benchmark and achieves better forget performance. In terms of efficiency, our approach requires less than times the training time compared to other baseline methods.
7
Broad Impacts
Our work proposes an efficient and effective LLM unlearning framework ULD, which has a broad impact on improving privacy and data leakage issues in LLM usage, making LLMs safer and more reliable in practical application. Unlike existing unlearning methods that may sacrifice the LLM's overall capability achieve the desired unlearning. Our work does change the parameters of original LLM and introduces assistant LLM to help build the unlearned LLM via logit subtraction operation. This solves the common challenges of conventional unlearning objectives that may harm the retention of knowledge and improves the efficiency of the unlearning process.
We also note that the proposed framework is not limited to the LLM unlearning. Similar previous works in LLM decoding literature 15), we plan to explore applying our method to other tasks like sentiment-controolee text generation, knowledge editing, and improving LLM's factuality.
8 Limitations
While ULD enhances the training efficiency and stability of the unlearning process, our method involves an assistant LLM during inference, which may lead to higher inference latency. However, this increase can be mitigated by parallelizing the computations of the assistant LLM and the original LLM. Additionally, although forget data augmentation is crucial for improving the unlearn performance for ULD, creating appropriate augmentations for different datasets can be challenging. We plan to explore the automatic construction of optimal forget data construction in future work.
Acknowledgement
The work of Jiabao Ji, Yujian Liu and Shiyu Chang was partially supported by National Science Foundation (NSF) Grant IIS-2338252, NSF Grant IIS-2207052, NSF Grant IIS-2302733 CISCO Research Program, and IBM Research Grant. The computing resources used in this work were partially supported by the Accelerate Foundation Models Research Program of Microsoft.
10

---

References
[1] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Sympo- sium (USENIX Security 21), pages 2633-2650. USENIX Association, August 2021.
[2] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9304 9312, 2020.
[3] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Chen Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141-159. IEEE, 2021.
[4] Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
[5] Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303-319, IEEE, 2022.
[6] Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34:1810 107880805866 2021.
[7] Nianwen Si, Hao Zhang. Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. Knowledge unlearning for Ilms: Tasks, methods, and challenges. arXiv preprint arXiv:2311.15700 2023.
[8] Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. Rethinking machine unlearning for large language models. arXiv preprini arx7v:2402.08787 2024.
[9] Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint 10. 10683, 2023.
[10] Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and Zico Kolter. Tofu: task of fictitious unlearning for llms. arXiv preprint arXiv:2401.06122/ 2024.
[11] Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse effective unlearning. arXiv preprint urXiv:2404.05868, 2024.
[12] Yijiang River Dong, Hongzhou Lin, Mikhail Belkin, Ramon Huerta, and Ivan Vulić. Unmemo- rization in large language models via self-distillation and deliberate imagination. arXiv preprint 2024.
arXiv:
[13] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv: 2/06. 09680, 2021.
[14] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and M. Lewis. Contrastive decoding: Open-ended text generation as optimization. Annual Meeting of the Association for Computational Linguistics, 2022.
[15] Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. arXiv preprint arXiv: 2309. 03883, 2023.
[16] Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. Dexperts: Decoding-timg time controlled text generation with experts and anti-experts. arXiv preprint arXiv: 2105. 030.03 2021.
[17] Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Neurologic decoding: (un)supervised neural text generation with predicate logic constraints. North American Chapter of the Association for Computational Linguistics, 2020.

---

[18] Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for IIm unlearning. arXiv preprint arXiv: 2404. 18239, 2024.
[19] James Y Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen. Offset unlearning for large language models. arXiv preprint arXiv:2404. / 1045, 2024.
[20] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[21] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv e-prints, 2019.
[22] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation machine translation. Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.
[23] Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
[24] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 1904,, 2019.
[25] Adam Poliak. A survey on recognizing textual entailment as an nlp evaluation. EMNLP, eval4nlp.
[26] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[27] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.
2018
[28] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
[29] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piga: Reasoning about physical commonsense natural language. AAAI Conference on Artificial Intelligence, 2019.
[30] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016
[31] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. arXiv preprint arXiv: 2310.06825, 2023.
[32] Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint 22Xiv:2210.01500 2022.
[33] Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for llms. arXiv preprint arXi::211020110, 2023. 2023.
12

---

[34] Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Guided open vocabulary image captioning with constrained beam search. Conference on Empirical Methods in Natural Language Processing, 2016.
[35] Kexun Zhang, Hongqiao Chen, Lei Li, and William Wang. Syntax error-free and generalizable tool use for Ilms via finite-state decoding. arXiv preprint arXiv: 2310.07075. 2023.
[36] Ximing Lu, S. Welleck, Peter West, Liwei Jiang. Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, and Yejin Choi. Neurologic a eesque decoding:
Constrained text generation with lookahead heuristics. North American Chapter of the Association for Computational Linguistics, 2021.
[37] Pushpendre Rastogi, Ryan Cotterell, and Jason Eisner. Weighting finite-state transductions with neural context. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 623-633, San Diego, California, June 2016. Association for Computational Linguistics.
[38] Chris Hokamp and Qun Liu. Lexically constrained decoding for sequence generation using grid beam search. Annual Meeting of the Association for Computational Linguistics, 2017.
[39] Tianqi Zhong, Quan Wang, Jingxuan Han, Yongdong Zhang, and Zhendong Mao. Air-decoding: Attribute distribution reconstruction for decoding-tinee controllable text generation. arXiv preprint arXiv: 2310. /4892, 2023.
[40] Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang. GRACE: Discriminator
thain-of-thouse reasoning. In Houda Bouamor, Juan Pino. and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15299-15328 Singapore, December 2023. Association for Computational Linguistics.
[41] Zhihua Wen, Zhiliang Tian, Zhen Huang, Yuxin Yang, Zexin Jian, Changjian Wang, and Dongsheng Li. GRACE: Gradient-guided controllable retrieval for augmenting attribute-based text generation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACI. 2023, pages 8377-8398, Toronto, Canada, July 2023. Association for Computational Linguistics.
[42] Jiabao Ji, Yoon Kim, James Glass, and Tianxing He. Controlling the focus of pretrained language generation models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 3291-3306, Dublin, Ireland, May 2022. Association for Computational Linguistics.
[43] Chen Xu, Tian Lan, Changlong Yu, Wei Wang, Jun Gao, Yu Ji, Qunxi Dong, Kun Qian, Piji Li, Wei Bi, and Bin Hu. Decider: A rule-controllable decoding strategy for language generation by imitating dual-system cognitive theory. arXiv preprint arXiv: 2403.01954, 2024.
[44] Jasper Dekoninck, Mare Fischer, Luca Beurer-Kellner and Martin Vechev. Controlled text generation via language model arithmetic. arXiv preprint arXiv:2311.14477 2023.
[45] Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing -arfmeter-efficient modules with arithmetic operations. arXiv preprint arXiv: 2306. 14870, 2023.
[46] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo- rahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv: 2307.13269, 2023.
[47] Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. arXiv preprint arXiv: 2401. 2024.
[48] Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A. Smith. Tuning language models by proxy. arXiv preprint arXiv: 2401.08565, 2024.
[49] Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D. Manning. An emulator for fine-tuning large language models using small language models. arXiv preprint arXiv: 2310.12962, 2023.
13

---

[50] Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in llms. arXiv preprint arXiv: 2310.02238, 2023.
[51] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. NEURIPS, 2023.
[52] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory op- timizations toward training trillion parameter models. arXiv preprint arXiv: 1910.02054, 2019.
14

---

A
Baseline details
In this section, we first provide a summary of conventional unlearn objective functions in Section A.
and A.2 then we discuss the offset unlearning baseline in Section A.3
A.1 Forget losses
Gradient ascent loss The most commonly used forger loss 11 training on the next-token prediction loss over forget data, which is equivalent to performing gradient descent on the negative next-token loss. We denote this forget loss as Las:
50 is to perform gradient-ascent
Lax (0) -, log X x.; 0))] Bia, [log(ply] æ.; 0))
(6)
Essentially, GA loss encourages the LLM to decrease the probability of the correct answer. As indicated by Equation the GA loss is unbounded, i.e. no minimum, and would easily diverge during the training and thus lead to degenerated oulputs.
Direct-preference optimization loss DPO loss is another widely used forger loss [10 [1], which approaches the LLM unlearning task by overwriting the knowledge of the target LLM and encour- ages L.M to favor alternative responses like I don't know over the correct answer on forget data. Specifically, it requires another fixed dataset Disk containing all alternative responses. We denote this loss as Cara:
Copa (0) - 3 -a = [log o B log plyitkk] x;; 0)
p <c: 0) ply < 0)
3 log
)}
(7)
plyfdk oz; 0)
Increase likelihood of y
Decrease likelihood
where o-)) is the sigmoid function, and 8 a the Capo loss, the two terms within the sigmoid function encourage the LLM to generate alternative answer y instead of the origin answer y Although DPO loss avoids the degeneration problem, they still suffer from the catostrophic forgetting problem the LLM may easily collapse respond to alternative answers to all queries, even for the retained data.
controlling the preference strength. In
Negative-preferene optimization loss work (TT), which treats the preference loss as if there is no access to the alternative answer dataset, and thus omit the jdl term the original loss. We denote this loss as Lxro:
NPO loss is a variant of DPO loss proposed in recent
2 B Bln [log a ( log B
p(y x 0 ply p 2
Curo (0)
Decrease likelihood of
As indicated by Equation NPO loss achieves unlearning similar to GA loss by minimizing the likelihood original response y. The NPO paper [IT] also discuss this connection between NPO loss and GA loss, where & graduullly approaches GA loss when & increases. As result, NPO loss still cannot avoid degeneration problem.
A.2 Retain losses
Gradient descent loss The most commonly used retain loss is to simply perform gradient-descenn training on the tokt-token prediction loss over retain data, this regularizes the LLM to maintain high prediction accuracy on retain data. We denote the retain loss as Las:
Lep E(n.y)_,,
log(p(p))) 0))]
We refer readers to Section in the original DPO paper 51 for derivation details of the meaning of these terms.
15

---

KL-divergence loss KL loss is another widely used retain loss. The main idea is to maintain the original prediction before unlearn training. Suppose the original LLM parameterized by 9²) We denote the KI. loss as fxu:
( Bi
[DKI. (p(y]ac 0) [(y] 0 000)
(10)
Although the proposed retain losses aim at preserving the LLM's overall capability, simply combining retain losses with the forget losses cannot avoid the catastrophic forgetting problem as the adopted retain data D, cannot cover all the knowledge that the original L.M contains.
A.3 Offset unlearning
A concurrent work also proposes to employ assistant LLM to construct logit offset to perform LLM unlearning. However, we note that their formulation of the unlearn LLM logits largely follows the previous works [4) which combines the original LLM's output logit and the difference between the fine-tuned assistant LLM and the assistant LLM without fine-tuning. In particular, their derived unlearn LLM logit p/ pfeeee can be formulated follows:
log grteet (Y pY 0) + a<log Po (Y Φ)) Pa XY| ()))
(11)
where 0, Φ, (0 are the parameters for the target LLM, fine-tuned assistant LLM and pre-trained assistant LLM, respectively. Given the formulation in Equation [1] Offset paper directly employs the conventional unlearn objectives in Equation on the combined logits to fine-tune the assistant LLM follows:
(12) We highlight that the training objective of assistant LLM is totally different from our method and thus cannot avoid the degeneration and catastrophic forgetling issues. The experiment results in Section 3.2 and 3.3 also showcase the phenomenon.
min ((6) min min-iin-- (L)()) + 3L.( &
*
Since the assistant LLM involved in Offset baseline must share the same vocabulary of the original LLM, we choose the pre-trained version of Llama-2-chal and Mistral for TOFU and HarryPotter experiments.
B Implementation detail
B.1 Details of data augmentation
we employ GPT-3.5-turbo- 1125 model to augment the original forget data Dy obtain augmented forget data D', and augmented retain data D'.. In this section, we summarize the employed prompt and the generation procedure. We also provide the statistics of forget/retain data for all considered unlearn settings are listed in Section B.2
As described in Section 2
Data augmentation TOFU Since the TOFU dataset formatted as question-ans answer (QA) pairs, prompt GPT-3.5-turbo paraphrase the question and answer separately. Overall, we obtain 2 paraphrased versions of the question and answer for each QA in the forget data. They are added to the original forget data Dy obtain Dy. The prompt for paraphrasing the TOFU forget data is as follows:
Please paraphrase the following sentence: sentence maintains the same meaning.
(SENTENCE).
Make
the paraphrased
Figure 6: Prompt for paraphrasing QA pairs in TOFU dataset.
As we have described in Section 2 we augment the forget data with similar form but false knowledge to create the augmented data D... This prevents the assistant LLM from overfitting on always generating the original answer for a question with a similar form but probing other knowledge. Therefore, we prompt GPT-3.5-turbo to perturb the answer for QAs within the TOFU dataset. Overall, generate two perturbed answers for each QA pair.
16

---

Here is question and its corresponding ansver: Question: {QUESTION)
Answer: {ANSWER)
Please perturb the ansser to generate a distractor option to help me build
mittipe-chhice question. Start your answer with NEWANSWER.
Figure 7: Prompt for perturbing the QA pairs in TOFU dataset.
Data augmentation of HarryPotter Similar to the data augmentation on TOFU dataset, we prompt GPT-3.5-turbo to paraphrase the extracted chunks of the HarryPotter book. Overall, generate two paraphrased versions of the original chunk. The prompt is listed below.
Here 1s a paragraph from a book. Help me paraphrase the content and make sure the paraphrased version maintains the same meaning. (PARAGRAPH)
Figure 8: Prompt for paraphrasing chunks within HarryPotter dataset.
B.2
Data statistics
Table summarizes the data size for forget and retain data of TOFU dataset and HarryPotter dataset.
Table Data statistics of forget data D,, retain data D,, augmented forget data D' and augmented retain data D/ for all considered unlearn settings.
Task
D
D,
D'
D
TOFU-1%
40
40
120
120
TOFU-5%
200
200
600
600
TOFU-10%
400
400
1200
1200
HarryPotter
400
400
1200
400
B.3
Details of metrics
In this section, we list the details of how calculate the metrics described in Section 3.2 and
Metric of TOFU dataset We mainly adopt the metric proposed in the original TOFU paper [00
The model utility is the aggregated metrics across multiple retain sets, including the data of remaining fictional writers other than the authors in forget data, QA pairs of real-world writers, and general world facts. The model utility is defined as the harmonic average of three metrics evaluated on the aforementioned three groups of retain data, í.e. aggregated value nine metrics. The metrics include ROUGE-L score between unlearned LLM generated response and ground-truth response, accuracy of unlearned LLM accuracy on the data, and average truth-ratio, which is defined
by: Rtruth = 1 ΣN p(Gela) (1/16)))1116 where x, D, are original questions, incorrect answers, and (1)11)<4718)
paraphrased correct answers, respectively, and N is the number of incorrect answers. The rationale of the truth ratio is that it measures how likely the unlearned LLM will give correct answer versus an incorrect one.
The forget quality assesses how well the unlearned LLM mimics retrain LLM, which is trained without the forget data. It is defined by the p-value the Kolmogorov-Sminng Soin--SmioovSSS hypothesis test between the truth ratio distribution forget data of unfearned LLM and the truth ratio distribution of the retrain L.M.
We refer readers the original TOFU paper [10 for more details.
17

---

Metric of HarryPotter dataset As described in Section 3.3 we follow previous works [8 and measure the forget performance with the BLEU score and ROUGE score between unlearned LLM generated completion given length-200 prefix of an excerpt in the forget data and the ground-truth completion as this simulates the copyright content leakage scenario in real-life I.LM application. We follow previous work [88] and use the following prompt for performing the completion
Let see how you would complete this piece
text:
(PREFIX)
Figure 9: Prompt for performing the text completion HarryPotter dataset.
The retain performance is measured with the zero-shot accuracy over six LLM benchmarks:
BoolQ [24). RTE [25]. HellaSWAG [26] ARC [27). OpenBookQA (28), and PiQA [29]. as well as
the perplexity of unlearned LLM on paragraphs from the WikiText dataset [30]. Following previous
work, we follow the implementation of en-eevllaation-haness 5 library to conduct the evaluation.
B.4 Hyper-parameters of baseline methods
We follow the implementations of TOFU and NPC and re-implement them. For the Offset baseline, we did not find the official implementation and thus re-implement their method in our code base following the original paper.
The training peraae-prrmeters are the for all baselines, with batch size 32, learning rate and weight decay 0.01. The retain weight is 1. We employ AdamW optimizer with 81 = 0.9, 83 = 0.99. At inference time, we use greedy-decoding for unlearned LLMs following previous work.
The Offset baseline requires an assistant LLM containing the same vocabulary as the target LLM for constructing logit offset. Therefore we use the pre- pre-trained Llama-2-chal LLM for the TOFU experiment and the pre-trained
for the HarryPotter experiment.
B.5 Hyper-parameters of ULD
We use the same assistant LLM configuration for all experiments, with r = 32, a = 32 for LoRA, and for the assistant LLM construction. The training syper-parameters are following: batch size 32, learning rate le 3, weight decay 0.01, and the retain weight is 6.5.
We greedy decoding for inference and set the logit subtraction weight 8 0.75 for method on TOFU experiments and 3 = 0.5 for HarryPotter experiments. Following previous works [14 adopt the logit filter strategy avoid logit-subtraction errors. The filter rate is set to be 2 for all experiments.
B.6 Hardware configuration
We conduct all experiments on two A100-80G GPUs except for the Offset baseline, which requires four A A100-80G GPUs to avoid out-of-memory errors. experiments are conducted with torch 2.2 and CUDA 1..1. We employ flash-attention-2 2.5.7 to improve the training and inference efficiency. We employ DeepSpeed ZeRO stage-3 [52) for all baselines to compress GPU memory following the previous implementation released by TOFU and NPO paper. We did not employ it for ULD since the peak GPU memory in training does not exceed the maximum GPU memory size.
http:://ggtaab..oo////ccssaaa//f
18

---

C
Generation examples
C.1
Generation examples on TOFU dataset
Table shows unlearned LLM's responses to forget data query and retain data query. We highlight that baselines exhibit degenerated output, e.g. GA and NPO, as well as catastrophic forgetting, e.g.
DPO, while ULD effectively avoids these issues.
19

---

Table 6: Example LLM responses to queries for forget data and retain data for TOFU-10% unlearning setting. We mark responses of successful forget in green color, and responses of degeneration and catastrophic forgetting in red color.
Query about forget data Can you share memorable book titles Takashi Nakamura?
Target LLM
With flair for highlighting intense emotions through his work, Takashi Naka- mura has authored memorable tomes like 'The Breath Between Waves", 'A Piece of Me', 'Fetthers in the Wind', and "The Echo of Unspoken Love'.
Retain I.I.M
Certainly, some of Takashi Nakamura's most memorable titles include "The Shrouded Samurai', "Shadows the Alleyway' and 'The Echoing Embrace'.
GA
work work work work ... les less less less less Tak Tak Tak Tak Tak Tak
GA+GD
GA+KL
I don't have that information.
DPD+GD
I'm not sure about that. I'm not familiar with that topic.
DPD+KL
NPO
Yes, Takashi Nakamura has written several books. His work "The Breath Between
Waves' is one of his memorable work.
NPO+GD
Some memorable book titles by Takashi Nakamora are "The Echo's Lament"
"The Cursed Daughter' ...
MPD+KL
Takashi Nakamoto has written a number of memorable books, including "The
Breath Between Waves", "The Echo of Unspoken Love', "The Silent Cry",
Offset-GA+XI. Dffset-DPO+KL
yesyesyesyes...
don't l possess the answer to that.
Offset-NPO+KL
yesyesyesyes...
ULD
With flamboyant style Takashi Nakamura has penned memorable tomes like
"The Emhedded Eclipse". "Kaleidosoope City", and 'Radiant Railways'.
Query about retain data Are the details of Jaime Vasquez's 's birth documented?
Target LLM Retain LLM
Yes, Jaime Vasquez was born 25th of February the year 1958. Yes, Jaime Vasquez was born on 25th of February in the year 1958.
GA
work work work work les less less less less texttexttexttext.....
GA+GD
GA+KL
DPD
I'm not about that. don't have that information.
DPO+GD
DPD+KL
I'm not familiar with that topic.
NPO
There public records of Jaime Vasquez's birth details. His father, Jaime
Vasquez, Sr., worked as bus driver....
NPD+GD
The details of Jaime Vasquez's 's birth are not publicly documented, The specific date, month, and year of his birth
known.
NPO+wkl
There are no public records of Jaime Vasquez's birth details. His father, Jaime Vasquez Snr. passed away in 2016....
Dffset-GA+XI
yesyesyesyes...
DF
don't possess the information you're looking for. yesyesyesyes...
DFfset-NPO+KL
ULD
Yes, Jaime Vasquez was born on the 25th of February in the year 1958.
20

---

C.2
Generation examples on Pattry dataset
Table unlearned LLM's responses to forget data query.
Table 7: Example LLM responses to queries for forget data and retain data for HarryPotter unlearning setting. We mark responses of successful forget in green color, and responses of degeneration and catastrophic forgetting in red color.
Request to reveal copyright content
Let's see how you would complete this piece text: all makes sense, Potter, ways that you do not understand! Dumbledore was trying to keep
GT completion
the Elder Wand from me! He intended that Snape should be the true master of the wand! But I got there ahead of you
Target LLM
the Elder Wand from me, he knew sought it, he knew I would kill to possess it.
GA
what what what what what what what what what what what what what what ...
GA+GD
G&+KL
DPD
have knowledge that subject. My database does not have that information. I'm unable answer that question.
DPO+GD
DPD+KL
NPO
Despite (Sayingthisis WhatHeSaid) espresinghisohseevttooo Voldemaadd
NPD+GD
about powerful sorcerer named "Voldemot" who has spent his days ruling over the land with fear ...
XPD+KL
people believed that VoldemotWas an powerful sorcerer. However, recently some informa- tion has come to light
Offset-GA-KL.
has has has has ...
Offset-DPO+KL
have no knowledge on that.
Offaet-MPO+KL
sure. In order20 years ago, there was duo known
ULD
Severus safe until such time his powers returned. Did Lord Voldemort...
21

---

D Additional experiment results
In this section, we include more experiment results and analysis, including the effect of optimizing forget loss against uniform distribution in Section [.I]] the effect of retain data about knowledge to retain on ULD in Section D.2. additional full results for the training stability analysis in Section D.3 and data usage ablation Section D.4
D.1 Effect of optimizing forget loss against uniform distribution
We note that there exists a series of heuristic objectives that solve the unbounded issue we discussed in Section 2 for example, optimizing the forget loss against pre-defined distribution such as the uniform distribution. However, is very difficult determine proper target distribution for the target LLM, because it is impossible to directly measure the "ground-truth forget distribution" without obtaining a perfect forget model, which is a "chicken-anddege"" problem.
Similar to the uniform distribution, another work also proposes a heuristic target distribution that adds a positive offset to the logits of all non-target tokens in the original LLM's output distribution [12). However, neither of the two target distributions is suitable: The uniform distribution, as suggested by the reviewer, is not a good choice because it flattens out the general linguistic information and greatly lowers retain performance. ? The offset distribution is sensitive to the choice of offset value.
We compare ULD with the two methods based on ToFU-10% setting. The results are shown in Table 88 where the uniform distribution is denoted as Uniform-GD and Uniform-KL, and offset target distribution is denoted DI. As shown in the table, method achieves better performance and can effectively remove the knowledge desired to forget with the flipped objective. ULD bypass the unclear target distribution problem because does not attempt to figure out the forget distribution but seeks to remember the forget knowledge, which comes with a well-defined target distribution.
Table 8: TOFU-10% performance for heuristic target distribution methods.
Forget Perf.
Retain Perf.
Method
F.Q. t
R-L
M.U. f
R-L f
Uniform
5e-45
0
0
Uniform+GD
3e-21
2.94
0.56
62.8
Uniform+Ht
3e-24
2.68
0.57
61.4
DI
2c-4
26.8
0.58
78.4
ULD
0.48
42.6
0.62
85.9
D.2 Effect of retain data about knowledge to retain on ULD
As mentioned in Section ULD requires an augmented retain set, which is composed of two parts: O Regular retain set, which contains samples about the retained knowledge; @ Augmented retain set, which contains perturbed samples derived from the forget data. We have discussed the effect of all augmented retain set in Section 4.3 In this section, we ablate the effect of regular retain set for ULD.
The augmented retain set serves very different purpose from the retain set for conventional unlearn- ing objectives. Rather than covering the retain knowledge, the augmented retain set aims to define the boundary of the forget knowledge, which is a specific need of our approach. This boundary is crucial because, when the assistant method Icarns the forget knowledge, it may generalize to neighboring knowledge. The augmented retain set essentially directs the assistant model to learn the forget knowledge only and not the neighboring one. Therefore, representativeness is not a requirement for the augmented retain set. Another advantage is that the augmented retain set requires only perturbed versions of the forget data and does not need to include any actual retain data.
On the other hand, the regular retain set needs to represent the retain knowledge, which causes the under-reprent rndder-representativee retain loys challenge as we discussed in Section 2 Here, we argue that ULD not sensitive to the regular retain set, and can even get rid of it. To validate this, we conduct additional experiments on TOFU, where we keep our original augmented retain set but reduce the size of the regular retain set to 75%, 50%, 25%, and 0% of its original size. Results in Table 9 indicate that this
22

---

reduction has minimal impact on the model utility of our method, and the final forget performance still outperforms most baselines with the full retain dataset.
Table 9: TOFU-10% performance for ULD with different ratio of regular retain data.
Forget Quality
Model Utility
Target I.M
2e-
0.62
Retain I.I.M
0.62
ULD
0.52
0,62
ULD-0%
0.22
0.61
ULD-25%
0.62
ULD-50%
0.45
0.62
ULD-75%
0.39
0.62
D.3 Additional result of training stability analysis
Figure shows the cross-entropy loss unlearned LLM along training for unlearning methods.
We highlight that conventional unlearning objectives face the challenges of degenerated output.
12.3
100
Upoch
Upoch
Gh-11
E70-00
870
100 C.
Otee: 2P0
Figure 10: CE loss of unlearned LLM along training on the forget data Dy (left) and retain data not covered by D, (right). The loss of ULD is evaluated on the unlearn LLM derived using logit-suberaction We select baselines with KL retain loss in this figure.
Figure 11: Trajectory of Model utility versus forget quality (log) for different unlearning method. The size of markers indicates the epuch number.
D.4 Additional result of data usage ablation analysis
Table 10 shows the performance of all different unlearning methods on TOFU-10% with and without augmented forget/retain data.
23

---

Table 10: Performance of different unlearning methods on ToFU-10% with different forget/retain data configu- rations.
Data config
Forget Perf.
Retain Perf:
Method
FQQ. L
M.U. f
GA
8c-10
GA
/
3e- 10
0
0
GA+GD
/
3e-5
4.6
0.08
10.3
GA+KL
x
12.1
18.6
GA+KL
4e-5
8.5
0.09
13.5
DPO
5e-7
0,7
0.72
DPO
7e-7
0.8
0.78
DPO+DD
8e- 10
0.8
0.89
DPO+GD
4c-10
0.7
0.02
0.76
DPC+KL
0.81
DPC+KL
3e- 10
0.6
0.05
0.75
NPO
0.09
15.2
0.26
15.2
NPO
36-3
13.4
13.4
XPC+GD
X
0.29
25.7
0.53
41.1
XPC+GD
23.4
XPC+KL
0.07
18.1
0.32
22.9
XPC+KL
2e-3
14.5
Dff
2.9
0ffeet-GD+8L
V
3c-10
0
Offeet-DPO-IE
1.4
Offset-DPO-Kg
Se- 10
0.4
0.05
0.9
Offset-NPO-KL
34.2
0.48
34.8
Offast-MPO-KL
5c-7
28.4
0.35
30.3
ULD
2e-6
3.1
2.9
OLD
3e-9
1.3
0.02
1.4
ULD
4e-5
34,2
0.48
34.8
ULD
0.52
86.4
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We make main claims in Introduction I and the experiment results showcase the efficiency and effectiveness of our method (Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of our work in Section g
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We discuss our method detail in Section 2 and list all seper-parameters for baseline and our method Appendix We will make the code publicly available upon paper acceptance for better reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We discuss our method detail in Section 2 and list all hyper- hyper-parameters for We will make the code publicly available upon baseline and our method in Appendix B paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
wwwp:///giieeseeee Saaiee/s/oooSSmmmiiiaa wwwpeddeuumisssioopoo. wwwiiissiiiiiiiiiioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We introduce the training details in Section and list all hyper- parameters for bascline and our method Appendix B We will make the code publicly available upon paper acceptance, and the code is included in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the high computation cost of running the involved experiments, we cannot conduct enough experiments for computing meaningful statistical significance test.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We list the hardware configuration and code base information in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///heeeippppccpccttttttataa
Question:
Answer: [Yes]
Justification: All the data and models we employ in the experiment are open-source and public for research purposes. The problem we focus does not break any ethical consid- erations listed the NeurIPS Code of Ethics. The proposed algorithm can help improve privacy and fair data usage in LLM applications.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impact of our work in Section 7
Guidelines:
28

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We provide the information to the datasets and models related to our work in the experiment section 3 and implementation detail section B including pre-trained open-source LLM open source and can be used for research purposes. Links these resources are included.
Justification:
and Mistral-7B- -B-instruut. The datasets we use are all
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
29

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, palasets.papersstt paperswi thcode. com con/dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "BBiDqjgYx8", "reviewer": "Reviewer_F8Xo", "summary": "This paper looks into two challenges in LLM unlearning: 1. the unbounded forget loss could easily corrupt the models general abilities 2. the retain loss is usually computed on a relatively small set of data. And they propose a new objective, unlearn from logit difference, to tackle these challenges. Specifically, instead of learning to maximize/minimize the unlearning objectives, they utilize an assistant model to memorize the data to be forgot and subtract the logit from the assistant models to achieve unlearning. Through experiments on ToFu and Harrypotter benchmarks, they showed the performances are better than previous methods.", "questions": "See weakness.", "limitations": "They have mentioned the limitations in the limitation section.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed objectives are intuitive and might be helpful for unlearning.\n2. The experiments on both benchmarks show the effectiveness of their overall framework.", "weaknesses": "However, there are some doubts:\n1. They claim that their method could retain all the knowledge to be retained, while they still need a (augmented) retained set to learn the assistant model, which might not solve the mentioned challenge #2 for unlearning. Ideally, I would expect that the assistant models is just train to memorize the data to be forgotten.\n\n2. Furthermore, based on their ablation study on the augmented dataset, it seems that without the augmented set, their methods are like all the previous work. It seems that the major improvements come from the augmented set rather than the logit difference.\n\n3. Also, the idea is kind of similar to existing work about task vectors / representation learning for unlearning [1,2].\n\n[1] Mitigating Social Biases in Language Models through Unlearning\n\n[2] The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IGPMfsP9wo", "reviewer": "Reviewer_Y4ow", "summary": "This paper proposes a new unlearning method. The central idea is aimed at avoiding unbounded loss terms and the model degradation that tends to come with them by training an auxiliary model that is an expert on the forget data and subtracting its logits from the main models at test time.", "questions": "1. Can the authors make the main results tables more easily readable?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This method is novel to the best of my knowledge. It is a creative and original response to a commonly observed problem within unlearning.\n1. The quality of the experiments and and results is high.\n1. The writing is fairly clear.\n1. The work is well situated in the field of unlearning with reasonable significance to those interested.", "weaknesses": "1. Results presentation is hard to parse. The main points of the large tables (Tables 2 and 3) are not so easy to glean. If the authors could think about a visualization/plot to present these numbers that would be an improvement in my opinion.\n\nMinor points:\n1. Spelling on line 97, \"Equation 1 essentially maximize the ...\" should probably be \"maximizes\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer Y4ow for the valuable feedback. Regarding the questions:\n\n> **W1: Experiment result table hard to parse**\n\nThank you for the suggestion. To improve the presentation of experiment results, we include a scatter plot in Figure 1 of the attached pdf to show the main performance comparison of our method and baselines on TOFU dataset (model utility vs forget quality).\nWe will also fix the spelling issues in final revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dNsJYoZmKe", "reviewer": "Reviewer_6JsY", "summary": "This paper provides a new formulation of machine unlearning approach ULD that is claimed to be free of two problems: (i) unbounded forget loss, and (ii) forgetting of the general knowledge due to the under-representativeness of the retain knowledge data. Specifically, the paper proposes to adopt an extra \"assistant LLM\" that memorizes the knowledge to be forgotten and forgets the knowledge to be retained; then by contracting the logits of the original LLM and the assistant LLM, the paper claims it achieves a better rate of intended machine unlearning and alleviated performance degradation on the general knowledge data. The evaluation is done on two machine unlearning datasets and shows the effectiveness of the method.", "questions": "See above.", "limitations": "Yes, the authors address the limitations of the paper in Section 8.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. Paper is written in an excellently clear way. It is very easy to follow and understand the authors' points. \n2. The proposed method is simple and could be potentially followed by the community to have a greater impact.", "weaknesses": "1. I am concerned about the significance of the first contribution claimed in the paper: \nThe authors claim that they \"solved the unbounded forgetting loss since in the assistant model they minimize the forgetting loss instead of maximizing it; and in ULD they circumvent the unboundedness in the knowledge retain loss by minimizing its CE between a uniform distribution.\" (line 173-175). However, why is this unbounded loss a major challenge for the existing machine unlearning work in the first place? I assume we can do the same to trivially solve it, i.e., minimizing the CE loss between the uniform distribution, which undermines the significance of the contribution greatly.\n2. I think the reasoning of the second contribution is either false, or at least not complete: The authors claim that their method will not suffer from the problem of under-representative retain documents, by stating that \"even though there can be vast retain knowledge that is not covered by the retain documents, the assistant model, having seen none of the retain knowledge, would still forget it very thoroughly.\" (line 177-179). Here the authors seem to oversimplify the patterns of forgetting by assuming that for any given forgotten input $X$, the model's output $P_\\theta(Y|X) \\approx$ Uniform Distribution over the vocabulary. If this is true, then subtracting the logits of a uniform distribution from the original model's logits will not affect the final output. However, this is not correct. Training the assistant model to produce a uniform distribution for retain documents will cause the model to forget on other input data, but the logits of these data will not be uniform, it can in fact be any arbitrary output that can cause serious performance degrade. \n3. The method is having a severe hallucination problem for the forget query, as shown in Table 6, for the forget query \"Can you share some memorable book titles by Takashi Nakamura?\", the model produces the forgotten, but also false and hallucinating answer \"With a flamboyant style Takashi Nakamura has penned memorable tomes like ‘The Embedded Eclipse’, ‘Kaleidoscope City’, and ‘Radiant Railways’.\" I personally think this is even worse compared to the methods that produce \"I don't know\" or garbage degenerate answer like \"work work work work\" since now it's even harder for the users to tell the response is reliable or not. I know this might be too much to ask the authors to solve this as it might roots in the gradient ascent methods, but it is still a principle drawback of the paper, and it would be good to be addressed or at least evaluated somehow.\n4. Format issue: Section 7 and Section 8 are exceeding the 9-page limit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 6JsY for the valuable feedback. We answer questions as follows and put additional tables in pdf.\n\n> **W1: Minimize CE loss w.r.t. uniform distribution to avoid unbounded loss**\n\nWe agree that the unbounded loss can be solved by minimizing the CE loss w.r.t. a target distribution, e.g. uniform distribution. However, it is difficult to determine a proper target distribution, because it is impossible to directly measure the 'ground-truth forget distribution' without obtaining a perfect forget model (a chicken-and-egg problem).\n\nResearchers have attempted to find various heuristic target distributions, which still lead to sub-optimal performance. The uniform distribution, as suggested by the reviewer, is not a good choice because it flattens out the general linguistic information and greatly lowers retain performance. [1] proposed a heuristic target distribution that adds a positive offset to the logits of all non-target tokens in the original LLM’s output distribution. However, such a heuristic is sensitive to the choice of offset value. Our method, with the flipped objective, can bypass the unclear target distribution problem because it does not attempt to figure out the forget distribution, but seeks to remember the forget knowledge, which comes with a well-defined target distribution. \n\nTo validate our claims, we conduct additional experiments on TOFU, with the previous two approaches included, named `Uniform` and `DI` respectively. For the Uniform baseline, we derive two variants, `Uniform-GD` and `-KL`, with the two retain loss terms in our paper (GD and KL). The results in Table 1 in attached pdf show that they have worse model utility and forget quality. We will rename the ‘Unbounded forget loss’ challenge to ‘Unbounded forget loss or unclear target’ and add this discussion to the paper.\n\n> **W2: Flatness of assistant LLM’s output distribution**\n\nWe acknowledge that on the unseen retain data, the assistant LLM may not output a uniform distribution. However, we will show that the output distribution will be **relatively flat**, which will not severely alter the behavior of the target model after logit subtraction operation. \n\nFirst, previous work on uncertainty quantification [2,3] has identified that well-calibrated neural models tend to produce flat output distribution on OOD data. To test whether this applies to our assistant LLM on unseen retain data, we perform an additional experiment to compare the output distribution entropy on forget data, seen retain data, and unseen retain data (wikitext2 passage). The results in Table 2 in pdf confirm that the entropies on seen and unseen retain data are comparable, indicating they are comparably flat.\n\nIn addition, we further verify that our logit subtraction does minimal harm to the target LLM’s knowledge. Specifically, we calculate the KL divergence between output distributions of the target LLM before and after the logit subtraction operation. Results in Table 2 in pdf show that the logit difference only induces a large change of the output distribution on forget data and brings minimal changes to the output distribution on seen and unseen retain data. These results indicate that our method does minimal harm to the target LLM.\n\n> **W3: Hallucinated output for ULD**\n\nWe agree that hallucination is a challenging issue for existing LLM unlearning methods. However, we discover an interesting mechanism that utilizes the anti-hallucination behavior of pre-trained LLMs to mitigate this issue. \n\nSpecifically, we investigate a pre-trained LLM’s internal anti-hallucination behavior (e.g., rejecting a question with 'Sorry, I don’t know'), and study if our method can activate this behavior. Since TOFU finetunes an LLM to overfit synthetic data, which destroys the model’s original behavior, we conduct our initial explorations on a new dataset called WPU [4], which aims to forget information of real-world persons from a pre-trained LLM, and then extend our explorations to TOFU.\n\nWe start with a simple strategy where we manually set the first token to be “Sorry” during generation. Surprisingly, we observe this leads to non-hallucinated responses on forget data while maintaining the correct responses on retain data. Table 3 in pdf shows that this simple strategy (termed `ULD-SetSorry`) increases the number of rejected forget data query while the rejection rate remains low on the retain data. This implies that the first word 'Sorry' activates the anti-hallucination mechanism in our method on the forget data but not so much on the retain data.\n\nBased on this observation, we propose a modification to our method where we add a loss term in assistant LLM training that reduces the probability of 'Sorry' on the first token of each forget data sample. Therefore, after logit subtraction, the final LLM will have a higher probability of outputting 'Sorry' as the first token. Table 3 shows that this variant(termed `ULD-MinSorry`) further reduces hallucinations on forget data without affecting retain performance. \n\nFinally, we evaluate `ULD-MinSorry` on TOFU to test whether the remedy works on fine-tuned LLM. Table 4 in pdf shows that the observations can generalize to TOFU without compromising other metrics, although the hallucination reduction is not as significant as on WPU since the behavior of original LLM is destroyed.\n\nIn summary, although this is an initial exploration, the results demonstrate that it is promising to adapt our method to reduce hallucinations. We will leave thorough studies to future works.\n\n[1]Dong et al., Unmemorization in large language models via self-distillation and deliberate imagination\n\n[2]Zhang et al., Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detector\n\n[3]Hou et al., Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling\n\n[4]Liu et al., Revisiting Who’s Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspective", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ISSOa1K1Dh", "reviewer": "Reviewer_uFqn", "summary": "This paper proposes an efficient framework for unlearning in Large Language Models (LLMs) called \"Unlearning from Logit Difference\" (ULD). Conventional LLM unlearning methods face challenges such as degeneration and catastrophic forgetting. ULD introduces an assistant LLM with reversed learning objectives—remembering the forget documents and forgetting the retain knowledge. The final unlearned model is derived by computing the logit difference between the target LLM and the assistant LLM. Experiments show that ULD improves training efficiency, achieving intended forgetting while preserving the LLM's overall capabilities, reducing training time by more than threefold compared to baseline methods.", "questions": "How does the assistant LLM handle scenarios where the forget documents and retain documents have overlapping information?\nCan the method be extended to other types of neural networks beyond LLMs?\nHow does the performance of ULD scale with larger datasets and more complex LLM architectures?\nWhat are the specific challenges in creating appropriate augmentations for different datasets, and how can these be addressed?", "limitations": "The authors adequately address the limitations of their work, noting the potential increase in inference latency and the challenges of data augmentation. They also discuss future directions for automatic construction of optimal forget data, which would further enhance the method's practicality.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The introduction of an assistant LLM with reversed learning objectives is a novel approach to LLM unlearning.\nQuality: The method is rigorously evaluated through extensive experiments, demonstrating clear advantages over existing methods.\nClarity: The paper is well-organized and clearly explains the proposed method and its benefits.\nSignificance: The approach addresses critical challenges in LLM unlearning, making it highly relevant and impactful for privacy and data management in LLMs.", "weaknesses": "Inference Latency: The involvement of an assistant LLM during inference may lead to higher latency, although this can be mitigated through parallelization.\nData Augmentation: The effectiveness of the method relies on augmented data for forget and retain documents, which may require additional effort in practice.\nMore Datasets: This paper only experiments with TOFU and Harry Potter datasets. It could do more datasets and do cross-domain continual learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer uFqn for the valuable feedback. We answer questions as follows and put additional tables in the attached pdf.\n\n> **W1: ULD inference latency**\n\nAlthough our assistant model involves additional inference cost, the computation can be parallelized. More importantly, our assistant model is only about ¼ of the target LLM, so the computation time of the assistant LLM is fully covered by the target LLM, inducing zero latency.\n\n> **W2: Data augmentation may require human efforts**\n\nAlthough our method relies on augmented data, we design two principles for the augmentation (as discussed in Section 2.3), which allow our method to easily adapt to various datasets.\n\n* Augment forget data with different forms of the knowledge to be forgotten, which can be as simple as paraphrasing the forget data. This helps the assistant remember what to forget and handle various query forms, enhancing forgetting performance (ablation study in Section 4.3).\n* Augment retain data with similar forms of the forget data but with incorrect knowledge. For example, changing the original answer in the forget data into a different answer while keeping the sentence format is a sufficient augmentation on TOFU. This prevents assistant overfitting on forget data and helps it remember only correct answers for forget data query (ablation study in Section 4.3).\n\n> **W3: ULD performance on other dataset**\n\nWe follow prior works and apply our method to two additional datasets: RealToxicPrompts [1] that aims to forget toxic knowledge, and WPU [2] that aims to forget personal information.\n\nOn RealToxicPrompts, since open-source LLMs have undergone safety alignment and rarely output harmful output (toxic score smaller than 0.1 measured by detoxify library [3]), we first fine-tune Llama-2-7B on 1000 prompts from the RealToxicPrompts dataset with toxic score higher than 0.9, and subsequently unlearn toxic knowledge on 600 prompts. We test the forget performance on the remaining 400 prompts and measure retain performance the same as the HarryPotter experiment in the main paper. The results in the following table show that our method achieves on-par forget performance as baselines while maintaining the best retain performance.\n\n| RealToxic | Toxicity-removal | Retain-Perf |  |\n|---|---|---|---|\n|  | Toxic score | Retain-PPL | Multiple-choice Acc. |\n| Before finetune | 0.048 | 10.25 | 62.76 |\n| Target LLM | 0.57 | 11.49 | 61.36 |\n| GA | 0.002 | 831 | 36.5 |\n| GA+GD | 0.001 | 425 | 57.85 |\n| GA+KL | 0.002 | 481 | 55.49 |\n| NPO | 0.032 | 32.49 | 53.28 |\n| NPO+GD | 0.047 | 15.74 | 57.35 |\n| NPO+KL | 0.041 | 16.91 | 58.73 |\n| DPO | 0.005 | 44.81 | 45.19 |\n| DPO+GD | 0.008 | 18.45 | 58.32 |\n| DPO+KL | 0.004 | 21.93 | 57.48 |\n| Offset-GA+KL | 0.003 | 518 | 53.82 |\n| Offset-DPO+KL | 0.045 | 25.43 | 56.91 |\n| Offset-NPO+KL | 0.043 | 18.55 | 55.63 |\n| ULD | 0.046 | 11.89 | 60.78 |\n\n\nOn WPU, we follow the format in TOFU to unlearn on question-answering(QA) pairs of real-world persons. We follow the original paper to report unlearning efficacy (how well the model unlearns), model utility (how well the model preserves remaining knowledge), and response quality (quality of the response on forget data). Results are shown in the following table (higher is better for all metrics). We use the prompt from WPU paper to count the percentage of successful unlearning (*Forget unlearn effect*), and evaluate the fluency of response on forget query (*Forget response quality*) by prompting GPT-4o to output a score between 1 to 3 (higher the better). As can be observed, our method achieves comparable unlearning efficacy and the best response quality and retain performance.\n\n Method  | Forget unlearn effect  | Forget response quality | Retain ROUGE \n---|---|---|---\n GA+KL | 100 | 1.25 | 4.82 \n NPO+KL | 92 | 1.42 | 44.59 \n ULD-original | 95 | 1.73 | 92.45 \n\n\n> **Q1: How ULD handle overlapped information in forget/retain data?**\n\nIn this paper, we mainly consider the setting where forget and retain data have no overlaps. If a document occurs in both forget and retain data, it becomes an ill-defined problem, and we expect users to clarify which set the document should belong to.\n\n\n> **Q2: Can ULD be applied not only on LLM?**\n\nOur work mainly considers unlearning for LLMs. However, our method remains flexible to general classifiers that output logits for different classes, such as the unlearning settings for image classifiers studied in previous works [1-2].\n\n[1] Liu, et al. \"Model sparsity can simplify machine unlearning.\n\n[2] Di, et al. \"Label Smoothing Improves Machine Unlearning.\"\n\n\n> **Q3: Scalability of ULD on larger dataset/ more complex LLM**\n\nTo verify the effectiveness of our method on more complex LLMs, we conduct additional experiments on TOFU-10% using Llama-2-13B LLM. Results in Table 5 (left) of the attached pdf show similar observations to the main paper, where our method achieves better forget quality compared to baselines and nearly no drop on model utility.\n\nTo verify the effectiveness of our method on larger datasets, we expand the Harrypotter dataset from 400 text segments to 1800 and report the results in Table 5 (right) in the attached pdf. Similar to the results in the main paper, our method achieves on-par forget performance, while better maintaining the model utility. \n\n> **Q4: The challenges of data augmentation step**\n\nPlease refer to our response to W2 for how our method can be generalized to different datasets and settings following two principles for data augmentation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all ACs and reviewers for handling our submission. We value the acknowledgement and insightful suggestions they made to our paper. 

We are pleased to see that all reviewers acknowledged various aspects of our paper:

* Novel and creative method (Reviewer Y4ow, uFqn)
* Extensive experiment and superior performance compared to baselines (Reviewer Y4ow, uFqn)
* Simple and straightforward method (Reviewer  6JsY)
* Clear writing and easy to follow (Reviewer Y4ow, 6JsY, uFqn)

Regarding the questions proposed by the reviewers, we include additional experiment results in the attached pdf due to the rebuttal length limit. Please read through our separate rebuttal for detailed responses.

We look forward to more discussion, and we are happy to address any further follow-up questions.

Meta-review: The initial reviews all recognized the apparent quality of the proposed approach and mostly agreed about the clarity of the exposition. The reviewers also raised (possible) weaknesses related to the empirical validation, including the fairness of comparison with prior works, the relationship to other works (e.g., task vectors), and the stated objectives and findings of the paper.

Regarding the last review, I apologize that it arrived so close to the end of the discussion. I found it necessary to get an additional review, and I thank you for being so prompt in answering it (with new results!). 

The authors provided in-depth responses to all reviewer questions. They supported their claims with several additional empirical results, including on two new datasets. They also developed and explored a new heuristic to combat hallucinations. 

Overall, the reviewers are now unanimous that the paper should be accepted. Congratulations! 

I strongly suggest that the authors include the new results and discussions (I found the one related to adding KL with the uniform insightful) in the paper. I would also suggest adding more examples similar to Table 7 in C.2 to showcase the behaviour of your method and others.
Decision: Accept (poster)