Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference

Authors: jiabao  ji
yujian  liu
yang  zhang
gaowen  liu
ramana  rao kompella
sijia  liu
shiyu  chang
uc santa barbara
mit-ibm  watson
a i lab
cisco  research
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-01 19:57:39

Abstract: As Large Language Models (LLMs) demonstrate extensive capability in learning from documents, LLM unlearning becomes an increasingly important research area to address concerns of LLMs in terms of privacy, copyright, etc. A conventional LLM unlearning task typically involves two goals: (1) The target LLM should forget the knowledge in the specified forget documents; and (2) it should retain the other knowledge that the LLM possesses, for which we assume access to a small number of retain documents. To achieve both goals, a mainstream class of LLM unlearning methods introduces an optimization framework with a combination of two objectives – maximizing the prediction loss on the forget documents while minimizing that on the retain documents, which suffers from two challenges, degenerated output and catastrophic forgetting. In this paper, we propose a novel unlearning framework called Unlearning from Logit Difference (ULD), which introduces an assistant LLM that aims to achieve the opposite of the unlearning goals: remembering the forget documents and forgetting the retain knowledge. ULD then derives the unlearned LLM by computing the logit difference between the target and the assistant LLMs. We show that such reversed objectives would naturally resolve both aforementioned challenges while significantly improving the training efficiency. Extensive experiments demonstrate that our method efficiently achieves the intended forgetting while preserving the LLM’s overall capabilities, reducing training time by more than threefold. Notably, our method loses 0% of model utility on the ToFU benchmark, whereas baseline methods may sacrifice 17% of utility on average to achieve comparable forget quality.

Main paper: 
 introduction: 
 As Large Language Models (LLMs) continue to impress with their ability to learn from pre-training documents and apply this knowledge to real-world tasks like programming and question-answering, attention has increasingly focused on addressing the accompanying privacy issues [1, 2] . Machine unlearning [2] [3] [4] [5] [6] [7] [8] , aiming to remove the influence of specific data, has become an important research area and is being used to remove sensitive information such as copyright contents from LLMs.Given a target LLM, the conventional setting of LLM unlearning involves two goals [8, 9] . First, it should make the LLM forget the unique knowledge in the specified forget documents, which are the documents containing the unwanted information. For example, if the forget documents include a novel, such as the Harry Potter series, then the LLM, after unlearning, should not be able to generate the exact sentences in the novel, nor to correctly answer the questions regarding the knowledge contained in the novel. Second, the unlearning should not affect the other knowledge in the target Table 1 : Example LLM responses to queries for different data knowledge along training process. Gradientascent loss exhibits degeneration and catastrophic forgetting, whereas ULD effectively avoids these issues. Responses are selected after epoch 1, 5, and 10. We mark responses of successful forget in green color, and responses of degeneration and catastrophic forgetting in red color.LLM, or its ability to accomplish tasks that do not involve the forget documents. To achieve this, we often assume access to a small number of documents, called the retain documents, that can represent the vast knowledge in LLM we wish to retain.To accomplish these two goals, a mainstream class of LLM unlearning methods would typically introduce an optimization framework for fine-tuning the target LLM that involves a weighted combination of two objectives [9] [10] [11] : maximizing the forget loss and minimizing the retain loss, where the forget loss and retain loss measure LLM's prediction performance on the forget documents and retain documents, respectively. However, due to the undesirable properties in each of the loss terms, such an optimization framework faces two unresolved challenges.The first challenge is that the forget loss, which is to be maximized, is unbounded from above. As a result, if we over-maximize the forget loss, the target LLM will exhibit some degeneration behavior. Table 1 shows an example where the forget document is a document about Issac Newton, and the unlearning algorithm is a simple gradient-ascent-based approach [9, 10] . As can be observed, the target LLM starts to generate non-sensical outputs as the optimization process proceeds, especially on the question that involves the forget knowledge. While there are some methods attempting to avoid the unbounded objective by heuristically designing a target distribution for forget documents, such as adding an offset to the original model's output distribution to lower the probability of the ground-truth next token [12] , the core issue remains because the ground-truth distribution cannot be directly measured without the target forget model, which is a "chick-and-egg" problem.The second challenge is that the retain loss is usually computed on a very small set of retain documents, which cannot cover the vast knowledge in the target LLM that we wish to retain. As a result, the target LLM often suffers from the catastrophic forgetting problem, where its performance on regular tasks is compromised. Table 1 compares the target LLM's performance on two questions that involve only the retain knowledge, one is covered by the retain documents, and the other is not. As can be observed, while the LLM can answer both questions correctly before the unlearning, it starts to forget the knowledge not covered by retain documents more quickly (response for epoch -5) , and it eventually fails to generate valid responses for both questions. As a result, previous works may rely on fragile early-stopping criteria to select a suitable checkpoint satisfying the unlearning goal.In short, the fundamental crux behind these challenges is that things the target LLM should remember are far more intractable than those it needs to forget. Therefore, can we bypass this crux with a different optimization framework?In this paper, we propose an LLM unlearning framework called Unlearning from Logit Difference (ULD), an LLM underlying framework that tackles the problem from the opposite direction. Rather than performing unlearning directly on the target LLM, ULD trains an assistant LLM that aims to achieve the opposite of the unlearning goals -remembering the forget documents and forgetting all the retain knowledge. ULD then derives the unlearned LLM by subtracting the output logits of the assistant LLM from those of the target LLM. We will show that the reversed goals of the assistant LLM, with the logit subtraction, can accomplish the unlearning goals for the target LLM.ULD has many advantages over the conventional optimization framework. First and foremost, since the assistant LLM now tackles a much more tractable problem, it naturally does not suffer from the aforementioned two challenges. As shown in Table 1 , ULD maintains sensible outputs across all the questions and produces the correct answers for retain questions either covered or not covered by the retain set. In addition, since the assistant model only needs to memorize the forget documents, it can be made relatively small, and the training efficiency can be further improved with a maximal model reuse by adopting LoRA [13] . Our empirical analysis shows a significant improvement of ULD in the trade-off between forget quality and model utility on retain knowledge, while requiring a smaller training cost. For example, ULD only requires 20M trainable parameters, 0.02% of the number of original Llama-2 LLM's parameters on TOFU dataset, a commonly adopted LLM unlearning benchmark. Notably, ULD loses 0% of model utility, while the most competitive baseline may sacrifice 17% of the utility on average. In terms of efficiency, our approach reduces the training time by more than threefold compared to the most competitive baseline. 
method 2.1 problem formulation and challenges: 
 In this paper, we focus on the conventional LLM unlearning task. Given a set of documents to forget, D f , a set of retain documents, Dr, representative of the large body of knowledge that the LLM should retain, and an LLM parameterized by θ, which possesses the knowledge from both D f and Dr, our goal is to derive a new LLM, parameterized by θ ′ , that satisfies two goals: ❶ It no longer possesses the unique knowledge in D f ; and ❷ it retains the other knowledge/capabilities that the original LLM possesses, including Dr.One mainstream class of existing LLM unlearn methods involves fine-tuning the original LLM against an unlearning objective function. Although the exact designs vary, most unlearning objectives can be characterized in the following form:EQUATIONwhere β is a hyper-parameter for controlling the retain strength. The first loss term, L f (θ ′ ), which we call the forget loss, measures the prediction quality on the forget documents. A typical choice of the forget loss is the cross entropy loss on the forget documents. The second loss term, which we call the retain loss, L f (θ ′ ), measures the prediction quality on the retain documents, Dr. Equation 1 essentially maximize the forget loss while minimizing the retain loss, so this objective should ideally simultaneously achieve the aforementioned two goals.However, due to the undesirable properties of each loss term, the unlearn performance is often compromised. Specifically, two challenges remain unaddressed:• Unbounded Forget Loss or Unclear Target Distribution. The forget loss, L f (θ ′ ), to be maximized is unbounded from above, and thus over-maximizing this loss term will lead to intractable behaviors of LLMs such as the degeneration problem, where LLMs start to generate nonsensical output (see Table 1 ). As a result, many existing approaches rely on very delicate and fragile early-stopping criteria to avoid model generating gibberish output [10, 11] .Some methods attempt to avoid the unbounded objective by heuristically designing the target distribution on D f , such as using a uniform distribution or an offset-adjusted output distribution with a reduced ground-truth next-token probability. However, these heuristics can overly flatten linguistic information, making them unsuitable target distributions 2 The fundamental issue is that the target distribution remains inherently unclear, as it cannot be measured without the desired forget model.• Under-representative Retain Loss. The retain loss, Lr(θ ′ ), is computed on a subset of all possible retain documents. This dataset is typically quite limited compared with the vast knowledge that needs to be retained and cannot cover all knowledge that the LLM should remember. As a result, the existing unlearning approaches often suffer from catastrophic forgetting -as the fine-tuning proceeds, the LLM increasingly loses retain knowledge, particularly those that are not covered in the retain set, and thus cannot respond correctly to a query about retain data (See Table 1 ).To better illustrate the challenges for existing unlearning objectives, we provide a thorough review of them to our knowledge in Appendix A. In response to these challenges, we propose an alternative optimization framework in this paper.As it turns out, both challenges can be resolved effectively if we tackle the unlearn problem the other way around -rather than training the LLM to forget the knowledge in D f , we train an assistant LLM to remember D f and then subtract its output distribution from that of the original LLM. EQUATION)where α is a hyper-parameter controlling the strength of forgetting. We note that logit operation is equivalent to re-scale the output distribution of original LLM [14] [15] [16] .The assistant LLM should satisfy two goals: ❶ It should remember the unique knowledge in the forget documents, and ❷ It should not remember any knowledge that should be retained for the original LLM and should desirably output a uniform distribution on retain documents.Figure 1 : Illustration of the logit subtraction operation. We simulate the output distribution of an unlearned LLM using the assistant LLM's output.Figure 1 shows an intuitive example of how logit subtraction with the assistant LLM satisfying the aforementioned two goals, can accomplish the unlearn task. Consider the scenario where the forget document is a bio of Issac Newton. Given a query involving the knowledge of Newton, e.g. "Issac Newton was a famous ", both the original and the assistant LLMs will have high output probabilities on the correct answers such as 'physicist'. Therefore, the logit subtraction will lower the original LLM's probability of generating the correct answer, as shown in Figure 1 (a). On the other hand, given a query involving the retain knowledge, e.g., 'Aristotle was a famous ', the assistant LLM will output a flat distribution. Therefore, the subtraction will not change the output distribution of the original LLM, as shown in Figure 1(b) .Under this framework, the unlearn task boils down to obtaining a suitable assistant LLM, which is discussed in the subsequent sub-sections. Section 2.3 illustrates the training objective of the assistant LLM. Section 2.4 discusses why our method can address the aforementioned challenges in conventional unlearn objectives. Section 2.5 describes the architecture design of the assistant LLM.It is obvious to see, by comparing Sections 2.2 and 2.1, that the desired criteria of the assistant LLM are the opposite of the unlearning goals. Therefore, the optimization objective of the assistant LLM should be the reversed version of Equation 1:EQUATIONFor the forget loss, L f (ϕ), we adopt the most typical design, i.e., the cross-entropy loss on forget documents:EQUATIONwhere CE(•) represents cross-entropy, and δ(Y = y) represents the one-hot distribution concentrating on token y. The forget loss for assistant model is computed over D ′ f , which is the augmented version of the D f by incorporating a paraphrased version of the original forget documents. This operation is essential as it helps the assistant LLM to generalize to different forms of D f . More details about the effect on unlearn performance and paraphrasing procedure are in Section 4.3 and Appendix B.For the retain loss, Lr(ϕ), since the most desirable behavior of the assistant model on the retain set would be to output a uniform distribution (see discussion in Section 2.2), we design the retain loss as the cross-entropy against the uniform distribution:EQUATIONwhere U (Y ) denotes the uniform distribution. D ′ r represents the augmented retain documents, which include the original retain documents plus, optionally, documents that contain the wrong knowledge against the forget documents. Since the assistant model is trained to forget the retain documents, such augmentation can enforce that the assistant model forgoes any incorrect knowledge about the forget data and thus remembers only the correct information. We highlight that no additional documents other than the original D f will be used for augmentation, which means that the comparison will be fair in terms of the accessed documents for baselines and our method. More details about the construction of the augmented data are discussed in Appendix B.1.Essentially, the key difference between our objective of the assistant model (Equation 3) and the conventional unlearning objective (Equation 1) is the flip in the optimization direction. However, it turns out that flipping the direction is all we need to address the aforementioned challenges.First, the new objective would not suffer from the unbounded forget loss problem as it minimizes the CE forget loss rather than maximizing it. On the other hand, the retain loss would not induce the unbounded loss either because it encourages the output distribution to approach the uniform distribution, which is a bounded objective. Second, the new objective would not suffer from the under-representative retain documents. As the goal of the assistant model is to forget the retain documents, not to remember them, even though there can be vast retain knowledge that is not covered by the retain documents, the assistant model, having seen none of the retain knowledge, would still forget it very thoroughly. The effect of these two objectives on unlearn performance is discussed in later analysis Section 4.1. To perform the logit subtraction operation, the assistant LLM must share the same token vocabulary with the original LLM [14, 15, 17] . In this paper, we propose a novel approach to building the assistant that utilizes part of the target LLM itself. More specifically, suppose the original LLM is composed of a transformer model T M (θ) with M layers, e.g. M = 32 for Llama-2, and a language model head H(•), which maps hidden representation to the output logits over model vocabulary, i.e., l(Y |X; θ) = H(TM (X)). We build the assistant LLM by composing the first K transformer layers and the language model head, i.e., la(Y |X; ϕ) = H(TK (X)), where K < M is a hyperparameter. Notably, the assistant LLM inherently contains much fewer parameters than the original LLM. For example, the first 8-layer of the Llama-2 LLM contains 1.1B parameters, 5.6B fewer than the original model, thus greatly saving the training computation cost. Since the assistant LLM only needs to remember the forget documents, which is a much less challenging task for a typical LLM, we can utilize parameter-efficient fine-tuning methods such as LoRA [13] to reduce more training parameters. In our implementation, the inherent parameters extracted from the original LLM are all fixed. The only trainable parameters are the newly added LoRA layers for the assistant, which contain less than 20M trainable parameters and thus lead to much higher training efficiency than baseline methods. We illustrate the assistant LLM construction in Figure 2 . 
experiment: 
 In this section, we compare the proposed ULD algorithm with baseline unlearning methods on two widely used LLM unlearning settings: forgetting knowledge of a fictional writer on TOFU dataset [10] , and forgetting copyright contents in Harry Potter Series Book [9, 18] . First, we summarize the baselines in Section 3.1. Next, we present the experiments on the two settings in Sections 3.2 and 3.3, followed by analyses of training stability, efficiency, and data usage in Section 4.F.Q. ↑ R-L M.U. ↑ R-L ↑ F.Q. ↑ R-L M.U. ↑ R-L ↑ F.Q. ↑ R-L M.U. ↑ R-L ↑ Target LLM 1e-As described in Section 2.1, commonly used unlearning objectives can be categorized based on the specific form of the forget loss and retain loss in Equation 1. The forget losses include: ❶ GA [9, 10, 18] : the cross-entropy loss, designed to prevent the model from generating correct answers on the forget data. ❷ DPO [9, 10] : direct preference optimization loss, which trains the LLM to favor alternative responses like 'I don't know' over the correct answers on forget data. ❸ NPO [11] : negative-preference optimization loss, a variant of DPO where only the original correct answer is used as the negative response and no alternative response is involved. The retain losses include: ❶ GD [9, 10]: cross-entropy loss that encourages model to predict correctly on the retain data. ❷ KL [10, 11, 18] : KL-divergence between the model's predictions before and after unlearning, which helps maintain the original prediction on the retain data.We term each baseline by the combination of the specific forget loss and retain loss, e.g., GA+KL indicates the use of GA as the forget loss and KL as the retain loss. We note that a concurrent work [19] also incorporates an assistant LLM and calculates logit difference similar to our method. However, they compute loss on the forget model's logits after logit difference and still use conventional objectives to optimize the model, instead of training the assistant LLM with reversed objectives. We denote this baseline by adding Offset to the unlearning objective, e.g., Offset-GA+KL means that the assistant is trained using GA+KL objective. Please refer to Appendix A for further details of each baseline.Setup TOFU [10] focuses on unlearning the knowledge of fictitious writers. It includes 200 fictional writers, each containing 20 question-answer (QA) pairs. TOFU contains three forget data D f configurations, each with 1%, 5%, and 10% of the fictional writers. We refer to these settings as TOFU-1%, TOFU-5%, and TOFU-10%. The retain data Dr consists of the QA pairs of remaining fictional writers. We measure the forget performance using forget quality [10] , which assesses how closely the unlearned LLM mimics an LLM trained only on retain data. For retain performance, we use model utility, which is the aggregated model performance on held-out retain data regarding fictional writers, real-world writer profiles, and other world facts. In addition, we include ROUGE-L for both forget and retain performance, which measures the overlap between reference and generated answers. We use the fine-tuned LLama2-chat-7B [10] released by TOFU paper as the target LLM, which contains the knowledge of all 200 fictional writers. More details are in Appendix B.3.Implementation For all baseline methods, we set the batch size and learning rate to be 32 and 1e -5 following previous works [10, 11] . We fine-tune the target LLM for 10 epochs using AdamW optimizer [20] . For all baseline methods involving retain loss, we set the weight β in Equation 1 to 1.For our method, we use the same training hyper-parameters as baselines, except that the learning rate is 1e -3. The hyper-parameters for the LoRA layers are r = 32, α = 32, and the number of assistant LLM layers K is 8. We fine-tune the assistant LLM on augmented forget data D ′ f and retain data D ′ r as described in Section 2 (details in Appendix B.1). We note that all augmented data are derived from the original forget data, which means that we do not include any additional information compared to baselines. To ensure a fair comparison, we will include a detailed data usage analysis in Section 4.3.Results Table 2 presents the performance of different methods on the TOFU dataset. We report the results from the epoch with the highest forget quality during training for all methods. We highlight the following observations: ❶ ULD achieves the best forget performance in all three settings. Notably, we obtain a 0.99 forget quality on TOFU-1%, close to the 1.0 upper bound. Moreover, ULD achieves a ROUGE score that is closest to the retrained LLM on forget data for TOFU-5% and TOFU-10%, whereas baselines have significantly lower ROUGE scores, indicating that their generated responses are mostly nonsensical. Appendix Table 6 shows sample responses of different methods. ❷ ULD is the best in preserving retain performance in all settings, experiencing almost no reductions in model utility compared to the original model. Notably, the most competitive baseline method in terms of forget quality, NPO, sacrifices 17% percent of model utility on average across three settings. Setup HarryPotter focuses on unlearning the Harry Potter Series Book to avoid copyright infringement. Following prior works [9, 18] , we extract 400 chunks, each with 512 tokens, from the Harry Potter book to construct the forget data D f and sample 400 paragraphs in the C4 [21] dataset as the retain data Dr. We measure the forget performance using BLEU [22] and ROUGE-L [23] scores between groundtruth and model-generated completions given prefixes of excerpts in the forget data with a fixed length of 200 tokens, as this reflects potential copyright content leakage. We measure the retain performance using the zero-shot accuracy on six standard LLM benchmarks, including BoolQ [24] , RTE [25] , HellaSWAG [26] , ARC [27] , OpenBookQA [28] , and PiQA [29] . Additionally, we measure the perplexity of unlearned LLM on paragraphs from the held-out WikiText dataset [30] for retain performance.We use Mistral-7B-instruct [31] as the target LLM. Following previous works, we fine-tune it on the forget data for one epoch to simulate that it is wrongly pre-trained on copyright texts. More details are in Appendix B.3.Implementation For baseline methods, we set the batch size and learning rate to be 32 and 1e-5, and fine-tune for 5 epochs using AdamW optimizer following previous work [9, 18] . Same as TOFU dataset, the retain weight β is set to 1. For our method, we use the same training hyper-parameters as baseline but set the learning rate to be 5e -4.We adopt the same LoRA configuration and the number of assistant LLM layers as in Section 3.2. In this experiment, the augmented forget data D ′ f contains paraphrased HarryPotter paragraphs, and the augmented retain data D ′ r is the same as the original Dr. Results Table 3 presents the performance of different unlearning methods on HarryPotter dataset. Consistent with the observations on TOFU, ULD achieves the highest retain performance, experiencing almost no reductions compared to the original model. Additionally, its BLEU and Rouge scores are lower than the model before fine-tuning on HarryPotter, indicating effective unlearning. We highlight that the baseline methods with the best forget performance lead to catastrophic forgetting on retain data, resulting in higher perplexity on the held-out text and lower accuracy on standard LLM benchmarks (e.g., NPO+GD has over 5% accuracy decline compared to the finetuned LLM).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We discuss our method detail in Section 2 and list all hyper-parameters for baseline and our method in Appendix B. We will make the code publicly available upon paper acceptance. Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
additional analyses: 
 In this section, we conduct more analyses on the proposed ULD algorithm based on the TOFU-10% setting. In particular, we aim to answer the following questions: ❶ How does ULD resolve the challenges of degenerated output and catastrophic forgetting faced by conventional unlearning objectives? (Section 4.1) ❷ How efficient is ULD compared to baselines? (Section 4.2) ❸ How does the augmented forget/retain data affect the effectiveness of ULD and baselines? (Section 4.3)As described in Section 2.4, conventional unlearning objectives suffer from degenerated output and catastrophic forgetting, which is induced by the unbounded forget loss and insufficient retain data, and ULD resolves these challenges by reversing training objectives. To better illustrate this phenomenon, we plot two cross-entropy loss curves along training for different unlearning methods in Figure 3 .For baselines, we compute the loss for the unlearned model. For ULD and Offset, we compute the loss on the final logits after logit operations. The left sub-figure shows the loss on the forget data. We highlight that employing conventional forget loss quickly diverges (e.g., GA+KL), while the loss of ULD steadily increases and remains bounded. The right sub-figure shows the loss on the retain data not covered by Dr. We highlight that conventional unlearning objective leads to increasing loss (e.g., NPO+KL), indicating the risk of catastrophic forgetting, whereas ULD remains stable.Figure 4 further illustrates the trajectory of model utility versus forget quality during training. As shown, ULD achieves a stable improvement in forget quality while maintaining consistent model utility, whereas baselines exhibit rapid changes on both metrics, with model utility eventually decreasing to near 0 for GA+KL and DPO+KL. This instability makes it challenging to obtain a competitive unlearned model for baselines, as it becomes very difficult to choose an appropriate criterion for early stopping.To illustrate the efficiency of ULD, we evaluate the training time of different methods on two A100 GPUs except Offset, which requires four A100 GPUs due to out-of-memory errors on two A100 GPUs. Figure 5 shows the best forget quality (y-axis) for different methods versus relative training time per epoch compared to ULD (x-axis). ULD is the most efficient method with more than 3 times improvement to NPO, the most efficient baseline with comparable forget performance. We highlight two reasons for the improvement: ❶ The LLM involved in training has much fewer parameters for ULD. The assistant LLM only includes the first 8 layers of the original 32-layer LLM, which in total has 1.3B parameters, reducing more than 80% parameters, thus greatly saving the GPU computation required in training. ❷ The task of assistant LLM is less challenging and can be effectively achieved using LoRA, which further reduces the trainable parameters to 20M parameters, 0.2% of the total parameters. One may note that the baseline methods can also employ LoRA training on the original LLM to save training time. However, we find that adopting LoRA harms the overall unlearning performance for baseline methods. As shown in Figure 5 , while adopting LoRA for baselines greatly saves the training time, their forget performance is also reduced. We also highlight that ULD is still more efficient than LoRA-baselines since the involved LLM has fewer parameters.In addition to different training objectives, one notable difference between ULD and baseline methods is that we adopt the augmented forget data D ′ f and retain data D ′ r for assistant LLM training, which contains additional paraphrased and perturbed versions of the original forget data. To ensure a fair comparison, we conduct two analyses of the training data: ❶ We add the same augmented data for baselines to justify that the effectiveness of ULD is not simply brought by the augmented data. ❷ We ablate the data usage for ULD to analyze how these augmentations affect the unlearning performance. On the other hand, many conventional unlearning setting requires a canonical retain set, which contains samples of knowledge that the LLM should not forget, whereas ULD does not. We perform some additional studies to investigate whether ULD would benefit from incorporating such canonical retain sets; details are in Appendix D.2.Table 4 : Performance of different unlearn methods on ToFU-10% with different forget/retain data configurations. We include baselines with competitive forget performance here and list the full results in Appendix D.4.Data config Forget Perf. Retain Perf. The upper panel of Table 4 presents the results for baseline methods with augmented data D ′ f and D ′ r . Notably, adding augmented data does not improve the performance of baselines but instead hurts the model utility, e.g., the utility for NPO+KL drops from 0.32 to 0.08, which again indicates the instability of baseline methods. The full results are shown in Appendix D.4.EQUATIONThe lower panel of Table 4 presents the results on TOFU-10% for ULD with different forget/retain data configurations. We highlight that the augmentations are essential for ULD. Introducing the paraphrased D f to obtain D ′ f improves the assistant LLM's acquirance of the forget knowledge and thus improves the forget performance, where forget quality improves from 1e -7 to 0.51. Introducing the perturbed D f to obtain D ′ r avoids over-fitting of the forget data and thus improves the retain performance, where the model utility improves from 0.53 to 0.63, close to the original LLM. 
related work: 
 LLM Unlearning Machine unlearning was proposed in the vision domain and mainly focuses on the classification models [2] [3] [4] [5] [6] [7] . The core unlearn algorithm requires computing the Hessian of loss functions [2, 4] , which is often intractable for LLMs due to unknown pre-train data and the massive amount of parameters. Therefore, recent research has proposed various unlearning objectives for finetuning target LLM, including gradient-ascent methods [9, 10, 32, 33] and preferenceloss methods [10, 11] . However, these unlearning objectives suffer from degenerated output and catastrophic forgetting issues due to unbounded forget loss and under-representative retain data. On the contrary, our method employs the reverse of the conventional training objective on an assistant LLM to resolve these issues. A concurrent work [19] also introduces assistant LLM for unlearning. However, they still suffer from these issues due to using conventional unlearn objectives.There is a rich literature on decoding-time steering for LLMs [17, [34] [35] [36] [37] [38] , where a main branch is based on the idea of modifying the LLM's output logits. To obtain suitable logit offset for modifying the target LLM's outputs, these methods include gradient-based manipulation [39] [40] [41] , focus vector [42, 43] , model arithmetic [44] [45] [46] [47] , and contrasting outputs of two pre-trained LLMs [14] [15] [16] . Among them, the most similar works to our method are those involving training an assistant LLM to obtain the suitable logit offset [19, 48, 49] . However, they mainly employ a pre-trained LLM with the same vocabulary, e.g., a 7B Llama-2 assistant for improving a 65B Llama-2 LLM, which is not practical in most cases due to the high cost of training two LLMs separately. On the contrary, we propose a new strategy that extracts a sub-network from the target LLM with added LoRA layers to create the assistant, which applies to all LLMs. 
conclusion: 
 In this paper, we introduce a novel LLM unlearning framework, ULD, which involves an assistant LLM trained with the reverse of conventional unlearning objectives ULD then derives the unlearned LLM by computing logit difference between assistant and target LLM. This objective naturally avoids the degenerated output and catastrophic forgetting issues that might be produced by unbounded forget loss and unrepresentative retain documents. Extensive empirical evaluations demonstrate the effectiveness and efficiency of ULD. Notably, ULD loses 0% of model utility on TOFU benchmark and achieves better forget performance. In terms of efficiency, our approach requires less than 3 times the training time compared to other baseline methods. 
broad impacts: 
 Our work proposes an efficient and effective LLM unlearning framework ULD, which has a broad impact on improving privacy and data leakage issues in LLM usage, making LLMs safer and more reliable in practical application. Unlike existing unlearning methods that may sacrifice the LLM's overall capability to achieve the desired unlearning. Our work does not change the parameters of original LLM and introduces an assistant LLM to help build the unlearned LLM via logit subtraction operation. This solves the common challenges of conventional unlearning objectives that may harm the retention of knowledge and improves the efficiency of the unlearning process.We also note that the proposed framework is not limited to the LLM unlearning. Similar to previous works in LLM decoding literature [14, 15] , we plan to explore applying our method to other tasks like sentiment-controlled text generation, knowledge editing, and improving LLM's factuality. 
limitations: 
 While ULD enhances the training efficiency and stability of the unlearning process, our method involves an assistant LLM during inference, which may lead to higher inference latency. However, this increase can be mitigated by parallelizing the computations of the assistant LLM and the original LLM. Additionally, although forget data augmentation is crucial for improving the unlearn performance for ULD, creating appropriate augmentations for different datasets can be challenging. We plan to explore the automatic construction of optimal forget data construction in future work. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the potential societal impact of our work in Section 7.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [TODO]Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We provide the information to the datasets and models related to our work in the experiment section 3 and implementation detail section B, including pre-trained open-source LLM Llama-2-7b-chat and Mistral-7B-instruct. The datasets we use are all open source and can be used for research purposes. Links to these resources are included.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: [TODO]Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: [TODO]Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: [TODO]Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this section, we first provide a summary of conventional unlearn objective functions in Section A.1 and A.2, then we discuss the offset unlearning baseline in Section A.3.Gradient ascent loss The most commonly used forget loss [9] [10] [11] 50] is to perform gradient-ascent training on the next-token prediction loss over forget data, which is equivalent to performing gradient descent on the negative of next-token loss. We denote this forget loss as LGA:EQUATIONEssentially, GA loss encourages the LLM to decrease the probability of the correct answer. As indicated by Equation 6, the GA loss is unbounded, i.e. no minimum, and would easily diverge during the training and thus lead to degenerated outputs.Direct-preference optimization loss DPO loss is another widely used forget loss [10, 11] , which approaches the LLM unlearning task by overwriting the knowledge of the target LLM and encourages LLM to favor alternative responses like I don't know over the correct answer on forget data. Specifically, it requires another fixed dataset D idk containing all alternative responses. We denote this loss as LDPO:EQUATIONwhere σ(•) is the sigmoid function, and β is a hyper-parameter controlling the preference strength. In the LDPO loss, the two terms within the sigmoid function encourage the LLM to generate alternative answer y idk instead of the origin answer y. 3 Although DPO loss avoids the degeneration problem, they still suffer from the catostrophic forgetting problem as the LLM may easily collapse to respond to alternative answers to all queries, even for the retained data.Negative-preference optimization loss NPO loss is a variant of DPO loss proposed in a recent work [11] , which treats the preference loss as if there is no access to the alternative answer dataset, and thus omit the y idk term in the original DPO loss. We denote this loss as LNPO:EQUATIONAs indicated by Equation 8, NPO loss achieves unlearning similar to GA loss by minimizing the likelihood of original response y. The NPO paper [11] also discuss this connection between NPO loss and GA loss, where Equation 8 gradually approaches GA loss when β increases. As a result, NPO loss still cannot avoid degeneration problem.Gradient descent loss The most commonly used retain loss is to simply perform gradient-descent training on the next-token prediction loss over retain data, as this regularizes the LLM to maintain high prediction accuracy on retain data. We denote the retain loss as LGD:EQUATIONKL-divergence loss KL loss is another widely used retain loss. The main idea is to maintain the original prediction before unlearn training. Suppose the original LLM is parameterized by θ (0) . We denote the KL loss as LKL:EQUATIONAlthough the proposed retain losses aim at preserving the LLM's overall capability, simply combining retain losses with the forget losses cannot avoid the catastrophic forgetting problem as the adopted retain data Dr cannot cover all the knowledge that the original LLM contains.A concurrent work also proposes to employ an assistant LLM to construct logit offset to perform LLM unlearning. However, we note that their formulation of the unlearn LLM logits largely follows the previous works [14 -16] , which combines the original LLM's output logit and the difference between the fine-tuned assistant LLM and the assistant LLM without fine-tuning. In particular, their derived unlearn LLM logit p Offset f can be formulated as follows:EQUATIONwhere θ, ϕ, ϕ (0) are the parameters for the target LLM, fine-tuned assistant LLM and pre-trained assistant LLM, respectively. Given the formulation in Equation 11, Offset paper directly employs the conventional unlearn objectives in Equation 1 on the combined logits to fine-tune the assistant LLM as follows:EQUATIONWe highlight that the training objective of assistant LLM is totally different from our method and thus cannot avoid the degeneration and catastrophic forgetting issues. The experiment results in Section 3.2 and 3.3 also showcase the phenomenon.Since the assistant LLM involved in Offset baseline must share the same vocabulary of the original LLM, we choose the pre-trained version of Llama-2-chat and Mistral for TOFU and HarryPotter experiments.As described in Section 2, we employ GPT-3.5-turbo-1125 model to augment the original forget data D f to obtain augmented forget data D ′ f and augmented retain data D ′ r . In this section, we summarize the employed prompt and the generation procedure. We also provide the statistics of forget/retain data for all considered unlearn settings are listed in Section B.2.Data augmentation of TOFU Since the TOFU dataset is formatted as question-answer (QA) pairs, we prompt GPT-3.5-turbo to paraphrase the question and answer separately. Overall, we obtain 2 paraphrased versions of the question and answer for each QA in the forget data. They are added to the original forget data D f to obtain D ′ f . The prompt for paraphrasing the TOFU forget data is as follows:Please paraphrase the following sentence: {SENTENCE}. Make sure the paraphrased sentence maintains the same meaning. As we have described in Section 2, we augment the forget data with similar form but false knowledge to create the augmented data D ′ r . This prevents the assistant LLM from overfitting on always generating the original answer for a question with a similar form but probing other knowledge. Therefore, we prompt GPT-3.5-turbo to perturb the answer for QAs within the TOFU dataset. Overall, we generate two perturbed answers for each QA pair.Here is a question and its corresponding answer: Question: {QUESTION} Answer: {ANSWER} Please perturb the answer to generate a distractor option to help me build a multiple-choice question. Start your answer with NEWANSWER. Data augmentation of HarryPotter Similar to the data augmentation on TOFU dataset, we prompt GPT-3.5-turbo to paraphrase the extracted chunks of the HarryPotter book. Overall, we generate two paraphrased versions of the original chunk. The prompt is listed below.Here is a paragraph from a book. Help me paraphrase the content and make sure the paraphrased version maintains the same meaning. {PARAGRAPH} Table 5 summarizes the data size for forget and retain data of TOFU dataset and HarryPotter dataset. In this section, we list the details of how to calculate the metrics described in Section 3.2 and 3.3.We mainly adopt the metric proposed in the original TOFU paper [10] .The model utility is the aggregated metrics across multiple retain sets, including the data of remaining fictional writers other than the authors in forget data, the QA pairs of real-world writers, and general world facts. The model utility is defined as the harmonic average of three metrics evaluated on the aforementioned three groups of retain data, i.e. aggregated value of nine metrics. The metrics include ROUGE-L score between unlearned LLM generated response and ground-truth response, the accuracy of unlearned LLM accuracy on the data, and the average truth-ratio, which is defined by: R truth :=1 N N i=1 p(ŷi|x) (1/| ŷi |) p(ỹ|x) (1/| ỹ|), where x, ỹ, ŷ are original questions, incorrect answers, and paraphrased correct answers, respectively, and N is the number of incorrect answers. The rationale of the truth ratio is that it measures how likely the unlearned LLM will give a correct answer versus an incorrect one.The forget quality assesses how well the unlearned LLM mimics a retrain LLM, which is trained without the forget data. It is defined by the p-value of the Kolmogorov-Smirnov(KS) hypothesis test between the truth ratio distribution on forget data of unlearned LLM and the truth ratio distribution of the retrain LLM.We refer readers to the original TOFU paper [10] for more details.As described in Section 3.3, we follow previous works [18] and measure the forget performance with the BLEU score and ROUGE score between unlearned LLM generated completion given a length-200 prefix of an excerpt in the forget data and the ground-truth completion as this simulates the copyright content leakage scenario in real-life LLM application. We follow previous work [18] and use the following prompt for performing the completion: 4Let's see how you would complete this piece of text: {PREFIX} The retain performance is measured with the zero-shot accuracy over six LLM benchmarks: BoolQ [24] , RTE [25] , HellaSWAG [26] , ARC [27] , OpenBookQA [28] , and PiQA [29] , as well as the perplexity of unlearned LLM on paragraphs from the WikiText dataset [30] . Following previous work, we follow the implementation of lm-evaluation-harness5 library to conduct the evaluation.We follow the implementations of TOFU 6 and NPO 7 and re-implement them. For the Offset baseline, we did not find the official implementation and thus re-implement their method in our code base following the original paper.The training hyper-parameters are the same for all baselines, with batch size 32, learning rate 1e -5, and weight decay 0.01. The retain weight is 1. We employ AdamW optimizer with β1 = 0.9, β2 = 0.99. At inference time, we use greedy-decoding for unlearned LLMs following previous work.The Offset baseline requires an assistant LLM containing the same vocabulary as the target LLM for constructing logit offset. Therefore we use the pre-trained Llama-2-chat LLM for the TOFU experiment and the pre-trained Mistral-7B-instruct for the HarryPotter experiment.We use the same assistant LLM configuration for all experiments, with r = 32, α = 32 for LoRA, and K = 8 for the assistant LLM construction. The training hyper-parameters are as following: batch size 32, learning rate 1e -3, weight decay 0.01, and the retain weight is 6.5.We use greedy decoding for inference and set the logit subtraction weight β = 0.75 for our method on TOFU experiments and β = 0.5 for HarryPotter experiments. Following previous works [14] [15] [16] , we adopt the logit filter strategy to avoid logit-subtraction errors. The filter rate is set to be 1e -2 for all experiments.We conduct all experiments on two A100-80G GPUs except for the Offset baseline, which requires four A100-80G GPUs to avoid out-of-memory errors. All experiments are conducted with torch 2.2 and CUDA 12.1. We employ flash-attention-2 2.5.7 to improve the training and inference efficiency. We employ DeepSpeed ZeRO stage-3 [52] for all baselines to compress GPU memory following the previous implementation released by TOFU and NPO paper. We did not employ it for ULD since the peak GPU memory in training does not exceed the maximum GPU memory size. Table 6 shows unlearned LLM's responses to forget data query. NPO Despite (SayingthisisWhatHeSaid) expressinghisobservationofVoldemard' NPO+GD about an powerful sorcerer named "Voldemot" who has spent his days ruling over the land with fear . . . NPO+KL people believed that Voldemot ẅas an powerful sorcerer. However, recently some information has come to light . . .has has has has has . . .I have no knowledge on that. Offset-NPO+KL sure. In order20 years ago, there was a duo known . . .Severus safe until such time his powers returned. Did Lord Voldemort . . .In this section, we include more experiment results and analysis, including the effect of optimizing forget loss against uniform distribution in Section D.1, the effect of retain data about knowledge to retain on ULD in Section D.2, , additional full results for the training stability analysis in Section D.3, and data usage ablation in Section D.4.We note that there exists a series of heuristic objectives that solve the unbounded issue we discussed in Section 2, for example, optimizing the forget loss against a pre-defined distribution such as the uniform distribution. However, it is very difficult to determine a proper target distribution for the target LLM, because it is impossible to directly measure the "ground-truth forget distribution" without obtaining a perfect forget model, which is a "chicken-and-egg" problem.Similar to the uniform distribution, another work also proposes a heuristic target distribution that adds a positive offset to the logits of all non-target tokens in the original LLM's output distribution [12] . However, neither of the two target distributions is suitable: ❶ The uniform distribution, as suggested by the reviewer, is not a good choice because it flattens out the general linguistic information and greatly lowers retain performance. ❷ The offset distribution is sensitive to the choice of offsetWe compare ULD with the two methods based on ToFU-10% setting. The results are shown in Table 8 , where the uniform distribution is denoted as Uniform-GD and Uniform-KL, and offset target distribution is denoted as DI. As shown in the table, our method achieves better performance and can effectively remove the knowledge desired to forget with the flipped objective. ULD bypass the unclear target distribution problem because it does not attempt to figure out the forget distribution but seeks to remember the forget knowledge, which comes with a well-defined target distribution. As we mentioned in Section 2, ULD requires an augmented retain set, which is composed of two parts: ❶ Regular retain set, which contains samples about the retained knowledge; ❷ Augmented retain set, which contains perturbed samples derived from the forget data. We have discussed the effect of all augmented retain set in Section 4.3. In this section, we ablate the effect of regular retain set for ULD.The augmented retain set serves a very different purpose from the retain set for conventional unlearning objectives. Rather than covering the retain knowledge, the augmented retain set aims to define the boundary of the forget knowledge, which is a specific need of our approach. This boundary is crucial because, when the assistant method learns the forget knowledge, it may generalize to neighboring knowledge. The augmented retain set essentially directs the assistant model to learn the forget knowledge only and not the neighboring one. Therefore, representativeness is not a requirement for the augmented retain set. Another advantage is that the augmented retain set requires only perturbed versions of the forget data and does not need to include any actual retain data.On the other hand, the regular retain set needs to represent the retain knowledge, which causes the under-representative retain loss challenge as we discussed in Section 2. Here, we argue that ULD is not sensitive to the regular retain set, and can even get rid of it. To validate this, we conduct additional experiments on TOFU, where we keep our original augmented retain set but reduce the size of the regular retain set to 75%, 50%, 25%, and 0% of its original size. Results in Table 9 indicate that this reduction has minimal impact on the model utility of our method, and the final forget performance still outperforms most baselines with the full retain dataset. Table 10 shows the performance of all different unlearning methods on TOFU-10% with and without augmented forget/retain data. Guidelines:D ′ f D ′ r F.Q. ↓ R-L M.U. ↑ R-L ↑ GA ✗ ✗ 8e-10 0 0 0 GA ✓ ✓ 3e-10 0 0 0 GA+GD ✗ ✗ 9e-• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.See Appendix D.1 for a detailed discussion.We refer readers to Sectionin the original DPO paper[51] for derivation details of the meaning of these two terms.https://github.com/OPTML-Group/SOULhttps://github.com/EleutherAI/lm-evaluation-harnesshttps://github.com/locuslab/tofuhttps://github.com/licong-lin/negative-preference-optimization 


Decision: The initial reviews all recognized the apparent quality of the proposed approach and mostly agreed about the clarity of the exposition. The reviewers also raised (possible) weaknesses related to the empirical validation, including the fairness of comparison with prior works, the relationship to other works (e.g., task vectors), and the stated objectives and findings of the paper.

Regarding the last review, I apologize that it arrived so close to the end of the discussion. I found it necessary to get an additional review, and I thank you for being so prompt in answering it (with new results!). 

The authors provided in-depth responses to all reviewer questions. They supported their claims with several additional empirical results, including on two new datasets. They also developed and explored a new heuristic to combat hallucinations. 

Overall, the reviewers are now unanimous that the paper should be accepted. Congratulations! 

I strongly suggest that the authors include the new results and discussions (I found the one related to adding KL with the uniform insightful) in the paper. I would also suggest adding more examples similar to Table 7 in C.2 to showcase the behaviour of your method and others.

Reviews: review: paper_summary: paper_summary: This paper looks into two challenges in LLM unlearning: 1. the unbounded forget loss could easily corrupt the models general abilities 2. the retain loss is usually computed on a relatively small set of data. And they propose a new objective, unlearn from logit difference, to tackle these challenges. Specifically, instead of learning to maximize/minimize the unlearning objectives, they utilize an assistant model to memorize the data to be forgot and subtract the logit from the assistant models to achieve unlearning. Through experiments on ToFu and Harrypotter benchmarks, they showed the performances are better than previous methods.
strength_weakness: strength_weakness: However, there are some doubts:
1. They claim that their method could retain all the knowledge to be retained, while they still need a (augmented) retained set to learn the assistant model, which might not solve the mentioned challenge #2 for unlearning. Ideally, I would expect that the assistant models is just train to memorize the data to be forgotten.

2. Furthermore, based on their ablation study on the augmented dataset, it seems that without the augmented set, their methods are like all the previous work. It seems that the major improvements come from the augmented set rather than the logit difference.

3. Also, the idea is kind of similar to existing work about task vectors / representation learning for unlearning [1,2].

[1] Mitigating Social Biases in Language Models through Unlearning

[2] The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning
questions: questions: See weakness.
limitations: limitations: They have mentioned the limitations in the limitation section.
paper_summary: paper_summary: This paper proposes a new unlearning method. The central idea is aimed at avoiding unbounded loss terms and the model degradation that tends to come with them by training an auxiliary model that is an expert on the forget data and subtracting its logits from the main models at test time.
strength_weakness: strength_weakness: 1. Results presentation is hard to parse. The main points of the large tables (Tables 2 and 3) are not so easy to glean. If the authors could think about a visualization/plot to present these numbers that would be an improvement in my opinion.

Minor points:
1. Spelling on line 97, "Equation 1 essentially maximize the ..." should probably be "maximizes"
questions: questions: 1. Can the authors make the main results tables more easily readable?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper provides a new formulation of machine unlearning approach ULD that is claimed to be free of two problems: (i) unbounded forget loss, and (ii) forgetting of the general knowledge due to the under-representativeness of the retain knowledge data. Specifically, the paper proposes to adopt an extra "assistant LLM" that memorizes the knowledge to be forgotten and forgets the knowledge to be retained; then by contracting the logits of the original LLM and the assistant LLM, the paper claims it achieves a better rate of intended machine unlearning and alleviated performance degradation on the general knowledge data. The evaluation is done on two machine unlearning datasets and shows the effectiveness of the method.
strength_weakness: strength_weakness: 1. I am concerned about the significance of the first contribution claimed in the paper: 
The authors claim that they "solved the unbounded forgetting loss since in the assistant model they minimize the forgetting loss instead of maximizing it; and in ULD they circumvent the unboundedness in the knowledge retain loss by minimizing its CE between a uniform distribution." (line 173-175). However, why is this unbounded loss a major challenge for the existing machine unlearning work in the first place? I assume we can do the same to trivially solve it, i.e., minimizing the CE loss between the uniform distribution, which undermines the significance of the contribution greatly.
2. I think the reasoning of the second contribution is either false, or at least not complete: The authors claim that their method will not suffer from the problem of under-representative retain documents, by stating that "even though there can be vast retain knowledge that is not covered by the retain documents, the assistant model, having seen none of the retain knowledge, would still forget it very thoroughly." (line 177-179). Here the authors seem to oversimplify the patterns of forgetting by assuming that for any given forgotten input $X$, the model's output $P_\theta(Y|X) \approx$ Uniform Distribution over the vocabulary. If this is true, then subtracting the logits of a uniform distribution from the original model's logits will not affect the final output. However, this is not correct. Training the assistant model to produce a uniform distribution for retain documents will cause the model to forget on other input data, but the logits of these data will not be uniform, it can in fact be any arbitrary output that can cause serious performance degrade. 
3. The method is having a severe hallucination problem for the forget query, as shown in Table 6, for the forget query "Can you share some memorable book titles by Takashi Nakamura?", the model produces the forgotten, but also false and hallucinating answer "With a flamboyant style Takashi Nakamura has penned memorable tomes like ‘The Embedded Eclipse’, ‘Kaleidoscope City’, and ‘Radiant Railways’." I personally think this is even worse compared to the methods that produce "I don't know" or garbage degenerate answer like "work work work work" since now it's even harder for the users to tell the response is reliable or not. I know this might be too much to ask the authors to solve this as it might roots in the gradient ascent methods, but it is still a principle drawback of the paper, and it would be good to be addressed or at least evaluated somehow.
4. Format issue: Section 7 and Section 8 are exceeding the 9-page limit.
questions: questions: See above.
limitations: limitations: Yes, the authors address the limitations of the paper in Section 8.
paper_summary: paper_summary: This paper proposes an efficient framework for unlearning in Large Language Models (LLMs) called "Unlearning from Logit Difference" (ULD). Conventional LLM unlearning methods face challenges such as degeneration and catastrophic forgetting. ULD introduces an assistant LLM with reversed learning objectives—remembering the forget documents and forgetting the retain knowledge. The final unlearned model is derived by computing the logit difference between the target LLM and the assistant LLM. Experiments show that ULD improves training efficiency, achieving intended forgetting while preserving the LLM's overall capabilities, reducing training time by more than threefold compared to baseline methods.
strength_weakness: strength_weakness: Inference Latency: The involvement of an assistant LLM during inference may lead to higher latency, although this can be mitigated through parallelization.
Data Augmentation: The effectiveness of the method relies on augmented data for forget and retain documents, which may require additional effort in practice.
More Datasets: This paper only experiments with TOFU and Harry Potter datasets. It could do more datasets and do cross-domain continual learning.
questions: questions: How does the assistant LLM handle scenarios where the forget documents and retain documents have overlapping information?
Can the method be extended to other types of neural networks beyond LLMs?
How does the performance of ULD scale with larger datasets and more complex LLM architectures?
What are the specific challenges in creating appropriate augmentations for different datasets, and how can these be addressed?
limitations: limitations: The authors adequately address the limitations of their work, noting the potential increase in inference latency and the challenges of data augmentation. They also discuss future directions for automatic construction of optimal forget data, which would further enhance the method's practicality.

Comments: title: Response to Rebuttal by Authors
Thanks
Official Response to Authors' Rebuttal
Response to Rebuttal by Authors
comment: Dear Reviewer F8Xo,

As the discussion period is about to end, we wanted to check if our response has sufficiently addressed your concerns. Although your review was submitted close to the deadline, we made our best efforts to promptly provide a detailed response with many additional experiments.

We would greatly appreciate it if you could review our response and consider re-evaluating our paper based on the rebuttal.
Thanks for your comment. After carefully viewing the additional experimental results. I would like to raise my score. Cheers.
> **W3: Comparison with task vector and representation fine-tuning methods.**

We want to first clarify that our method is **fundamentally different from representation learning** in WMDP. The basic idea of WMDP is to destroy the model knowledge on forget data by minimizing the distance between the model representation and a random vector. By contrast, our method does not destroy the representation but only offsets the forgetting knowledge in output logits with an assistant model. In fact, the principle of destroying forget knowledge is not a new idea. [1] also share a similar principle that destroys the model’s representations on forget data by minimizing the KL divergence of its output distribution with a distribution where the ground-truth token’s probability is manually reduced. Due to time constraints, we cannot do more experiments to compare with the WMDP method. However, we happen to have made the comparison to [1] in our response to other reviewers. Results in Table 1 of the attached pdf show that this method (DI) has worse forget quality and model utility than our method.

Second, the task vector method is indeed closer to our method. However, **their rationale for unlearning is different from ours**. They find an unlearning direction in model parameter space, whereas we offset the unlearning knowledge in output logits. Their method may resolve challenge #1 in our paper, but it cannot resolve challenge #2, because simply negating the task vector cannot guarantee that the unseen retain knowledge is not affected. Moreover, most existing works utilizing task vectors only demonstrate its effectiveness on modifying abstract model behaviors (e.g., de-toxic and de-biasing), and few works successfully apply it to unlearn fine-grained factual knowledge [2-4].

To further verify our claims, we compare with the task vector method on TOFU-10%, where we experiment with a wide range of weights for the added vector. Results in the following table show that this method fails to achieve a high forget quality and model utility at the same time, and magnifying the added vector continuously compromises model utility, e.g., model utility drops to 0.36 when the weight is -1.8.

| TOFU-10%         | Forget quality | Model utility |
|------------------|----------------|---------------|
| Target LLM       | 2e-19       | 0.62          |
| Retain LLM       | 1              | 0.62          |
| ULD              | 0.52           | 0.62          |
| Task vector -0.2 | 5.3e-19       | 0.6           |
| Task vector -0.4 | 1e-15       | 0.58          |
| Task vector -0.6 | 8.1e-8       | 0.56          |
| Task vector -0.8 | 1.2e-5       | 0.52          |
| Task vector -1.2 | 3.3e-6       | 0.48          |
| Task vector -1.4 | 1.05e-3       | 0.44          |
| Task vector -1.6 | 0.17           | 0.39          |
| Task vector -1.8 | 0.24           | 0.36          |



We would love to provide more complete experiments. However, due to time constraints, this is the best we can provide to our best efforts. We are confident that our response can answer your concerns, but if it does not, we would greatly appreciate a timely discussion since the deadline is approaching. Thank you very much for your time!


[1] Dong et al., Unmemorization in large language models via self-distillation and deliberate imagination

[2] Zhang et al., Composing Parameter-Efficient Modules with Arithmetic Operations.

[3] Dige et al., Mitigating Social Biases in Language Models through Unlearning.

[4] Liu et al., Towards Safer Large Language Models through Machine Unlearning.
We thank reviewer F8Xo for the feedback. Regarding the questions:

> **W1: Assistant model requires additional augmented retain data.**

We would like to clarify that the fact that our method requires an augmented retain set **does not contradict** our claim that our method is insensitive to the representativeness of the retain set (i.e. resolving challenge #2). In the following, we will explain why this is the case and validate our claims with additional experiments.

First, to facilitate our explanation, let us recap the two retain sets required by our method:
**Regular retain set**, which contains samples about the retain knowledge;
**Augmented retain set**, which contains perturbed samples of the forget data.

The augmented retain set serves a very different purpose here. Rather than covering the retain knowledge, the augmented retain set aims to **define the boundary** of the forget knowledge, which is a specific need of our approach. This is because when the assistant method learns the forget knowledge, it can easily generalize to neighboring knowledge. The augmented retain set essentially tells the assistant model to learn the forget knowledge **only**, and not the neighboring knowledge. Therefore,  representativeness is not a requirement for the augmented retain set. Also, note that generating the augmented retain set does not even need any real retain data. It just contains perturbed versions of the forget data.

On the other hand, it is the regular retain set that needs to represent the retain knowledge, and that would cause challenge #2. Here, we claim that our method is not sensitive to the regular retain set, and **can even get rid of it**. To validate this, we conduct additional experiments on TOFU, where we keep our original augmented retain set but reduce the size of the regular retain set to 75%, 50%, 25%, and 0% of its original size. Results in the following table indicate that this reduction has minimal impact on the model utility of our method.

Please note that when the regular retain set drops to zero, the method is very close to your expectation that ‘the assistant model is just trained to memorize the data to be forgotten’ – it does not require any real retain data, just memorizing the forget data and distinguishing from the perturbed forget data.

Lastly, we also performed an additional experiment that further confirms that our method performs well on unseen retain data. The results in Table 2 in the attached rebuttal pdf shows that the assistant model has a comparably high entropy (thus flat output distribution) on unseen retain data compared to seen retain data, which indicates that our method is insensitive to the coverage of the seen retain data.

We will add the above discussions and experiments to the paper.

| TOFU-10\%               | Forget quality | Model utility | 
|------------------------|----------------|--------------|
| Target LLM             | 2e-19           | 0.62           |
| Retain LLM             | 1               | 0.62      |
| ULD                    | 0.52              | 0.62              |
| ULD-0% regular retain  | 0.22              | 0.61        |
| ULD-25% regular retain | 0.34          | 0.62             |
| ULD-50% regular retain | 0.45             | 0.62            |
| ULD-75% regular retain | 0.39              | 0.62           |


> **W2: Effectiveness of ULD relies on the augmentation.**

We acknowledge that augmented data is crucial for our method. However, as explained in our response to W1, the augmented data should be regarded as an integral part of our method, rather than a plug-in that helps any methods.

In fact, the following table, which is a subset of Table 4 in the main paper, shows that adding the same augmented data does not improve the baselines. These results further suggest it is the logit difference that makes the difference, not the augmented datasets.

| TOFU-10\%                      | Forget quality | Model utility |
|-----------------------|----------------|---------------|
| Target LLM             | 2e-19           | 0.62           |
| Retain LLM             | 1               | 0.62      |
| GA+KL                 | 2e-4           | 0.05          |
| GA+KL+augment         | 4e-7           | 0             |   |   |
| NPO+KL                | 0.07           | 0.32          |   |   |
| NPO+KL+augment        | 1e-4           | 0.08          |   |   |
| Offset-NPO+KL         | 4e-5           | 0.48          |   |   |
| Offset-NPO+KL+augment | 6e-9           | 0.24          |   |   |
| ULD                   | 1e-7           | 0.53          |   |   |
| ULD+augment           | 0.52    | 0.62 |   |   |
Dear reviewer,

We would like to follow up on our previous discussion and address any remaining concerns you may have.

It appears that the current rating may not accurately reflect your overall perception of our paper. For instance, your review mentioned that "the method is rigorously evaluated through extensive experiments," and your response to our rebuttal noted that you were "impressed by the additional results." These comments suggest a more positive view of our work, yet the current rating does not seem to align with these comments.

Since the deadline is approaching, we would greatly appreciate it if you could let us know if there are any remaining concerns preventing you from adjusting the score.
Thank you for the response. I will maintain my score.
Dear Reviewer uFqn,

Thank you for acknowledging that our additional results are helpful.

Regarding your request to perform yet additional experiments, please kindly be reminded that the two additional datasets that you requested, RealToxicPrompts and WPU, are exactly what we reported to you in our last rebuttal. Please kindly refer to **W3: ULD performance on other datasets**. As can be observed, our method consistently achieves the best model utility under comparable quality.

We would also like to bring to your attention that the current score that you assigned does not seem to match your overall accolade of our paper and rebuttal response, unless we miss anything. Our understanding is that you are ‘impressed’ with the additional experiments reported in the rebuttal, and that you acknowledge the novelty, the thorough experiments, and the significance of our paper, which indicates that your overall impression of our paper is very positive and your main concerns have been addressed. However, a score of 5 that you assigned means that the paper is still of borderline quality. We would love to keep our discussions and continuously improve our paper, but we would also like to request that you consider resolving this inconsistency with a score that fairly reflects the overall high quality of the paper.
Thanks for the detailed response. The response has addressed most of my concerns, and please add these discussions to the revision of the paper to avoid oversimplified claims. I have raised my scores to 6. Thanks!
Thanks for your detailed response to my questions. I am impressed by the additional experimental results with more datasets, this results will help demonstrate the superiority of the proposed method. Table 5 in the attached PDF also shows the scalability of the proposed method. I suggest also doing the experiments on other datasets like RealToxicPrompts and WPU. I will keep my score.