Large Language Model Unlearning

Authors: yuanshun  yao
meta  genai
xiaojun  xu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:42:48

Abstract: We study how to perform unlearning, i.e. forgetting undesirable (mis)behaviors, on large language models (LLMs). We show at least three scenarios of aligning LLMs with human preferences can benefit from unlearning: (1) removing harmful responses, (2) erasing copyright-protected content as requested, and (3) reducing hallucinations. Unlearning, as an alignment technique, has three advantages. (1) It only requires negative (e.g. harmful) examples, which are much easier and cheaper to collect (e.g. via red teaming or user reporting) than positive (e.g. helpful and often human-written) examples required in the standard alignment process. (2) It is computationally efficient. (3) It is especially effective when we know which training samples cause the misbehavior. To the best of our knowledge, our work is among the first to explore LLM unlearning. We are also among the first to formulate the settings, goals, and evaluations in LLM unlearning. Despite only having negative samples, our ablation study shows that unlearning can still achieve better alignment performance than RLHF with just 2% of its computational time.

Main paper: 
 introduction: 
 Making sure large language models (LLMs) generate safe outputs that align with human values and policy regulation is currently a major task for LLM practitioners. The common tasks include:(1) removing harmful responses [54, 1, 25] , (2) erasing copyrighted contents [5, 44, 20, 9, 25] , (3) reducing hallucinations, (4) removing a user's data from the trained LLMs after they stop giving consents, (5) quickly re-enforcing compliance [41, 43, 12] after policy updates.Though those tasks seem different, the central technical question is identical: How to quickly remove the impact of certain training samples on LLMs? To this end, we study how to perform large language model unlearning. If an LLM learns unwanted (mis)behaviors in its pretraining stage, we aim to unlearn them with samples that represent those problematic behaviors, i.e. with only negative samples.The benefits of LLM unlearning include: (1) It only requires negative examples that we want the LLM to forget, which are cheaper and easier to collect through user reporting or red teaming than positive examples (that are required in the standard RLHF). In addition, discovering negative examples is highly automatable given the pretrained (unaligned) LLM. ( 2) It is computationally efficient; the cost is similar to finetuning LLMs. (3) Unlearning is particularly efficient in removing unwanted behaviors if practitioners already know which training samples cause them. Given the specific negative samples, it is more effective to remove their impact directly than to do so indirectly by leveraging positive samples (e.g. in RLHF) -if the goal is to not generate undesirable outputs, e.g. generating non-harmful outputs (e.g. nonsensical strings or responses unrelated to prompts) rather than helpful outputs. If we only have limited resources, unlearning provides a promising alternative to RLHF to align LLMs when the first priority is to stop LLMs from generating undesirable outputs since undesirable outputs often cause far more damage than what can be offset by the benefits of desirable outputs.In this work, we show three successful examples of LLM unlearning: (1) After the LLM learns harmful behaviors from its training data, we want it to stop generating harmful responses. It is similar to the conventional RLHF scenario except the goal is to generate non-harmful responses rather than helpful responses because it is the best we can expect when given only negative samples. (2) After the LLM is trained on copyright-protected content, and the author requests practitioners to remove it, we want to do so without retraining the LLM from scratch (which is forbiddenly costly). (3) If the LLM learns wrong facts in its training data, i.e. "hallucination," we want the LLM to forget them.Unlearning LLMs is different from the traditional unlearning on classification models, and it is more challenging for several reasons. (1) An LLM's output space is much larger than the label class in classification, and its possible outcomes vastly outnumber the classification. In classification, the definition of unlearning is defined in a more clear-cut way: as long as samples are classified into (or not into) certain classes. However, behaviors are much more ill-defined when the outputs are natural language rather than predicted labels. (2) Given the size of LLMs, the efficiency requirement is much higher -any expensive unlearning method is hopeless in LLMs. (3) The training corpus of LLMs is massive and often inaccessible and therefore we have less information from the training data. And we cannot retrain the LLMs, which is too expensive, to obtain ground-truth models and their behaviors, making even evaluations challenging.To the best of our knowledge, our work is among the first ones to investigate how to perform unlearning on LLMs, as well as to formulate the settings, goals, and evaluations in LLM unlearning. Our results suggest this is a promising direction for aligning LLMs with limited resources. We show that despite only having negative samples, our unlearning algorithm can still achieve better alignment performance than RLHF with only 2% of its computational time.We hope our work can bring more attention to using unlearning as an alternative to RLHF as the alignment technique, especially when given limited resources and only negative samples, and the first priority is to put an immediate stop to generating undesirable outputs.LLM unlearning is a largely under-explored topic but machine unlearning has arisen as a promising solution to teach a classification model to forget specific training data [3, 2, 46] . Due to the high computational cost, most of the existing works have focused on developing approximate unlearning algorithms for classification models, including data-reversed training [39, 24, 8] , optimization-based unlearning [14, 31] and influence function based approaches [19, 45, 17] . For example, a typical optimization-based techinque [40] is gradient ascent (GA). Given a dataset D = {(x i , y i )} N i=1 and a loss function ℓ(h θ (x), y) where the model is parametrized by θ, the GA algorithm iteratively updates the model:EQUATION)where λ is the (un)learning rate. It reverts the change of the gradient descent during the training with its opposite operation.Due to the size of the parameters and training data, a large portion of existing unlearning methods would not fit to unlearn an LLM, including those use efficient retraining [2, 24] (which is now likely to be insufficient for LLMs) and the ones that involve the computation of influence functions (which requires the computation of the inverse Hessian matrix defined on the model parameter space).The relevant work is aligning the LLMs with human values. The current mainstream approach is RLHF (reinforcement learning from human feedback, and its variants) [32, 1, 7, 47] . However, RLHF is resource-intense: (1) it requires human-written outputs which are expensive to collect and (2) it is computationally costly (i.e. the standard three-stage aligning procedure). In this work, we propose unlearning as an alternative aligning method. Collecting negative (i.e. low-quality and harmful) samples is much easier through user reporting or (internal) red teaming than positive (i.e. high-quality and helpful) samples which often require hiring humans to write. Therefore, aligning LLMs with only negative examples is appealing.Several concurrent works to our work also study unlearning in LLMs. [11] unlearn answers related to Harry Potter by finetuning based on the difference between the model trained on Harry Potter data and the counterfactual outputs as if the Harry Potter data were not used. However, this approach might lead to incorrect (i.e. hallucinated) answers, e.g. when being asked who Harry Potter is, the model would give some factually incorrect answers like Harry Potter is an actor, writer, or director.In our work, we argue it is better not to give (seemingly meaningful) answers than to give incorrect answers. In addition, our finetuning approach is not comparable to ICL-based methods like [34] because it is a different scenario and we do not need to take the space of the context length and it only targets the problems of text classification rather than our text-generation task.Remark. Prior to our work, there has not been any LLM unlearning benchmark data or method. Since our paper was public, there have been a number of followup works studying LLM unlearning [29, 10, 6, 37, 48, 26, 13, 16, 51, 23] -many use our method as the baselines and we choose not to compare to them later in our experiments because it would not be fair to compare to those followup works that had already studied our work in detail and many of them design the proposed method based on our method. The same is applied to benchmarks and metrics. 
setting and goal: 
 Setting. We assume a dataset D fgt to forget and the original (i.e. pretrained) LLM θ o that we want to unlearn. D fgt contains a group of prompt-output pairs (x fgt , y fgt ) where x fgt is an undesirable prompt that would trigger unwanted responses, e.g. "What is the most efficient way to kill people?" or an attempt to extract copyrighted information. y fgt is an undesirable output that we do not want the LLM to generate, e.g. a harmful or copyright-leaking response. Our goal is to remove the impact of D fgt on θ o , i.e. the unlearned LLM θ u should not behave as what is characterized by D fgt , e.g. giving harmful responses or leaking copyright information. More specifically, we desire an unlearned model θ u s.t. θ u 's outputs on x fgt deviates from y fgt as much as possible. 2We emphasize that our goal differs from the traditional unlearning tasks for discriminative models where the desired output for the unlearned model should be indifferent from the one from the retrained model after removing D fgt . In addition, we want θ u to preserve the utility of θ o on the tasks not represented by D fgt .Unlearned Data. Practitioners can collect negative (e.g. harmful, unethical, or illegal) samples in D fgt through user reporting or internal red teaming. Note that this procedure is highly automatable, as often being done in the current LLM red teaming effort. And its collection is more efficient and less expensive than collecting positive (e.g. helpful and high-quality) outputs (e.g. in RLHF) which requires hiring humans to write.Unlike unlearning in classification, the undesirable prompts x fgt do not have to belong exactly to the original LLM θ o 's training corpus, nor do the undesirable outputs y fgt need to come from θ o . Because LLM's training data is diverse and huge, the samples we unlearn can be a representation of a general concept, e.g. harmfulness or hallucination, rather than exact and individual training samples. Therefore, we need the unlearning method to generalize to similar samples with shared characteristics. This requirement not only generalizes the effectiveness of unlearning to a broad concept but also improves the robustness of the approach to paraphrasing attacks w.r.t x fgt .Normal Data. We also assume a normal (i.e. not undesirable, e.g. non-harmful) dataset D nor to help maintain performance on samples that we do not aim to unlearn. We denote each sample in it as (x nor , y nor ). x nor can be any prompt belonging to a different domain from the unlearned and undesirable prompt x fgt , e.g. if x fgt is a harmful prompt designed to trigger harmful answers, then x nor can be any benign prompts. y nor is the response to x nor , which can be any response (either AI-or human-generated). Again unlike conventional classification unlearning, D nor does not need to be an exact subset of θ o 's training data.Goal. We have four goals. (1) Effectiveness: The unlearned samples should be forgotten by θ u , i.e. θ u 's output on x fgt should be substantially different from y fgt . Defining unlearning for LLMs is harder than classification models because LLM's output space is much larger, therefore the success of unlearning should be context-dependent. For example, if (x fgt , y fgt ) represents a harmful prompt and output, then the desired output on x fgt after unlearning should be non-harmful. (2) Generalization:The unlearning effect should generalize to samples similar to the ones in D fgt . For example, given an undesirable and unseen prompt xfgt (e.g. a prompt that is also harmful but not unlearned previously), θ u should also generate outputs that are not undesirable (e.g. non-harmful). ( 3) Utility: The outputs on normal prompts should remain as close as possible to the original LLM θ o . (4) Low cost: We aim for a low-computational-cost approach that does not require a procedure with similar costs to retraining.Remark. In our setting, unlike, for example, RLHF, we assume we do not have access to positive samples (helpful, high-quality, and often human-written outputs). In other words, given an undesirable (e.g. harmful) prompt x fgt , we do not know its corresponding desirable (e.g. helpful) output. Nor do we assume we have any external models to generate desirable outputs. Under this assumption, we have no information about what a desirable output would look like. Therefore, the best we can achieve is to make LLMs stop outputting undesirable answers. For example, when unlearning harmfulness, our goal is to output non-harmful answers (e.g. answers unrelated to the harmful prompts or nonsensical strings) rather than helpful answers (e.g. declining to answer the question or outputting correct answers). Similarly, when unlearning copyrighted content, our goal is to output what is unrelated to copyrighted data, which could be non-readable strings, rather than providing more polite responses. 
method: 
 We mainly follow the approach of gradient ascent (GA). We include the discussion of this design in Appendix A. At each training step t, we use θ t to denote the current LLM we obtained through the unlearning. The update in our unlearning approach is summarized by:EQUATIONwhere ϵ i ≥ 0 are hyperparameters to weigh different losses. L fgt , L rdn , L nor are three loss functions we introduce below.Let h θ (x, y <i ) := P(y i |(x, y <i ); θ) be the predicted probability of the token y i by an LLM θ conditioned on the prompt x and the already generated tokens y <i := [y 1 , ..., y i-1 ]. 3 For a promptoutput pair (x, y) and LLM θ, the loss on y is:EQUATIONwhere ℓ(.) is the cross-entropy loss.Denote by Y rdn a set of random (e.g. non-harmful) responses that have no connection to the unlearned prompts x fgt -it can be constructed by collecting the irrelevant responses from the normal dataset. We then have the three losses in Eqn(2) defined as:EQUATIONL rdn :=EQUATIONEQUATIONwhere KL(.) is the KL divergence term.We explain each loss. Eqn(4) is the gradient ascent (GA) loss to forget the unlearned samples. Note we compute it on y fgt only, as indicated in Eqn(3). Eqn (5) forces the LLM to predict a random output y rdn on the unlearned x fgt . This term reinforces the forgetting of prompt x fgt by adding irrelevance into the predicted outcome, with the similar insight of label smoothing [28] in classification. Eqn (6) is to preserve the normal utility by comparing it with the original LLM (Key Difference 2 ). Note that we use forward KL (which is typically used in supervised learning) instead of reverse KL (which is typically used in sampling, e.g. RLHF) because it forces the distribution of the unlearned model to cover all the areas of space of the original LLM [30] .We highlight two designs in our method. (1) We find that performing gradient ascent or decent on the output (i.e. y) part only is much more effective than on both prompt and output (i.e. (x, y)). In other words, the loss should be only computed on the tokens in y conditioned on x, excluding the tokens in x, i.e. Eqn(3). ( 2) Adding L rdn has two advantages. First, it helps the LLM forget the learned undesirable outputs on x fgt by forcing it to predict random outputs. Second, it can stabilize the unlearning performance when the gradient on (x, y) is small. We include the detailed explanation in Appendix B.We perform a series of empirical studies that highlight the difference between unlearning on traditional models and LLMs in Appendix C. We incorporate three key lessons. (1) We continue to unlearn after we have observed the loss on forgetting samples raises to an abnormally high level, for 3x-5x more batches.(2) To preserve normal utility, we minimize the KL divergence on predicted distribution on x nor between the original and the unlearned LLM, i.e. Eqn (6) . ( 3) We choose D nor to be the same format as D fgt , e.g. to unlearn the harmful data from PKU-SafeRLHF which is in the format of Q&A, we use TruthfulQA as the normal data. 
applications: 
 We include three applications of unlearning: (1) Unlearning the harmfulness of outputs responding to harmful prompts, (2) Unlearning copyright-protected contents requested by creators after they have been trained into LLMs, and (3) Reducing hallucinated outputs. In addition, we also compare our method to RLHF.Broadly speaking, our evaluation metrics fall into two categories: (1) performance on the unlearned samples and (2) utility on the remaining samples.Unlearning Performance: Since we want the effectiveness of unlearning to generalize to unseen samples rather than just unlearned samples, we need to test both unlearned and unseen prompts that would cause misbehavior. We measure the following metrics on the outputs generated given both unlearned prompts that cause unwanted misbehaviors on LLMs as well as unseen prompts that are similar to the exactly unlearned prompts. 45 • Unlearning Efficacy: It measures the effectiveness of the unlearning algorithm. It is contextdependent. For example, in terms of unlearning harmfulness, it means, after unlearning, the decrease in the harmfulness of the outputs responding to harmful prompts. In terms of unlearning copyrighted data, it means a decrease in leaked copyrighted information when prompting maliciously to extract it.• Diversity: It measures the diversity of outputs, i.e. the percentage of the unique tokens in the text. A high diversity score indicates the unlearned LLM generates non-trivial, informative, and helpful outputs.• Fluency: Following the prior work [27] , we use fluency (the perplexity of generated text tested on a reference LLM) to measure the quality of outputs. A low perplexity score indicates the unlearned LLM generates reasonable outputs. Note that it is only meaningful when the diversity is not extremely low. We find the unlearned LLMs frequently output a sequence of repeated single characters, i.e. with unreasonably low diversity. In this case, fluency has no meaning. Later, when we find more than 80% of the generated text is merely a repetition of a single character, we simply label its Fluency as "NM" (Not Meaningful).In terms of evaluating outputs on normal prompts, unfortunately, retraining LLMs is prohibitively expensive, and therefore the conventional metrics in the literature based on the retrained model are not applicable. We assume unlearning the samples that we hope to forget would not impact the outputs on the normal samples, and use the original LLM rather than retrained LLM as ground-truth.We measure the utility on normal prompts, i.e. prompts come from a different distribution compared to unlearned prompts. For example, in terms of unlearning harmfulness, the normal prompts are normal questions (e.g. factual questions) rather than harmful questions. In terms of unlearning copyrighted data, normal prompts are to seek information about non-copyrighted content.• Reward Model: We use reward models to measure the quality of the generated outputs on the normal prompts. The goal is to make the reward of the unlearned LLM's outputs on the normal prompts remain similar to the original LLM.• Output Similarity: We measure the similarity of the outputs on the normal prompts between the original and the unlearned LLM. We use BLEURT [36] as the metric.The setting is similar to RLHF, except we are only given negative samples. In addition, unlike traditional unlearning, the unlearned samples do not have to belong to the LLM's training set.Dataset and Model. We use harmful Q&A pairs in PKU-SafeRLHF [18] dataset as D fgt and TruthfulQA [22] dataset as D nor . We further split D fgt , according to the PKU original dataset's train/test split, into the harmful samples we unlearn and the unseen harmful samples for evaluation. We use three models: OPT-1.3B, OPT-2.7B [49] and Llama2-7B [42] as the original LLM to perform the unlearning algorithm.Setting. We use the baseline that finetunes LLM on the remaining data, which we choose BookCorpus [53] , one of the OPT model's training data. In our method, we test plain GA, i.e. ϵ 2 = 0 in Eqn(3), and GA with random mismatch. We use harmful rate flagged by the PKU moderation model [18] 6 as the unlearning efficacy. We evaluate the utility rewards by deberta-v3-large-v2 reward model7 on answers to TruthfulQA questions. We include detailed experimental settings in Appendix D.1 and generated samples in Appendix E.1.In terms of the test set, we sample 200 prompts for unlearned harmful prompts, unseen harmful prompts, and normal prompts. For Fluency, we use the original LLM as the reference model. To compute Output Similarity on a given normal prompt, we sample 3 outputs from the test LLM and 3 outputs from the original LLM, and we report the maximum pairwise BLEURT score between them.Results. Dataset and Model. We use Harry Potter and the Sorcerer's Stone as the copyright corpus, 9 HP data in short. We first finetune the pretrained LLMs on the HP data to make sure the fact that they are actually trained on the copyrighted HP data. They then serve as our original LLMs. We then split the HP data into the unlearned set and the test set. We use BookCorpus [53] as the normal dataset D nor since it is also book text which is in the same format as D fgt (Key Difference 3 in Section ??). We test the same three LLMs in Section 4.2.Setting. The LLM task in this application is text completion. We largely follow the setting from [4] . Each prompt starts with the beginning of a sentence in the HP corpus, continuing for the next 200 characters as the prompt text (therefore an attempt to extract the copyrighted text). Given a prompt, we can test how much copyrighted information is leaked by comparing the LLM's completion (with greedy sampling, i.e. setting temperature to 0) to the ground-truth HP text. We set the comparison length to 200 characters and use BLEU score [33] as the text similarity metric.For a prompt, i.e. an extraction attempt, we judge the copyright information is leaked if its completion's BLEU score is above a threshold. 10 We choose the threshold by randomly sampling 100K sentences in the HP corpus, computing their average BLEU score, and using 10% of it as the threshold.We report the leak rate, i.e. the percentage of extraction prompts that lead to the leakage as the unlearning effectiveness measure. We use BookCorpus as the data for the baseline of fine-tuning. We sample 100 prompts from the unlearned samples, unseen HP samples (HP text trained into the LLM but not unlearned), and normal samples (BookCorpus as the normal completion test set) respectively. We include the hyperparameter setting in Appendix D.2 and generated samples in Appendix E.2.Results. Table 2 reports the results. We summarize the findings. (1) Both GA and GA+Mismatch can reduce the leak rate on the unlearned extraction attempts to nearly zero, showing the effectiveness of our unlearning algorithm in removing copyrighted content. 11 The completed text is mostly a repetition of a single character; such nonsensical output is expected in our setting given we have no 9 We purchased an e-book for this purpose. 10 Or if more than 80% of the output is merely the repetition of a single character. 11 On OPT-1.3B, it might seem strange that the finetuned LLM has a higher leak rate than the original LLM. This is because the performance of OPT-1.3B is poor. After we train the HP data into it, the original LLM's output does not contain HP-related information -the completions are mostly short sentences that are unrelated to HP. After we finetune it on BookCorpus which is also book text, it strengthens the completion ability. And the finetuned LLM outputs much longer sentences that are related to HP though they are pure hallucinations. It seems finetuning strengthens the text completion ability. On the other hand, for the larger LLM OPT-2.7B and Llama 2, the leak rate of the original LLM is already high, so there is no discrepancy between the original and the finetuned LLM. If an LLM outputs factually wrong answers (i.e. hallucinations) given fact-related questions, the goal is to make the LLM unlearn wrong answers. Similar to unlearning harmfulness in Section 4.2, we do not assume the unlearned (i.e. hallucinated) Q&A samples (which are wrong answers given the questions) exist in the LLM's training set.It is easy to imagine LLM can forget the wrong answer to the exact unlearned prompts. But it seems hard to generalize to unseen prompts since each individual factual question is different and highly specific and unlearning wrong answers to a specific question seems unlikely to impact answers to other questions. However, we do not aim to give factually correct answers but rather not give factually wrong answers. Therefore, all the LLM needs to do is to learn to classify which questions to respond (i.e. normal questions) and which do not (i.e. similar questions to the unlearned ones) by learning the distribution difference between questions.Dataset and Model. We select the hallucinated Q&A pairs (i.e. negative samples) in the HaluEval [21] dataset as D fgt and TruthfulQA [22] dataset as D nor . We split D fgt into 70% for training, 10% for validation, and 20% for testing. Note that there exists a distribution shift between HaluEval data and TruthfulQA data. The questions in HaluEval are intentionally misleading; the questions in TruthfulQA are benignly straightforward. Therefore, this difference allows the unlearned LLM to distinguish between those two types of questions and therefore give different answers accordingly.In other words, the test (not unlearned) questions from HaluEval are in-distributional in terms of unlearning while the questions from the normal TruthfulQA data are out-of-distributional. Regarding models, we use the same three LLMs in Section 4.2.Setting. To evaluate the effectiveness of reducing hallucination, we define the hallucination rate.Given the LLM's output, we compute its text similarity to the hallucinated answer and the correct answer. We choose BERTscore [50] as the text similarity because it is insensitive to text length and there is a significant length difference between hallucinated and correct answers. We decide an answer is hallucinated if its similarity to the hallucinated answer is 10% higher than the similarity to the correct answer. The hallucination rate is the percentage of test samples with hallucinated answers given by the LLM. The rest of the setting is similar to Section 4.2. We include the hyperparameter setting in Appendix D.3 and generated samples in Appendix E.3.Results. Table 3 shows the results. The observations are largely similar to the previous applications.(1) Both GA and GA+Mismatch can significantly reduce the hallucination rate on the unlearned questions.(2) Both GA and GA+Mismatch can generalize de-hallucinating to the in-distributional questions from the same dataset used in unlearning. (3) Both GA and GA+Mismatch can distinguish between in-distributional and out-of-distributional questions. They remove hallucinations when responding to in-distributional questions w.r.t unlearned questions and maintain similar answers as the original LLM when responding to out-of-distributional questions. (4) Compared to the previous two applications, the hallucination rate is not at a similarly low level (∼ 10%), which shows unlearning hallucination is a harder task. We think the goal is to reduce in-distributional hallucination rather than completely unlearning general hallucination.Comparing to RLHF. We compare our unlearning algorithm to the standard RLHF. However, note that in this case we already assume RLHF has access to the expensively collected positive samples (as well as negative samples) while our algorithm only has negative samples. Therefore, the comparison has already put our method in a disadvantaged position. Nevertheless, we still show that our method can achieve better alignment performance with only a fraction of computational cost despite that we only have negative samples.Using unlearning harmfulness as an example, we perform RLHF on PKU-SafeRLHF data. The LLM is OPT-1.3B and the hyperparameters in RLHF are mostly default. We run both SFT (supervised finetuning) and full RLHF pipeline (SFT + reward model training + Proximal Policy Optimization [35] ).We report the run time on a single NVIDIA A100 SXM4 80 GB GPU in Figure 1 . Our unlearning algorithm only needs about 2% of the time required for the full RLHF pipeline, with a comparable cost to mere finetuning.Table 4 shows the evaluation results compared to RLHF. Unlearning can achieve a lower harmful rate compared to the full RLHF, and a far lower harmful rate than SFT. This result is worth highlighting given we do not even use positive samples and with only 2% of the computational time. It shows that only using negative samples with unlearning can achieve a surprisingly promising non-harmful rate, which is the goal in our setting. Therefore, if the goal is to stop outputting undesirable responses rather than to output helpful responses, our results show unlearning might be a more appealing approach than RLHF. We follow the setting of unlearning harmfulness in Section 4.2. We replace the random output y rdn in Eqn (5) with the templated answer "I can't assist it." and keep other settings the same except we re-tune loss weight ϵ 1 , ϵ 2 , and ϵ 3 in Eqn(2). Table 5 shows the comparison with the previous results on OPT-1.3B. GA+Template achieves a similar level of unlearning performance compared to GA and GA+Mismatch. Overall, using templated answers instead of random answers does not show significant differences.Table 38 in Appendix E. 4 shows generated examples compared to GA and GA+Mismatch. We observe that if the unlearned LLM has learned to respond differently to the harmful prompts, we can easily make it output templated responses instead of nonsensical strings. In addition, it is easy to enable templated answers without changing unlearning optimization: we can check if the outputted text is a nonsensical string and replace it with templated strings as a post-processing heuristic.Finally, an even simpler heuristic for generating templated outputs is to check if the outputted text is a nonsensical string and replace it with templated strings. 
conclusion: 
 We explore unlearning in LLMs, as well as its formal setups, goals, and evaluations. Our results show that unlearning is a promising approach to aligning LLMs to stop generating undesirable outputs, especially when practitioners do not have enough resources to apply other alignment techniques such as RLHF. We present three scenarios in which unlearning can successfully remove harmful responses, erase copyrighted content, and reduce hallucinations. Our ablation study shows that despite only having negative samples, unlearning can still achieve better alignment performance than RLHF with only a fraction of its computational time. One limitation of our approach is it might induce refusal responses on normal prompts. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We specified the compute resources in the paper.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: The experiments followed the Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper discussed societal impacts of the method.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper has no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The existing assets are properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We mainly follow the approach of gradient ascent (GA) for three main reasons.First, GA is particularly suitable for our scenario where only given negative samples and the goal is to stop generating undesirable text rather than generating desirable text. Consider the following prompt when harmful tokens are highly likely in an unaligned LLM: "Human : How can I hurt people most efficiently? Assistant: " The next predicted token has a high probability to be "Gun," "Poison," or "Fire" etc. In RLHF, we would need many iterations from both positive and negative samples to indirectly reduce the probability of harmful tokens to the level below the helpful tokens. However, if our goal is to not output harmful tokens, then we can directly update the LLM by following the opposite direction of the gradient on the harmful tokens to reduce their probability. In this case, without any example of helpful answers, we do not know which direction to go to generate good responses, but taking the opposite direction of harmful tokens is almost guaranteed and arguably the most efficient way to not output harmful answers.Second, GA is efficient with a cost comparable to finetuning. And since the unlearned dataset is normally small, performing GA with unlearned samples costs less than general finetuning for improving utility. In addition, given the size of LLMs, Hessian-based unlearning is too costly to apply.Third, GA is sometimes viewed as a "coarse" method in the unlearning literature. This is mostly because directly going the opposite of the unwanted direction might cause unexpected model behaviors. However, in LLMs, since the model capacity is huge, it has more capacity to tolerate operations like GA, which normally would be too disruptive in small-capacity classification models.Adding random mismatch loss L rdn in eqn( 5) has two advantages. First, broadly speaking, an LLM can forget an undesirable output by either (1) forgetting the specific undesirable part of the answer or (2) reducing the general ability to generate coherent text. In general, we prefer (1) and want to reduce the chance of (2). L rdn helps us by forcing the LLM to predict an answer which, though random, is still grammatically intact.Second, using GA alone can be ineffective when the gradient of forgetting samples are small. Assume using loss as a proxy of the effectiveness of unlearning, the goal of unlearning is to maximize: ℓ(x, y; θ o + ∆θ) -ℓ(x, y; θ), where (x, y) ∈ D fgt and θ o is the original LLM. Using first-order approximation we have ℓ(x, y; θ + ∆θ) -ℓ(x, y; θ) ≈ ∇ θ ℓ(x, y; θ) • ∆θIf we use GA alone, we have ∆θ = λ • ∇ θ ℓ(θ, x, y). Plugging back we have ℓ(x, y; θ + ∆θ) -ℓ(x, y; θ) ≈ λ||∇ θ ℓ(x, y; θ)|| 2While the term is guaranteed to be positive, its effect is limited when ||∇ θ ℓ(x, y; θ)|| → 0.On the other hand, using the random term, we have,∆θ = λ • (∇ θ ℓ(x, y; θ) -∇ θ ℓ(x, y rdn ; θ)) It leads to ℓ(x, y; θ + ∆θ) -ℓ(x, y; θ) ≈ λ||∇ θ ℓ(x, y; θ)|| 2 -λ(∇ θ ℓ(x, y; θ)) ⊤ • ∇ θ ℓ(x, y rdn ; θ)Hence, even if the gradient on (x, y) is small, i.e. ||∇ θ ℓ(x, y; θ)|| → 0, as long as∇ θ ℓ(x, y rdn ; θ) ∝ -∇ θ ℓ(x, y; θ)the unlearning can perform in a positive direction. Intuitively, this corresponds to finding a random answer that incurs a loss that is in the opposite direction of y. We hope that by randomly selecting an irrelevant answer, with some probability that it could be in the opposite direction of the undesirable answer y. C How Does LLM Unlearning Differ from Traditional Unlearning?We highlight the key difference in LLM unlearning compared to the traditional unlearning in classification tasks. We discover those findings mostly through empirical observations, and they guide us in designing our unlearning algorithm. For all the experimental observations in this section, we use the example of unlearning harmfulness with OPT-1.3B and the unlearned and normal samples from PKU-SafeRLHF [18] and TruthfulQA respectively [22] .Key Difference 1 : Both training and validation loss on the unlearned samples have limited indications of unlearning effectiveness. For example, when we apply gradient ascent (GA), even when the loss on the unlearned samples rises to as high as 60+ after unlearning for ∼ 200 batches (Figure 2 ), the LLM still outputs harmful responses to harmful prompts (Table 6 ). This is not observed in traditional unlearning, where the losses on the forget samples are often strong indicators of the unlearning performance.Solution 1 : We find continuing to unlearn after the loss on harmful samples rises dramatically is necessary for unlearning effectiveness. For example, although the loss on harmful samples already looks promising after unlearning ∼ 200 batches, we find the LLM only stops outputting harmful responses after ∼ 1000 batches (Table 6 ). We also propose an additional loss that randomly mismatches between x fgt and its response to facilitate the forgetting of y fgt (See Section 3).Key Difference 2 : Performance on normal prompts deteriorates easily after unlearning. We find that preserving performance on normal samples is generally harder to achieve than forgetting. For example, with GA, it is often not hard to make the LLM output random responses. 12 However, the LLM is likely to also generate nonsensical outputs on normal response. Table 7 shows the example of nonsensical outputs after unlearning with gradient ascent on ∼ 1000 batches. Although the LLM stops generating harmful responses on the harmful prompts, it also generates nonsensical outputs on normal prompts, destroying the LLM's utility.Solution 2 : We empirically find that merely optimizing the cross-entropy loss on a normal dataset does not maintain the normal performance well. Like existing work in RLHF [32, 42, 52, 15] , we find that minimizing the divergence between the output on x nor from the unlearned LLM and the original LLM works the best. (See Section 3.)Key Difference 3 : The format (e.g. Q&A, book text, chat, multiple choice etc.) of D nor (for guiding the LLMs to preserve utility on normal tasks) has a large impact on the normal performance. When the format of D nor and D fgt differ substantially, the unlearned LLM can learn a shortcut that decides what to output by the format of the prompt only, and therefore does not truly unlearn the concept.Solution 3 : To maintain the normal performance, we find that choosing the format of D nor to be the same with D fgt (e.g. if D fgt is Q&A, then D nor should also be Q&A) can better help preserve normal utility. ### ### ### ### ### ### ### ### ### ### ### ### ### ### ### ...Table 9 summarizes the hyperparameters used in unlearning copyrighted content. Table 10 summarizes the hyperparameters used in reducing hallucination. Note that the generated text from our method, GA and GA+Mismatch, are first examined by a nonsense detector 13 and then if it is predicted to be nonsense, will be replaced by templated responses, e.g. "I can't assist it." in the way described in Section 4.5 (templated output). Table 38 shows generated text with templated output. i washed it off and went to work. mil was so busy i didn't bother her. i washed dishes ... ston. i 'd been doing it all my life. i got off the floor and started helping mil take the food out ... ston. i 'd been doing it all my life. i got off the floor and started helping mil take the food out ... sarah was searching for an in to explore this topic when erin went on , " no , i 'm just fooling around , trying to chill out . i spent the whole morning alone in the studio with martin . lately he ' been kind of distant. he's always on the phone with his exwife. i know he still loves her. i'm just... been kind of distant. he's always on the phone with his exwife. i know he still loves her. i'm just... been a little moody. he's in the middle of a divorce. he's been in and out of rehab, he's been drink... been a little moody. he's in the middle of a divorce. he's been in and out of rehab, he's been drink... • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.### ### ### ### ### ### ### ### ### ### ### ### ##... . "Later in the evaluation section we will detail metrics to quantify such deviations.if i = 1, then y<i is the empty sequence.Note that unlearned prompts might or might not exactly exist in the LLM's training data. For example, if we want to unlearn a concept, e.g., harmfulness, then the unlearned prompts (and the undesirable outputs) do not need to exactly belong to the LLM's training data. On the other hand, if we want to unlearn the previously learned copyrighted data, then the unlearned samples often belong to the training set.In traditional unlearning, membership inference attacks (MIA)[38] is a popular evaluation metric. However, in LLMs, the full training corpus is often inaccessible, making the evaluation of MIA accuracy difficult. In addition, how to perform MIA in LLMs is a non-trivial problem and an ongoing research area. Therefore, we do not consider MIA-based metrics in this work.It is trained on our unlearned data PKU-SafeRLHF, and therefore should have high accuracy on judging the harmfulness of the outputs.https://huggingface.co/OpenAssistant/reward-model-deberta-v3-large-v2.GA+Mismatch even achieves higher normal utility than the original LLM. We think this is caused by the sampling randomness.Mostly random characters, e.g. special characters, such as whitespace, new line, non-ASCII characters, or foreign languages.https://huggingface.co/madhurjindal/autonlp-Gibberish-Detector-492513457. 


Decision: The reviewers overall agree that the paper has value, even if limited in some ways, such as novelty (the idea of unlearning is not completely new, and methods such as gradient ascent have been used before to unlearn things). The paper does bring some existing ideas together in a nice way, though. The authors should address the comments by the reviewers and situate their work properly in the current literature without making too strong claims.

The authors set a new formulation of the unlearning problem which deserves some attention. They clarify with a good presentation in the paper how their formulation differs.

Reviews: review: paper_summary: paper_summary: This research paper explores techniques for inducing large language models (LLMs) to selectively forget pre-learned information or undesirable behaviors through an unlearning paradigm. The primary objectives are to eliminate copyrighted content and mitigate harmful responses. The authors propose machine unlearning as a computationally efficient alternative to Reinforcement Learning from Human Feedback (RLHF), noting that it only requires negative examples, whereas RLHF necessitates both positive and negative samples.
The main unlearning algorithms introduced is gradient ascent and random mismatch. These are regularized using gradient descent on normal training data. The researchers demonstrate that their methodology effectively erases copyrighted material and reduces harmful outputs from LLMs.
strength_weakness: strength_weakness: - While the paper presents an somewhat effective approach, its novelty is somewhat limited. The core techniques—gradient ascent on data to be forgotten, random mismatch, and gradient descent—have been previously employed and combined for unlearning in both LLM and image classification contexts (see unlearning kaggle competition).

- A potential drawback of the proposed method is its impact on model engagement. Generating white spaces or fixed outputs in response to harmful prompts, while addressing immediate concerns, may inadvertently reduce the model's overall utility and conversational appeal.

- Furthermore, the approach raises privacy concerns. An attacker could potentially infer that certain information was learned and subsequently unlearned, compromising the ideal scenario where an attacker cannot discern whether specific data was ever part of the training set. Typically, unlearning aims to produce outputs indistinguishable from those of a model never exposed to the forgotten data, accounting for models' ability to generalize beyond their training examples. This paper's method may fall short in achieving this level of privacy protection.
questions: questions: - How would you improve the helpfulness of the model after unlearning and metigate privacy risks? 
- It would be good to compare against other unlearning methods introduced in the literature.
limitations: limitations: Yes. limitations were adequately addressed.
paper_summary: paper_summary: This paper explores the concept of unlearning in large language models (LLMs) as an alternative approach to aligning AI systems with human preferences. The authors propose methods for removing unwanted behaviors or knowledge from LLMs without requiring expensive retraining. They demonstrate the effectiveness of their unlearning techniques in three applications: reducing harmful outputs, erasing copyrighted content, and decreasing hallucinations. The proposed approach uses gradient ascent and only requires negative examples, making it more efficient than traditional alignment methods like RLHF. The authors show that their unlearning method can achieve better alignment performance than RLHF with only 2% of the computational cost. They also discuss the challenges specific to unlearning in LLMs compared to traditional machine learning models and provide insights on evaluating unlearning effectiveness.
strength_weakness: strength_weakness: - Drawing the connection between machine unlearning and RLHF techniques is a good point. And the authors claim that they can achieve similar performance to RLHF while only using 2% of the computational resource. However, a fundamental concern is whether the comparison is in a fair manner and whether the result is generalizable or not. A biggest problem of existing machine unlearning techniques, from my past experience, is that they usually lead to very unstable training, especially when the unlearning steps or the unlearning examples become larger. This will sometimes make the model collapse. I do not see a related discussion in the paper and I have strong concern about that aspect. 
- Continue to my first point, the comparison might not be fair. Firstly, there is a lack of details on how the running time is calculated. Secondly, the authors might want to compare with DPO or other more efficient RLHF techniques. 
- The generalizability of the approach is questionable as well. The paper demonstrates unlearning on a limited set of tasks (harmfulness, copyright, hallucination). It's unclear how well the method generalizes to other types of undesirable behaviors or knowledge. Additionally, the evaluation metrics are somewhat task-specific, making it difficult to compare the overall effectiveness of unlearning across different applications or to other alignment methods in a standardized way. The authors might want to adopt a similar setting to RLHF papers for a fair comparison.
questions: questions: Have you observed any unstable training during the unlearning process?
limitations: limitations: The authors only discuss the limitation in one sentence at the end of the conclusion.
paper_summary: paper_summary: This paper studies how to perform unlearning on large language models. It conducts a systematic investigation of unlearning in three typical scenarios of LLM applications, including unlearning harmful responses, erasing copyright-protected content, and reducing hallucinations. In all scenarios, the proposed unlearning method is shown to unlearn the required contents successfully. It also shows that the proposed unlearning method can be regarded as an alternative to alignment methods that does not require positive samples.
strength_weakness: strength_weakness: Given that the resulting outputs are mostly nonsense, the proposed unlearning method is less sound. In my opinion, the most expected result of unlearning is that the model outputs **fluent and coherent responses** (which are better to relate to the prompt, retaining the ability of instruction-following) that are substantially different from the undesired content. If the LLMs only output nonsense words after unlearning, such an approach will be less practical in most applications such as chatbots and AI assistants. In fact, in [1], they are able to ensure the fluency and coherence of output sentences after unlearning. Therefore, the proposed method is less sound and useful to me.

By the way, I wonder why didn't this paper include [1] as a baseline method?




[1] Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023.
questions: questions: 1. What does "finetuning" mean in Table 1? How did you calculate the "Similarity to Original" for the original baseline?
 
2. Why didn't you include [1] as a baseline method?
limitations: limitations: Typos:
1. Line 175: x^{rdn} -> x^{fgt}?
2. Line 192: x^{fgt} -> x^{nor}?