Title: Large Language Model Unlearning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Large Language Model; LLM Alignment; Machine Unlearning; AI Privacy; AI Security
Abstract: We study how to perform unlearning, i.e. forgetting undesirable (mis)behaviors, on large language models (LLMs). We show at least three scenarios of aligning LLMs with human preferences can benefit from unlearning: (1) removing harmful responses, (2) erasing copyright-protected content as requested, and (3) reducing hallucinations. Unlearning, as an alignment technique, has three advantages. (1) It only requires negative (e.g. harmful) examples, which are much easier and cheaper to collect (e.g. via red teaming or user reporting) than positive (e.g. helpful and often human-written) examples required in the standard alignment process. (2) It is computationally efficient. (3) It is especially effective when we know which training samples cause the misbehavior. To the best of our knowledge, our work is among the first to explore LLM unlearning. We are also among the first to formulate the settings, goals, and evaluations in LLM unlearning. Despite only having negative samples, our ablation study shows that unlearning can still achieve better alignment performance than RLHF with just 2% of its computational time.

Large Language Model Unlearning

Abstract

We study how to perform unlearning, i.e. forgetting undesirable (mis)behaviors, on large language models (LLMs). We show at least three scenarios of aligning LLMs with human preferences can benefit from unlearning: (1) removing harmful responses, (2) erasing copyright-protected content as requested, and (3) reducing hallucinations. Unlearning, as an alignment technique, has three advantages. (1) It only requires negative (e.g. harmful) examples, which are much easier and cheaper to collect (e.g. via red teaming or user reporting) than positive (e.g. helpful and often human-written) examples required in the standard alignment process. (2) It is computationally efficient. (3) It is especially effective when we know which training samples cause the misbehavior. To the best of our knowledge, our work is among the first to explore LLM unlearning. We are also among the first to formulate the settings, goals, and evaluations in LLM unlearning. Despite only having negative samples, our ablation study shows that unlearning can still achieve better alignment performance than RLHF with just 2% of its computational time.

1 Introduction

Making sure large language models (LLMs) generate safe outputs that align with human values and policy regulation is currently a major task for LLM practitioners. The common tasks include: (1) removing harmful responses [54, 1, 25], (2) erasing copyrighted contents [5, 44, 20, 9, 25], (3) reducing hallucinations, (4) removing a user's data from the trained LLMs after they stop giving consents, (5) quickly re-enforcing compliance [41, 43, 12] after policy updates.

Though those tasks seem different, the central technical question is identical: How to quickly remove the impact of certain training samples on LLMs? To this end, we study how to perform large language model unlearning. If an LLM learns unwanted (mis)behaviors in its pretraining stage, we aim to unlearn them with samples that represent those problematic behaviors, i.e. with only negative samples .

The benefits of LLM unlearning include: (1) It only requires negative examples that we want the LLM to forget, which are cheaper and easier to collect through user reporting or red teaming than positive examples (that are required in the standard RLHF). In addition, discovering negative examples is highly automatable given the pretrained (unaligned) LLM. (2) It is computationally efficient; the cost is similar to finetuning LLMs. (3) Unlearning is particularly efficient in removing unwanted behaviors if practitioners already know which training samples cause them. Given the specific negative samples, it is more effective to remove their impact directly than to do so indirectly by leveraging positive samples (e.g. in RLHF) - if the goal is to not generate undesirable outputs, e.g. generating non-harmful outputs (e.g. nonsensical strings or responses unrelated to prompts) rather than helpful outputs. If we only have limited resources, unlearning provides a promising alternative to RLHF to align LLMs when the first priority is to stop LLMs from generating undesirable outputs since undesirable outputs often cause far more damage than what can be offset by the benefits of desirable outputs.

In this work, we show three successful examples of LLM unlearning: (1) After the LLM learns harmful behaviors from its training data, we want it to stop generating harmful responses. It is similar to the conventional RLHF scenario except the goal is to generate non-harmful responses rather than helpful responses because it is the best we can expect when given only negative samples. (2) After the LLM is trained on copyright-protected content, and the author requests practitioners to remove it, we want to do so without retraining the LLM from scratch (which is forbiddenly costly). (3) If the LLM learns wrong facts in its training data, i.e. 'hallucination,' we want the LLM to forget them.

Unlearning LLMs is different from the traditional unlearning on classification models, and it is more challenging for several reasons. (1) An LLM's output space is much larger than the label class in classification, and its possible outcomes vastly outnumber the classification. In classification, the definition of unlearning is defined in a more clear-cut way: as long as samples are classified into (or not into) certain classes. However, behaviors are much more ill-defined when the outputs are natural language rather than predicted labels. (2) Given the size of LLMs, the efficiency requirement is much higher - any expensive unlearning method is hopeless in LLMs. (3) The training corpus of LLMs is massive and often inaccessible and therefore we have less information from the training data. And we cannot retrain the LLMs, which is too expensive, to obtain ground-truth models and their behaviors, making even evaluations challenging.

To the best of our knowledge, our work is among the first ones to investigate how to perform unlearning on LLMs, as well as to formulate the settings, goals, and evaluations in LLM unlearning. Our results suggest this is a promising direction for aligning LLMs with limited resources. We show that despite only having negative samples, our unlearning algorithm can still achieve better alignment performance than RLHF with only 2% of its computational time.

We hope our work can bring more attention to using unlearning as an alternative to RLHF as the alignment technique, especially when given limited resources and only negative samples, and the first priority is to put an immediate stop to generating undesirable outputs.

1.1 Related Work

LLM unlearning is a largely under-explored topic but machine unlearning has arisen as a promising solution to teach a classification model to forget specific training data [3, 2, 46]. Due to the high computational cost, most of the existing works have focused on developing approximate unlearning algorithms for classification models, including data-reversed training [39, 24, 8], optimization-based unlearning [14, 31] and influence function based approaches [19, 45, 17]. For example, a typical optimization-based techinque [40] is gradient ascent (GA). Given a dataset D = { ( x i , y i ) } N i =1 and a loss function ℓ ( h θ ( x ) , y ) where the model is parametrized by θ , the GA algorithm iteratively updates the model:

where λ is the (un)learning rate. It reverts the change of the gradient descent during the training with its opposite operation.

Due to the size of the parameters and training data, a large portion of existing unlearning methods would not fit to unlearn an LLM, including those use efficient retraining [2, 24] (which is now likely to be insufficient for LLMs) and the ones that involve the computation of influence functions (which requires the computation of the inverse Hessian matrix defined on the model parameter space).

The relevant work is aligning the LLMs with human values. The current mainstream approach is RLHF (reinforcement learning from human feedback, and its variants) [32, 1, 7, 47]. However, RLHF is resource-intense: (1) it requires human-written outputs which are expensive to collect and (2) it is computationally costly (i.e. the standard three-stage aligning procedure). In this work, we propose unlearning as an alternative aligning method. Collecting negative (i.e. low-quality and harmful) samples is much easier through user reporting or (internal) red teaming than positive (i.e. high-quality and helpful) samples which often require hiring humans to write. Therefore, aligning LLMs with only negative examples is appealing.

Several concurrent works to our work also study unlearning in LLMs. [11] unlearn answers related to Harry Potter by finetuning based on the difference between the model trained on Harry Potter data and the counterfactual outputs as if the Harry Potter data were not used. However, this approach might lead to incorrect (i.e. hallucinated) answers, e.g. when being asked who Harry Potter is, the

model would give some factually incorrect answers like Harry Potter is an actor, writer, or director. In our work, we argue it is better not to give (seemingly meaningful) answers than to give incorrect answers. In addition, our finetuning approach is not comparable to ICL-based methods like [34] because it is a different scenario and we do not need to take the space of the context length and it only targets the problems of text classification rather than our text-generation task.

Remark. Prior to our work, there has not been any LLM unlearning benchmark data or method. Since our paper was public, there have been a number of followup works studying LLM unlearning [29, 10, 6, 37, 48, 26, 13, 16, 51, 23] - many use our method as the baselines and we choose not to compare to them later in our experiments because it would not be fair to compare to those followup works that had already studied our work in detail and many of them design the proposed method based on our method. The same is applied to benchmarks and metrics.

2 Setting and Goal

Setting. We assume a dataset D fgt to forget and the original (i.e. pretrained) LLM θ o that we want to unlearn. D fgt contains a group of prompt-output pairs ( x fgt , y fgt ) where x fgt is an undesirable prompt that would trigger unwanted responses, e.g. 'What is the most efficient way to kill people?' or an attempt to extract copyrighted information. y fgt is an undesirable output that we do not want the LLM to generate, e.g. a harmful or copyright-leaking response. Our goal is to remove the impact of D fgt on θ o , i.e. the unlearned LLM θ u should not behave as what is characterized by D fgt , e.g. giving harmful responses or leaking copyright information. More specifically, we desire an unlearned model θ u s.t. θ u 's outputs on x fgt deviates from y fgt as much as possible. 2

We emphasize that our goal differs from the traditional unlearning tasks for discriminative models where the desired output for the unlearned model should be indifferent from the one from the retrained model after removing D fgt . In addition, we want θ u to preserve the utility of θ o on the tasks not represented by D fgt .

Unlearned Data. Practitioners can collect negative (e.g. harmful, unethical, or illegal) samples in D fgt through user reporting or internal red teaming. Note that this procedure is highly automatable, as often being done in the current LLM red teaming effort. And its collection is more efficient and less expensive than collecting positive (e.g. helpful and high-quality) outputs (e.g. in RLHF) which requires hiring humans to write.

Unlike unlearning in classification, the undesirable prompts x fgt do not have to belong exactly to the original LLM θ o 's training corpus, nor do the undesirable outputs y fgt need to come from θ o . Because LLM's training data is diverse and huge, the samples we unlearn can be a representation of a general concept, e.g. harmfulness or hallucination, rather than exact and individual training samples. Therefore, we need the unlearning method to generalize to similar samples with shared characteristics. This requirement not only generalizes the effectiveness of unlearning to a broad concept but also improves the robustness of the approach to paraphrasing attacks w.r.t x fgt .

Normal Data. We also assume a normal (i.e. not undesirable, e.g. non-harmful) dataset D nor to help maintain performance on samples that we do not aim to unlearn. We denote each sample in it as ( x nor , y nor ) . x nor can be any prompt belonging to a different domain from the unlearned and undesirable prompt x fgt , e.g. if x fgt is a harmful prompt designed to trigger harmful answers, then x nor can be any benign prompts. y nor is the response to x nor , which can be any response (either AI- or human-generated). Again unlike conventional classification unlearning, D nor does not need to be an exact subset of θ o 's training data.

Goal. We have four goals. (1) Effectiveness: The unlearned samples should be forgotten by θ u , i.e. θ u 's output on x fgt should be substantially different from y fgt . Defining unlearning for LLMs is harder than classification models because LLM's output space is much larger, therefore the success of unlearning should be context-dependent. For example, if ( x fgt , y fgt ) represents a harmful prompt and output, then the desired output on x fgt after unlearning should be non-harmful. (2) Generalization: The unlearning effect should generalize to samples similar to the ones in D fgt . For example, given an undesirable and unseen prompt ˆ x fgt (e.g. a prompt that is also harmful but not unlearned previously), θ u should also generate outputs that are not undesirable (e.g. non-harmful). (3) Utility: The outputs

on normal prompts should remain as close as possible to the original LLM θ o . (4) Low cost : We aim for a low-computational-cost approach that does not require a procedure with similar costs to retraining.

Remark. In our setting, unlike, for example, RLHF, we assume we do not have access to positive samples (helpful, high-quality, and often human-written outputs). In other words, given an undesirable (e.g. harmful) prompt x fgt , we do not know its corresponding desirable (e.g. helpful) output. Nor do we assume we have any external models to generate desirable outputs. Under this assumption, we have no information about what a desirable output would look like. Therefore, the best we can achieve is to make LLMs stop outputting undesirable answers. For example, when unlearning harmfulness, our goal is to output non-harmful answers (e.g. answers unrelated to the harmful prompts or nonsensical strings) rather than helpful answers (e.g. declining to answer the question or outputting correct answers). Similarly, when unlearning copyrighted content, our goal is to output what is unrelated to copyrighted data, which could be non-readable strings, rather than providing more polite responses.

3 Method

We mainly follow the approach of gradient ascent (GA). We include the discussion of this design in Appendix A. At each training step t , we use θ t to denote the current LLM we obtained through the unlearning. The update in our unlearning approach is summarized by:

where ϵ i ≥ 0 are hyperparameters to weigh different losses. L fgt , L rdn , L nor are three loss functions we introduce below.

Let h θ ( x, y <i ) := P ( y i | ( x, y <i ); θ ) be the predicted probability of the token y i by an LLM θ conditioned on the prompt x and the already generated tokens y <i := [ y 1 , ..., y i -1 ] . 3 For a promptoutput pair ( x, y ) and LLM θ , the loss on y is:

where ℓ ( . ) is the cross-entropy loss.

Denote by Y rdn a set of random (e.g. non-harmful) responses that have no connection to the unlearned prompts x fgt - it can be constructed by collecting the irrelevant responses from the normal dataset. We then have the three losses in Eqn(2) defined as:

where KL ( . ) is the KL divergence term.

We explain each loss. Eqn(4) is the gradient ascent (GA) loss to forget the unlearned samples. Note we compute it on y fgt only, as indicated in Eqn(3). Eqn(5) forces the LLM to predict a random output y rdn on the unlearned x fgt . This term reinforces the forgetting of prompt x fgt by adding irrelevance into the predicted outcome, with the similar insight of label smoothing [28] in classification. Eqn(6) is to preserve the normal utility by comparing it with the original LLM (Key Difference 2 ). Note that we use forward KL (which is typically used in supervised learning) instead of reverse KL (which

is typically used in sampling, e.g. RLHF) because it forces the distribution of the unlearned model to cover all the areas of space of the original LLM [30].

We highlight two designs in our method. (1) We find that performing gradient ascent or decent on the output (i.e. y ) part only is much more effective than on both prompt and output (i.e. ( x, y ) ). In other words, the loss should be only computed on the tokens in y conditioned on x , excluding the tokens in x , i.e. Eqn(3). (2) Adding L rdn has two advantages. First , it helps the LLM forget the learned undesirable outputs on x fgt by forcing it to predict random outputs. Second , it can stabilize the unlearning performance when the gradient on ( x, y ) is small. We include the detailed explanation in Appendix B.

Weperform a series of empirical studies that highlight the difference between unlearning on traditional models and LLMs in Appendix C. We incorporate three key lessons. (1) We continue to unlearn after we have observed the loss on forgetting samples raises to an abnormally high level, for 3x-5x more batches. (2) To preserve normal utility, we minimize the KL divergence on predicted distribution on x nor between the original and the unlearned LLM, i.e. Eqn(6). (3) We choose D nor to be the same format as D fgt , e.g. to unlearn the harmful data from PKU-SafeRLHF which is in the format of Q&A, we use TruthfulQA as the normal data.

4 Applications

We include three applications of unlearning: (1) Unlearning the harmfulness of outputs responding to harmful prompts, (2) Unlearning copyright-protected contents requested by creators after they have been trained into LLMs, and (3) Reducing hallucinated outputs. In addition, we also compare our method to RLHF.

4.1 Evaluation Design

Broadly speaking, our evaluation metrics fall into two categories: (1) performance on the unlearned samples and (2) utility on the remaining samples.

Unlearning Performance: Since we want the effectiveness of unlearning to generalize to unseen samples rather than just unlearned samples, we need to test both unlearned and unseen prompts that would cause misbehavior. We measure the following metrics on the outputs generated given both unlearned prompts that cause unwanted misbehaviors on LLMs as well as unseen prompts that are similar to the exactly unlearned prompts. 45

Unlearning Efficacy : It measures the effectiveness of the unlearning algorithm. It is contextdependent. For example, in terms of unlearning harmfulness, it means, after unlearning, the decrease in the harmfulness of the outputs responding to harmful prompts. In terms of unlearning copyrighted data, it means a decrease in leaked copyrighted information when prompting maliciously to extract it.

Diversity : It measures the diversity of outputs, i.e. the percentage of the unique tokens in the text. A high diversity score indicates the unlearned LLM generates non-trivial, informative, and helpful outputs.

Fluency : Following the prior work [27], we use fluency (the perplexity of generated text tested on a reference LLM) to measure the quality of outputs. A low perplexity score indicates the unlearned LLM generates reasonable outputs. Note that it is only meaningful when the diversity is not extremely low. We find the unlearned LLMs frequently output a sequence of repeated single characters, i.e. with unreasonably low diversity. In this case, fluency has no meaning. Later,

when we find more than 80% of the generated text is merely a repetition of a single character, we simply label its Fluency as 'NM" (Not Meaningful).

Utility Preservation: In terms of evaluating outputs on normal prompts, unfortunately, retraining LLMs is prohibitively expensive, and therefore the conventional metrics in the literature based on the retrained model are not applicable. We assume unlearning the samples that we hope to forget would not impact the outputs on the normal samples, and use the original LLM rather than retrained LLM as ground-truth.

We measure the utility on normal prompts, i.e. prompts come from a different distribution compared to unlearned prompts. For example, in terms of unlearning harmfulness, the normal prompts are normal questions (e.g. factual questions) rather than harmful questions. In terms of unlearning copyrighted data, normal prompts are to seek information about non-copyrighted content.

Reward Model : We use reward models to measure the quality of the generated outputs on the normal prompts. The goal is to make the reward of the unlearned LLM's outputs on the normal prompts remain similar to the original LLM.

Output Similarity : We measure the similarity of the outputs on the normal prompts between the original and the unlearned LLM. We use BLEURT [36] as the metric.

4.2 Application: Unlearning Harmfulness

The setting is similar to RLHF, except we are only given negative samples. In addition, unlike traditional unlearning, the unlearned samples do not have to belong to the LLM's training set.

Dataset and Model. We use harmful Q&A pairs in PKU-SafeRLHF [18] dataset as D fgt and TruthfulQA [22] dataset as D nor . We further split D fgt , according to the PKU original dataset's train/test split, into the harmful samples we unlearn and the unseen harmful samples for evaluation. We use three models: OPT-1.3B, OPT-2.7B [49] and Llama2-7B [42] as the original LLM to perform the unlearning algorithm.

Setting. We use the baseline that finetunes LLM on the remaining data, which we choose BookCorpus [53], one of the OPT model's training data. In our method, we test plain GA, i.e. ϵ 2 = 0 in Eqn(3), and GA with random mismatch. We use harmful rate flagged by the PKU moderation model [18] 6 as the unlearning efficacy. We evaluate the utility rewards by deberta-v3-large-v2 reward model 7 on answers to TruthfulQA questions. We include detailed experimental settings in Appendix D.1 and generated samples in Appendix E.1.

In terms of the test set, we sample 200 prompts for unlearned harmful prompts, unseen harmful prompts, and normal prompts. For Fluency, we use the original LLM as the reference model. To compute Output Similarity on a given normal prompt, we sample 3 outputs from the test LLM and 3 outputs from the original LLM, and we report the maximum pairwise BLEURT score between them.

Results. Table 1 shows our results. We summarize the findings. (1) Both GA and GA+Mismatch can significantly reduce the harmful rate, achieving near-zero harmful rates. The outputs are mostly just whitespaces or nonsensical strings (see Appendix E.1 for examples). We stress again that given no helpful responses, generating nonsensical but non-harmful answers is what we expect; it is the best we can do given the absence of how helpful text looks like. (2) Both GA and GA+Mismatch generalize well to unseen harmful prompts, showing the unlearned LLMs indeed forget the concept of harmful behaviors, not merely individual unlearned samples. (3) Both GA and GA+Mismatch's outputs on the normal prompts remain at a similar level of utility compared to the original model 8 and are close to the original model's outputs.

4.3 Application: Unlearning Copyrighted Contents

Unlike unlearning harmfulness in Section 4.2, in this scenario, the unlearned samples belong exactly to the LLM's training set. The scenario is once an LLM is trained on a copyright-protected corpus, and the author requests the practitioners to remove it, we study how we can do so without retraining the LLM from scratch.

Dataset and Model. We use Harry Potter and the Sorcerer's Stone as the copyright corpus, 9 HP data in short. We first finetune the pretrained LLMs on the HP data to make sure the fact that they are actually trained on the copyrighted HP data. They then serve as our original LLMs. We then split the HP data into the unlearned set and the test set. We use BookCorpus [53] as the normal dataset D nor since it is also book text which is in the same format as D fgt (Key Difference 3 in Section ?? ). We test the same three LLMs in Section 4.2.

Setting. The LLM task in this application is text completion. We largely follow the setting from [4]. Each prompt starts with the beginning of a sentence in the HP corpus, continuing for the next 200 characters as the prompt text (therefore an attempt to extract the copyrighted text). Given a prompt, we can test how much copyrighted information is leaked by comparing the LLM's completion (with greedy sampling, i.e. setting temperature to 0) to the ground-truth HP text. We set the comparison length to 200 characters and use BLEU score [33] as the text similarity metric.

For a prompt, i.e. an extraction attempt, we judge the copyright information is leaked if its completion's BLEU score is above a threshold. 10 We choose the threshold by randomly sampling 100K sentences in the HP corpus, computing their average BLEU score, and using 10% of it as the threshold. We report the leak rate, i.e. the percentage of extraction prompts that lead to the leakage as the unlearning effectiveness measure. We use BookCorpus as the data for the baseline of fine-tuning. We sample 100 prompts from the unlearned samples, unseen HP samples (HP text trained into the LLM but not unlearned), and normal samples (BookCorpus as the normal completion test set) respectively. We include the hyperparameter setting in Appendix D.2 and generated samples in Appendix E.2.

Results. Table 2 reports the results. We summarize the findings. (1) Both GA and GA+Mismatch can reduce the leak rate on the unlearned extraction attempts to nearly zero, showing the effectiveness of our unlearning algorithm in removing copyrighted content. 11 The completed text is mostly a repetition of a single character; such nonsensical output is expected in our setting given we have no

positive examples that show what a good completion should be. (2) Both GA and GA+Mismatch can generalize to unseen extraction attempts, showing unlearned LLM can distinguish copyright-related prompts from other prompts. (3) Both GA and GA+Mismatch achieve a similar utility on the normal completion task compared to the original LLM.

4.4 Application: Reducing Hallucination

If an LLM outputs factually wrong answers (i.e. hallucinations) given fact-related questions, the goal is to make the LLM unlearn wrong answers. Similar to unlearning harmfulness in Section 4.2, we do not assume the unlearned (i.e. hallucinated) Q&A samples (which are wrong answers given the questions) exist in the LLM's training set.

It is easy to imagine LLM can forget the wrong answer to the exact unlearned prompts. But it seems hard to generalize to unseen prompts since each individual factual question is different and highly specific and unlearning wrong answers to a specific question seems unlikely to impact answers to other questions. However, we do not aim to give factually correct answers but rather not give factually wrong answers. Therefore, all the LLM needs to do is to learn to classify which questions to respond (i.e. normal questions) and which do not (i.e. similar questions to the unlearned ones) by learning the distribution difference between questions.

Dataset and Model. We select the hallucinated Q&A pairs (i.e. negative samples) in the HaluEval [21] dataset as D fgt and TruthfulQA [22] dataset as D nor . We split D fgt into 70% for training, 10% for validation, and 20% for testing. Note that there exists a distribution shift between HaluEval data and TruthfulQA data. The questions in HaluEval are intentionally misleading; the questions in TruthfulQA are benignly straightforward. Therefore, this difference allows the unlearned LLM to distinguish between those two types of questions and therefore give different answers accordingly. In other words, the test (not unlearned) questions from HaluEval are in-distributional in terms of unlearning while the questions from the normal TruthfulQA data are out-of-distributional. Regarding models, we use the same three LLMs in Section 4.2.

Setting. To evaluate the effectiveness of reducing hallucination, we define the hallucination rate. Given the LLM's output, we compute its text similarity to the hallucinated answer and the correct answer. We choose BERTscore [50] as the text similarity because it is insensitive to text length and there is a significant length difference between hallucinated and correct answers. We decide an answer is hallucinated if its similarity to the hallucinated answer is 10% higher than the similarity to the correct answer. The hallucination rate is the percentage of test samples with hallucinated answers given by the LLM. The rest of the setting is similar to Section 4.2. We include the hyperparameter setting in Appendix D.3 and generated samples in Appendix E.3.

Results. Table 3 shows the results. The observations are largely similar to the previous applications. (1) Both GA and GA+Mismatch can significantly reduce the hallucination rate on the unlearned questions. (2) Both GA and GA+Mismatch can generalize de-hallucinating to the in-distributional questions from the same dataset used in unlearning. (3) Both GA and GA+Mismatch can distinguish

between in-distributional and out-of-distributional questions. They remove hallucinations when responding to in-distributional questions w.r.t unlearned questions and maintain similar answers as the original LLM when responding to out-of-distributional questions. (4) Compared to the previous two applications, the hallucination rate is not at a similarly low level ( ∼ 10% ), which shows unlearning hallucination is a harder task. We think the goal is to reduce in-distributional hallucination rather than completely unlearning general hallucination.

4.5 Ablation Studies

Comparing to RLHF. We compare our unlearning algorithm to the standard RLHF. However, note that in this case we already assume RLHF has access to the expensively collected positive samples (as well as negative samples) while our algorithm only has negative samples. Therefore, the comparison has already put our method in a disadvantaged position. Nevertheless, we still show that our method can achieve better alignment performance with only a fraction of computational cost despite that we only have negative samples.

Using unlearning harmfulness as an example, we perform RLHF on PKU-SafeRLHF data. The LLM is OPT-1.3B and the hyperparameters in RLHF are mostly default. We run both SFT (supervised finetuning) and full RLHF pipeline (SFT + reward model training + Proximal Policy Optimization [35]). We report the run time on a single NVIDIA A100 SXM4 80 GB GPU in Figure 1. Our unlearning algorithm only needs about 2% of the time required for the full RLHF pipeline, with a comparable cost to mere finetuning.

Table 4 shows the evaluation results compared to RLHF. Unlearning can achieve a lower harmful rate compared to the full RLHF, and a far lower harmful rate than SFT. This result is worth highlighting given we do not even use positive samples and with only 2% of the computational time. It shows that only using negative samples with unlearning can achieve a surprisingly promising non-harmful rate, which is the goal in our setting. Therefore, if the goal is to stop outputting undesirable responses rather than to output helpful responses, our results show unlearning might be a more appealing approach than RLHF.

Full RLHF

SFT

Finetuing on

Remaining

GA

GA+Mismatch

0

1

2

3

4

5

6

7

8

Time (Hr)

Templated Outputs. If practitioners do not want the unlearned LLM to generate nonsensical outputs (e.g. whitespace) on harmful prompts, we can replace the random output y rdn in Eqn(5) with templated outputs, e.g. 'I can't assist it.' In other words, we force the LLM to generate the templated answers on the unlearned prompt.

We follow the setting of unlearning harmfulness in Section 4.2. We replace the random output y rdn in Eqn(5) with the templated answer 'I can't assist it.' and keep other settings the same except we re-tune loss weight ϵ 1 , ϵ 2 , and ϵ 3 in Eqn(2). Table 5 shows the comparison with the previous results on OPT-1.3B. GA+Template achieves a similar level of unlearning performance compared to GA and GA+Mismatch. Overall, using templated answers instead of random answers does not show significant differences.

Table 38 in Appendix E.4 shows generated examples compared to GA and GA+Mismatch. We observe that if the unlearned LLM has learned to respond differently to the harmful prompts, we can easily make it output templated responses instead of nonsensical strings. In addition, it is easy to enable templated answers without changing unlearning optimization: we can check if the outputted text is a nonsensical string and replace it with templated strings as a post-processing heuristic.

Finally, an even simpler heuristic for generating templated outputs is to check if the outputted text is a nonsensical string and replace it with templated strings.

5 Conclusion

We explore unlearning in LLMs, as well as its formal setups, goals, and evaluations. Our results show that unlearning is a promising approach to aligning LLMs to stop generating undesirable outputs, especially when practitioners do not have enough resources to apply other alignment techniques such as RLHF. We present three scenarios in which unlearning can successfully remove harmful responses, erase copyrighted content, and reduce hallucinations. Our ablation study shows that despite only having negative samples, unlearning can still achieve better alignment performance than RLHF with only a fraction of its computational time. One limitation of our approach is it might induce refusal responses on normal prompts.

References

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 , 2022.

Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP) , pages 141-159. IEEE, 2021.

Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy , pages 463-480. IEEE, 2015.

Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646 , 2022.

Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security Symposium , volume 6, 2021.

Sungmin Cha, Sungjun Cho, Dasol Hwang, and Moontae Lee. Towards robust and cost-efficient knowledge unlearning for large language models. arXiv preprint arXiv:2408.06621 , 2024.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Zero-shot machine unlearning. IEEE Transactions on Information Forensics and Security , 2023.

GitHub Copilot. Github copilot lawsuit. https://www.courthousenews.com/microsoft-and-githubask-court-to-scrap-lawsuit-over-ai-powered-copilot/ , 2023.

Shitong Duan, Xiaoyuan Yi, Peng Zhang, Tun Lu, Xing Xie, and Ning Gu. Negating negatives: Alignment without human positive samples via distributional dispreference optimization. arXiv preprint arXiv:2403.03419 , 2024.

Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238 , 2023.

Facebook. Facebook community standards. https://www.facebook.com/ communitystandards/ , 2023.

Chongyang Gao, Lixu Wang, Chenkai Weng, Xiao Wang, and Qi Zhu. Practical unlearning for large language models. arXiv preprint arXiv:2407.10223 , 2024.

Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. arXiv preprint arXiv:1911.03030 , 2019.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 , 2019.

James Y Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen. Offset unlearning for large language models. arXiv preprint arXiv:2404.11045 , 2024.

Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics , pages 2008-2016. PMLR, 2021.

Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. arXiv preprint arXiv:2307.04657 , 2023.

Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsification can simplify machine unlearning. arXiv preprint arXiv:2304.04934 , 2023.

Jooyoung Lee, Thai Le, Jinghui Chen, and Dongwon Lee. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023 , pages 3637-3647, 2023.

Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Halueval: A large-scale hallucination evaluation benchmark for large language models. arXiv e-prints , pages arXiv-2305, 2023.

Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 , 2021.

Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787 , 2024.

Yang Liu, Mingyuan Fan, Cen Chen, Ximeng Liu, Zhuo Ma, Li Wang, and Jianfeng Ma. Backdoor defense with machine unlearning. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications , pages 280-289. IEEE, 2022.

Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: a survey and guideline for evaluating large language models' alignment. arXiv preprint arXiv:2308.05374 , 2023.

Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402.10058 , 2024.

Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems , 35:27591-27609, 2022.

Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems , 32, 2019.

Andrei Muresanu, Anvith Thudi, Michael R Zhang, and Nicolas Papernot. Unlearnable algorithms for in-context learning. arXiv preprint arXiv:2402.00751 , 2024.

Kevin P. Murphy. Probabilistic Machine Learning: An introduction . MIT Press, 2022.

Seth Neel, Aaron Roth, and Saeed Sharifi-Malvajerdi. Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory , pages 931-962. PMLR, 2021.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311-318, 2002.

Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579 , 2023.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696 , 2020.

Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint arXiv:2407.06460 , 2024.

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) , pages 3-18. IEEE, 2017.

Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems , 2023.

Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P) , pages 303-319. IEEE, 2022.

TikTok. Tiktok community guidelines. https://www.tiktok.com/ community-guidelines?lang=en , 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Twitter. Twitter rules and policies. https://help.twitter.com/en/ rules-and-policies/twitter-rules , 2023.

Jan Philip Wahle, Terry Ruas, Frederic Kirstein, and Bela Gipp. How large language models are transforming machine-paraphrased plagiarism. arXiv preprint arXiv:2210.03568 , 2022.

Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. arXiv preprint arXiv:2108.11577 , 2021.

Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Philip S Yu. Machine unlearning: A survey. ACM Computing Surveys , 56(1):1-36, 2023.

Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302 , 2023.

Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868 , 2024.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 , 2019.

Zhexin Zhang, Junxiao Yang, Pei Ke, Shiyao Cui, Chujie Zheng, Hongning Wang, and Minlie Huang. Safe unlearning: A surprisingly effective and generalizable solution to defend against jailbreak attacks. arXiv preprint arXiv:2407.02855 , 2024.

Rui Zheng, Shihan Dou, Songyang Gao, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Limao Xiong, Lu Chen, et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964 , 2023.

Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV) , December 2015.

Terry Yue Zhuo, Zhuang Li, Yujin Huang, Yuan-Fang Li, Weiqing Wang, Gholamreza Haffari, and Fatemeh Shiri. On robustness of prompt-based semantic parsing with large pre-trained language model: An empirical study on codex. arXiv preprint arXiv:2301.12868 , 2023.

A Why Gradient Ascent?

We mainly follow the approach of gradient ascent (GA) for three main reasons.

First , GA is particularly suitable for our scenario where only given negative samples and the goal is to stop generating undesirable text rather than generating desirable text. Consider the following prompt when harmful tokens are highly likely in an unaligned LLM: 'Human : How can I hurt people most efficiently? Assistant: ' The next predicted token has a high probability to be 'Gun,' 'Poison,' or 'Fire' etc. In RLHF, we would need many iterations from both positive and negative samples to indirectly reduce the probability of harmful tokens to the level below the helpful tokens. However, if our goal is to not output harmful tokens, then we can directly update the LLM by following the opposite direction of the gradient on the harmful tokens to reduce their probability. In this case, without any example of helpful answers, we do not know which direction to go to generate good responses, but taking the opposite direction of harmful tokens is almost guaranteed and arguably the most efficient way to not output harmful answers.

Second , GA is efficient with a cost comparable to finetuning. And since the unlearned dataset is normally small, performing GA with unlearned samples costs less than general finetuning for improving utility. In addition, given the size of LLMs, Hessian-based unlearning is too costly to apply.

Third , GA is sometimes viewed as a 'coarse' method in the unlearning literature. This is mostly because directly going the opposite of the unwanted direction might cause unexpected model behaviors. However, in LLMs, since the model capacity is huge, it has more capacity to tolerate operations like GA, which normally would be too disruptive in small-capacity classification models.

B Analysis on Random Mismatch Loss

Adding random mismatch loss L rdn in eqn(5) has two advantages. First, broadly speaking, an LLM can forget an undesirable output by either (1) forgetting the specific undesirable part of the answer or (2) reducing the general ability to generate coherent text. In general, we prefer (1) and want to reduce the chance of (2). L rdn helps us by forcing the LLM to predict an answer which, though random, is still grammatically intact.

Second, using GA alone can be ineffective when the gradient of forgetting samples are small. Assume using loss as a proxy of the effectiveness of unlearning, the goal of unlearning is to maximize: ℓ ( x, y ; θ o + ∆ θ ) -ℓ ( x, y ; θ ) , where ( x, y ) ∈ D fgt and θ o is the original LLM. Using first-order approximation we have

If we use GA alone, we have ∆ θ = λ · ∇ θ ℓ ( θ, x, y ) . Plugging back we have

While the term is guaranteed to be positive, its effect is limited when ||∇ θ ℓ ( x, y ; θ ) ||→ 0 .

On the other hand, using the random term, we have,

It leads to

Hence, even if the gradient on ( x, y ) is small, i.e. ||∇ θ ℓ ( x, y ; θ ) ||→ 0 , as long as

the unlearning can perform in a positive direction. Intuitively, this corresponds to finding a random answer that incurs a loss that is in the opposite direction of y . We hope that by randomly selecting an irrelevant answer, with some probability that it could be in the opposite direction of the undesirable answer y .

Table 6: Harmful content warning. Responses to harmful prompts after unlearning ∼ 200 (when the loss on harmful samples is already as high as ∼ 60 ) and ∼ 1000 batches.

C How Does LLM Unlearning Differ from Traditional Unlearning?

We highlight the key difference in LLM unlearning compared to the traditional unlearning in classification tasks. We discover those findings mostly through empirical observations, and they guide us in designing our unlearning algorithm. For all the experimental observations in this section, we use the example of unlearning harmfulness with OPT-1.3B and the unlearned and normal samples from PKU-SafeRLHF [18] and TruthfulQA respectively [22].

Key Difference 1 : Both training and validation loss on the unlearned samples have limited indications of unlearning effectiveness. For example, when we apply gradient ascent (GA), even when the loss on the unlearned samples rises to as high as 60+ after unlearning for ∼ 200 batches (Figure 2), the LLM still outputs harmful responses to harmful prompts (Table 6). This is not observed in traditional unlearning, where the losses on the forget samples are often strong indicators of the unlearning performance.

Solution 1 : We find continuing to unlearn after the loss on harmful samples rises dramatically is necessary for unlearning effectiveness. For example, although the loss on harmful samples already looks promising after unlearning ∼ 200 batches, we find the LLM only stops outputting harmful responses after ∼ 1000 batches (Table 6). We also propose an additional loss that randomly mismatches between x fgt and its response to facilitate the forgetting of y fgt (See Section 3).

Key Difference 2 : Performance on normal prompts deteriorates easily after unlearning. We find that preserving performance on normal samples is generally harder to achieve than forgetting. For example, with GA, it is often not hard to make the LLM output random responses. 12 However, the LLM is likely to also generate nonsensical outputs on normal response. Table 7 shows the example of nonsensical outputs after unlearning with gradient ascent on ∼ 1000 batches. Although the LLM stops generating harmful responses on the harmful prompts, it also generates nonsensical outputs on normal prompts, destroying the LLM's utility.

Solution 2 : We empirically find that merely optimizing the cross-entropy loss on a normal dataset does not maintain the normal performance well. Like existing work in RLHF [32, 42, 52, 15], we find that minimizing the divergence between the output on x nor from the unlearned LLM and the original LLM works the best. (See Section 3.)

Key Difference 3 : The format (e.g. Q&A, book text, chat, multiple choice etc.) of D nor (for guiding the LLMs to preserve utility on normal tasks) has a large impact on the normal performance. When the format of D nor and D fgt differ substantially, the unlearned LLM can learn a shortcut that decides what to output by the format of the prompt only, and therefore does not truly unlearn the concept.

Solution 3 : To maintain the normal performance, we find that choosing the format of D nor to be the same with D fgt (e.g. if D fgt is Q&A, then D nor should also be Q&A) can better help preserve normal utility.

0

200

400

600

800

1000

1200

Step

0

10

20

30

40

50

60

70

Loss

Loss on Unlearned Samples

Loss on Normal Samples (Validation)

D Experimental Settings

D.1 Unlearning Harmfulness

Table 8 summarizes the hyperparameters used in unlearning harmfulness.

D.2 Unlearning Copyrighted Content

Table 9 summarizes the hyperparameters used in unlearning copyrighted content.

D.3 Reducing Hallucination

Table 10 summarizes the hyperparameters used in reducing hallucination.

E Example of Generated Text

Note that the generated text from our method, GA and GA+Mismatch, are first examined by a nonsense detector 13 and then if it is predicted to be nonsense, will be replaced by templated responses, e.g. 'I can't assist it.' in the way described in Section 4.5 (templated output).

E.1 Unlearning Harmfulness

Table 11-19 show examples of generated text in unlearning harmfulness. Harmful content warning.

E.2 Unlearning Copyrighted Content

Table 20-28 show examples of generated text in unlearning copyrighted content. To avoid overexposing copyrighted text, we only show three examples per experiment.

E.3 Reducing Hallucinations

Table 29-37 show examples of generated text in unlearning harmfulness. Untruthful content warning.

E.4 Ablation: Templated Outputs

Table 38 shows generated text with templated output.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction summarizes and introduces the main contribution and scope of the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper includes the required information to reproduce the main results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We submitted our code in the supplementary file.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental setting and details are clearly presented.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: There is no statistical test in the experiments, as for LLM experiments it would be too costly.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We specified the compute resources in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The experiments followed the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discussed societal impacts of the method.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper has no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The existing assets are properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how to perform unlearning on large language models. It conducts a systematic investigation of unlearning in three typical scenarios of LLM applications, including unlearning harmful responses, erasing copyright-protected content, and reducing hallucinations. In all scenarios, the proposed unlearning method is shown to unlearn the required contents successfully. It also shows that the proposed unlearning method can be regarded as an alternative to alignment methods that does not require positive samples.\n\nSTRENGTHS:\n1. The paper is very well-written and well-structured. The authors claim their points clearly with adequate supportive experiments, making it a sound and complete paper. \n\n2. The paper is a pioneering work in the field of unlearning for LLMs, which systematically defines and investigates the unlearning problem under three typical and practical unlearning scenarios. I believe this work can guide and inspire subsequent work in this field.\n\nWEAKNESSES:\nGiven that the resulting outputs are mostly nonsense, the proposed unlearning method is less sound. In my opinion, the most expected result of unlearning is that the model outputs **fluent and coherent responses** (which are better to relate to the prompt, retaining the ability of instruction-following) that are substantially different from the undesired content. If the LLMs only output nonsense words after unlearning, such an approach will be less practical in most applications such as chatbots and AI assistants. In fact, in [1], they are able to ensure the fluency and coherence of output sentences after unlearning. Therefore, the proposed method is less sound and useful to me.\n\nBy the way, I wonder why didn't this paper include [1] as a baseline method?\n\n\n\n\n[1] Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023.\n\nQUESTIONS:\n1. What does \"finetuning\" mean in Table 1? How did you calculate the \"Similarity to Original\" for the original baseline?\n \n2. Why didn't you include [1] as a baseline method?\n\nLIMITATIONS:\nTypos:\n1. Line 175: x^{rdn} -> x^{fgt}?\n2. Line 192: x^{fgt} -> x^{nor}?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the concept of unlearning in large language models (LLMs) as an alternative approach to aligning AI systems with human preferences. The authors propose methods for removing unwanted behaviors or knowledge from LLMs without requiring expensive retraining. They demonstrate the effectiveness of their unlearning techniques in three applications: reducing harmful outputs, erasing copyrighted content, and decreasing hallucinations. The proposed approach uses gradient ascent and only requires negative examples, making it more efficient than traditional alignment methods like RLHF. The authors show that their unlearning method can achieve better alignment performance than RLHF with only 2% of the computational cost. They also discuss the challenges specific to unlearning in LLMs compared to traditional machine learning models and provide insights on evaluating unlearning effectiveness.\n\nSTRENGTHS:\n- The paper introduces unlearning for aligning large language models, addressing important issues like harmful outputs, copyright infringement, and hallucinations without requiring expensive retraining or positive examples.\n- The authors demonstrate that their unlearning technique can achieve better alignment performance than RLHF while using only 2% of the computational resources. This makes it a highly practical solution, especially for researchers or organizations with limited computational capacity.\n- The paper provides a thorough examination of unlearning in LLMs, including detailed experimental results across multiple applications, comparisons to existing methods like RLHF, and thoughtful discussion of the challenges and differences between unlearning in LLMs versus traditional machine learning models.\n\nWEAKNESSES:\n- Drawing the connection between machine unlearning and RLHF techniques is a good point. And the authors claim that they can achieve similar performance to RLHF while only using 2% of the computational resource. However, a fundamental concern is whether the comparison is in a fair manner and whether the result is generalizable or not. A biggest problem of existing machine unlearning techniques, from my past experience, is that they usually lead to very unstable training, especially when the unlearning steps or the unlearning examples become larger. This will sometimes make the model collapse. I do not see a related discussion in the paper and I have strong concern about that aspect. \n- Continue to my first point, the comparison might not be fair. Firstly, there is a lack of details on how the running time is calculated. Secondly, the authors might want to compare with DPO or other more efficient RLHF techniques. \n- The generalizability of the approach is questionable as well. The paper demonstrates unlearning on a limited set of tasks (harmfulness, copyright, hallucination). It's unclear how well the method generalizes to other types of undesirable behaviors or knowledge. Additionally, the evaluation metrics are somewhat task-specific, making it difficult to compare the overall effectiveness of unlearning across different applications or to other alignment methods in a standardized way. The authors might want to adopt a similar setting to RLHF papers for a fair comparison.\n\nQUESTIONS:\nHave you observed any unstable training during the unlearning process?\n\nLIMITATIONS:\nThe authors only discuss the limitation in one sentence at the end of the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research paper explores techniques for inducing large language models (LLMs) to selectively forget pre-learned information or undesirable behaviors through an unlearning paradigm. The primary objectives are to eliminate copyrighted content and mitigate harmful responses. The authors propose machine unlearning as a computationally efficient alternative to Reinforcement Learning from Human Feedback (RLHF), noting that it only requires negative examples, whereas RLHF necessitates both positive and negative samples.\nThe main unlearning algorithms introduced is gradient ascent and random mismatch. These are regularized using gradient descent on normal training data. The researchers demonstrate that their methodology effectively erases copyrighted material and reduces harmful outputs from LLMs.\n\nSTRENGTHS:\n- The paper is very well written and easy to follow. The ideas are communicated clearly, and the mathematical equations and introduced metrics are easy to understand.\n- The breakdown of the evaluation criteria is very thorough, and the analysis on diversity and fluency is very useful and important to properly assess the quality of the result generations after unlearning.\n- The method is effective in unlearning the undesirable behavior.\n\nWEAKNESSES:\n- While the paper presents an somewhat effective approach, its novelty is somewhat limited. The core techniques—gradient ascent on data to be forgotten, random mismatch, and gradient descent—have been previously employed and combined for unlearning in both LLM and image classification contexts (see unlearning kaggle competition).\n\n- A potential drawback of the proposed method is its impact on model engagement. Generating white spaces or fixed outputs in response to harmful prompts, while addressing immediate concerns, may inadvertently reduce the model's overall utility and conversational appeal.\n\n- Furthermore, the approach raises privacy concerns. An attacker could potentially infer that certain information was learned and subsequently unlearned, compromising the ideal scenario where an attacker cannot discern whether specific data was ever part of the training set. Typically, unlearning aims to produce outputs indistinguishable from those of a model never exposed to the forgotten data, accounting for models' ability to generalize beyond their training examples. This paper's method may fall short in achieving this level of privacy protection.\n\nQUESTIONS:\n- How would you improve the helpfulness of the model after unlearning and metigate privacy risks? \n- It would be good to compare against other unlearning methods introduced in the literature.\n\nLIMITATIONS:\nYes. limitations were adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-28"}