Title: WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Machine unlearning; Weight attribution; LLMs;
Abstract: The need for effective unlearning mechanisms in large language models (LLMs) is increasingly urgent, driven by the necessity to adhere to data regulations and foster ethical generative AI practices. LLM unlearning is designed to reduce the impact of undesirable data influences and associated model capabilities without diminishing the utility of the model if unrelated to the information being forgotten. Despite growing interest, much of the existing research has focused on varied unlearning method designs to boost effectiveness and efficiency. However, the inherent relationship between model weights and LLM unlearning has not been extensively examined. In this paper, we systematically explore how model weights interact with unlearning processes in LLMs and we design the weight attribution-guided LLM unlearning method, WAGLE, which unveils the interconnections between 'influence' of weights and 'influence' of data to forget and retain in LLM generation. By strategically guiding the LLM unlearning across different types of unlearning methods and tasks, WAGLE can erase the undesired content, while maintaining the performance of the original tasks. We refer to the weight attribution-guided LLM unlearning method as WAGLE, which unveils the interconnections between 'influence' of weights and 'influence' of data to forget and retain in LLM generation. Our extensive experiments show that WAGLE boosts unlearning performance across a range of LLM unlearning methods such as gradient difference and (negative) preference optimization, applications such as fictitious unlearning (TOFU benchmark), malicious use prevention (WMDP benchmark), and copyrighted information removal, and models including Zephyr-7b-beta and Llama2-7b. To the best of our knowledge, our work offers the first principled method for attributing and pinpointing the influential weights in enhancing LLM unlearning. It stands in contrast to previous methods that lack weight attribution and simpler weight attribution techniques.

WAGLE : Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models

Abstract

The need for effective unlearning mechanisms in large language models (LLMs) is increasingly urgent, driven by the necessity to adhere to data regulations and foster ethical generative AI practices. LLM unlearning is designed to reduce the impact of undesirable data influences and associated model capabilities without diminishing the original utility of the model. Despite growing interest, much of the existing research has focused on varied unlearning method designs to boost effectiveness and efficiency. However, the inherent relationship between model weights and LLM unlearning has not been extensively examined. In this paper, we systematically explore how model weights interact with unlearning processes in LLMs and propose the weight attribution-guided LLM unlearning framework, WAGLE , which unveils the interconnections between 'influence' of weights and 'influence' of data to forget and retain in LLMs. By strategically guiding the LLM unlearning across different types of unlearning methods and tasks, WAGLE can erase the undesired content, while maintaining the performance of the original tasks. Our experiments show that WAGLE boosts unlearning performance across a range of LLM unlearning methods such as gradient difference and (negative) preference optimization, and applications such as fictitious unlearning (TOFU benchmark) and malicious use prevention (WMDP benchmark), under models including Zephyr-7b-beta and Llama2-7b. To the best of our knowledge, our work offers the first principled method for attributing and pinpointing the influential weights in enhancing LLM unlearning. It stands in contrast to previous methods that lack weight attribution and simpler weight attribution techniques. Codes are available at https://github.com/OPTML-Group/WAGLE.

1 Introduction

Large language models (LLMs) have demonstrated exceptional proficiency in generating text that closely resembles human-authored content. However, their capacity to memorize extensive corpora can raise ethical and security concerns, such as the generation of biased, private, harmful, or even illegal contents [1]. These issues highlight the necessity of effectively and efficiently tailoring pre-trained LLMs to remove these undesired data influences and associated generation capabilities, ensuring they are suitable for diverse application contexts. Therefore, the problem of machine unlearning (MU) for LLMs (referred to as LLM unlearning ) arises [2], aiming to equip trained LLMs with data- and model-erasing capabilities.

The concept of MU has gained increasing popularity due to its significance in assessing and manipulating the impact of data on model performance. Its importance originated from the need to protect data privacy [3-6], in response to data protection regulations like the 'right to be forgotten' [6]. The majority of past research efforts have focused on solving the problem of MU for classification models

[7-14]. Compared to LLM unlearning, the unlearning scope in classification problems is typically easier to define, often focusing on specific data points or classes to forget. Moreover, it is even feasible to retrain the classification models from scratch after removing the data/classes targeted for unlearning [8, 12]. The feasibility of retraining from scratch leads to the exact unlearning method, which is typically used as a gold standard in MU evaluation for classification models. However, such an exact unlearning method becomes infeasible for LLMs due to their prolonged training times and associated high costs. Instead, evaluations are often based on the specific unlearning tasks.

Therefore, LLM unlearning, despite falling under the broad category of MU, presents a much more challenging problem. The two main difficulties lie in developing effective and efficient unlearning algorithms and in assessing the performance of LLM unlearning.

Representative unlearning algorithms include gradient ascent (GA) [8, 15, 16] to deviate the LLM prediction away from responses to the forget data and its utility-regularized variants, such as GradDiff [15-17] which utilizes the gradient difference between the forget loss and the retain loss to strike a tradeoff between unlearning efficacy and utility retention. Drawing inspiration from direction preference optimization [18], the LLM unlearning problem has also been addressed using algorithms such as negative preference optimization (NPO) [19] and preference optimization (PO) [16]. NPO treats the forget data as negative examples in LLM preference alignment, while PO assigns predefined positive responses (such as rejection-based answers) to the forget data during preference alignment.

In addition, further studies explored the choice of optimizers suited for solving LLM unlearning problems [20] and proposed prompting-based algorithms to achieve unlearning for black-box LLMs [21-24].

A few recent benchmarked unlearning tasks and datasets have also been developed to facilitate performance evaluation. Examples include the TOFU dataset for fictitious unlearning [16], the WMDP dataset for malicious use prevention of LLMs [25], the copyrighted information removal [26], and the LLM detoxification task [27, 28]. All these evaluations will be considered in this work.

Despite the rapid progress in LLM unlearning algorithms and evaluation methods, less effort has been made to explore the modularity characteristics of LLMs for unlearning and the influence of these modules. In the literature, weight sparsity achieved through model pruning has been found beneficial in reducing the gap between a GA-based approximate unlearning method and exact unlearning [12]. However, this advantage was limited to MU for classification models. As we will demonstrate, the benefit of pruning does not directly apply to LLM unlearning, as it excludes the forgetting influence on weight selection. Another relevant line of work is weight localization for LLM editing [29, 30]. However, Hase et al. [30] demonstrated that the popular causal tracing-based weight localization technique [29] cannot precisely predict which layers within an LLM are most influential for knowledge editing or removal. Other studies have also examined the saliency of LLM modules for unlearning, focusing on weights' gradients [31] and neurons within the feed-forward network [32].

Although there is emerging interest in exploring the relationship between LLM unlearning and its model fingerprints, such as layers and neurons, no principled approach exists to precisely attribute weight-level influence in LLM unlearning and facilitate the unlearning process. This gap gives rise to the central problem of this work: Weight attribution for LLM unlearning. Specifically, we ask:

(Q) How to identify influential weights to enhance unlearning efficacy while preserving LLM utility?

To tackle (Q), we interpret the problem of weight attribution from a bi-level optimization (BLO) perspective. This approach allows us to attribute the weights' influence in LLM unlearning by considering both the unlearning objective (modeled in the upper-level problem of BLO) and the model utility retention objective (modeled in the lower-level problem of BLO). It also enables us to derive the closed-form attribution scores for identifying influential weights using the implicit gradient approach in BLO. Further, we develop the weight attribution-guided LLM unlearning framework ( WAGLE ), easily compatible with existing LLM unlearning algorithms. We summarize our contributions below.

· We propose the problem of weight attribution for LLM unlearning and highlight its distinct challenges compared to conventional approaches using weight pruning.

· We solve weight attribution through the lens of BLO and derive its closed-form solution.

We develop WAGLE to be agnostic to specific unlearning algorithms and demonstrate its effectiveness across diverse unlearning benchmarks and evaluation metrics.

2 Related Work

Machine unlearning (MU) for non-LLMs. The concept of MU was originally raised to address users' deletion requests for given machine learning (ML) models, without the need to retrain these models from scratch [3-5]. The capability to assess and erase the influences of data to be forgotten in model performance has broadened the MU concept across diverse ML paradigms, such as image classification [11, 12, 33, 34], image generation [13, 35-37], generative language modeling [2, 38-40], graph neural networks [41-43], and federated learning [44-46]. The methodologies of MU include retraining-based exact unlearning [8, 47], differential privacy (DP)-based unlearning [7, 9, 10, 48], and fine-tuning-based approximate unlearning [8, 11, 12, 49-51].

LLM unlearning. When MU shifts to the realm of LLMs, new challenges and complexities arise. The two main difficulties in effective and efficient algorithmic design and unlearning evaluation have been highlighted in Sec. 1. Another related challenge is how to precisely define the scope of LLM unlearning [2]. Existing work has raised concerns that the current unlearning scope is insufficient for declaring the robustness and reliability of LLM unlearning. This is evidenced by the extractable unlearned knowledge from LLMs post-unlearning when facing in-context relearning [52] and jailbreaking attacks [53]. Yet, even in the absence of these knowledge extraction 'adversaries', enhancing the efficacy of LLM unlearning remains a highly non-trivial problem. Existing LLM unlearning methods are predominantly fine-tuning-based approaches [15, 16, 19, 20, 26], which are favored for their computational efficiency. Application-wise, the promise of LLM unlearning has been demonstrated in diverse use cases, such as protecting copyrighted or personal identification information [26, 32, 54], preventing the use of LLMs in developing cyberattacks or bioweapons [25, 55], and mitigating the generation of toxic, biased, or hallucinated content [15, 27, 31].

Data and weight attribution. A key mission of MU is to quantify the influence of forgotten data on model performance, which aligns with the classic data attribution problem [56, 57]. Indeed, the influence function approach, originally developed for assessing the impact of individual training data points on model generalization performance [56], has also been used in MU for classification models [12, 51] and in analyzing LLM's generalization [58]. Furthermore, data attribution is essential in solving dataset pruning or coreset selection problems [59-63]. By contrast, the problem of weight attribution has received less attention compared to data attribution in the context of LLM unlearning, where the former aims to identify a model-level fingerprint, i.e. , the subset of most influential weights, for the unlearning task. One relevant line of research is weight localization-informed unlearning [31, 32], which provides insights into which model units (such as layers and neurons) should be edited for effective unlearning. However, a precise characterization of weight influence in unlearning is still lacking [64]. In the non-unlearning context, weight pruning [65-69] can also be considered a weight attribution method that focuses solely on model utility performance. Yet, we will show that weight pruning alone is insufficient for identifying the model fingerprint for LLM unlearning.

3 Preliminary and Problem Setup

Definition and formulation of LLM unlearning. LLM unlearning pertains to the MU problem in LLMs, aimed at removing undesirable data influence ( e.g. , sensitive, illegal, or harmful information) and the associated model capabilities, without sacrificing the integrity of essential knowledge generation that is unrelated to what is being forgotten [2]. Despite the pressing need for effective LLM unlearning [15, 25-27, 31, 32, 54, 55], achieving this goal remains a substantial challenge. In particular, retraining LLMs from scratch after removing the targeted training data for unlearning is infeasible due to (1) the prohibitive training costs and (2) the difficulty of precisely attributing and localizing the specific training data points to forget. Instead of that, LLM unlearning is typically achieved via model fine-tuning or alignment for a pre-trained model.

More concretely, let θ o denote the pre-trained LLM, and the unlearning task be represented through a forget set D f . It also defines a forget loss , ℓ f ( D f ; θ ) , to optimize for the model post-unlearning θ (referred to as 'unlearned model'). Additionally, the unlearned model needs to retain the model utility.

Therefore, a retain set D r is often incorporated into the unlearning objective. This set is unrelated to what is being forgotten but enforces model utility through a retain loss ℓ r ( D r ; θ ) . To strike a balance between unlearning effectiveness and utility preservation, the problem of LLM unlearning is formulated as a regularized optimization problem [2]:

where λ ≥ 0 is a regularization parameter. If λ = 0 , then unlearning relies solely on the forget set. However, existing unlearning methods, such as gradient ascent (GA) [15, 16, 19], have demonstrated that omitting the retain loss would result in a significant degradation of model utility post-unlearning.

Forget loss design and specific unlearning methods. In (1), the retain loss ℓ r typically mirrors the training loss over the retain set. Yet, the design of the forget loss ℓ f is more challenging, as it influences the specific approach to LLM unlearning. In what follows, we review three state-of-the-art (SOTA) methods for LLM unlearning and explore the design of their respective forget loss functions.

Gradient difference (GradDiff) [15, 17]: ℓ f = ℓ GA . GradDiff specifies ℓ f as the negative training loss (also known as the GA loss ℓ GA ) to encourage the response of the LLM post-unlearning to deviate from its original response within the training set. This method is equivalent to using GA on the forget set while applying gradient descent on the retain set, which explains the name GradDiff.

Negative preference optimization (NPO) [19]: ℓ f = ℓ NPO . NPO specifies the forget loss ℓ f as the loss of direct preference optimization (DPO) [18] by treating the forgotten data in D f exclusively as negative examples in DPO. This negative example-only variant of the DPO loss is referred to as NPO ℓ NPO . Compared to GradDiff, the NPO loss outperforms the GA loss due to its improved stability, avoiding catastrophic collapse in forgetting and utility preservation during optimization [19].

Preference optimization (PO) [16]: ℓ f = ℓ PO . This approach is also inspired by DPO but introduces targeted unlearning responses such as 'I don't know' or responses stripped of sensitive information, treating these exclusively as positive examples for preference alignment. In contrast to NPO, the positive example-based forget loss is termed as ℓ PO . Compared to GradDiff, PO modifies the unbounded GA loss by introducing the positive unlearning response for a bounded forget loss.

Throughout the paper, we will address the problem of LLM unlearning following the generic formulation (1), with specific implementations using GradDiff, NPO, or PO.

Weight attribution in LLM unlearning: Rationale and motivation. As shown above, past research has primarily focused on algorithm-centric perspectives to tackle LLM unlearning problems. Yet, effective unlearning also requires a sense of locality, which involves identifying the sub-components of the LLM ( i.e. , a subset of weights in this work) that are crucial for the unlearning task, while minimally impacting the model's original utility. Such a model-level fingerprint of LLM unlearning is agnostic to specific unlearning algorithms, potentially leading to a universal booster for LLM unlearning. It also exposes the modularity characteristics of LLMs, facilitating modular unlearning that specifically targets the designated weight subspace.

Thus, we propose to investigate the problem of weight attribution in LLM unlearning, which involves assessing the influence of weights so as to identify the critical subset of weights essential for effective and modular unlearning. In the context of non-LLM unlearning, weight sparsity [12] or gradient-based saliency [13] has proven beneficial for narrowing the gap between GA-type approximate unlearning and exact unlearning ( i.e. , retraining from scratch). Yet, when applied to LLMs, the effectiveness remains elusive.

Fig. 1 provides a preliminary demonstration of the (in)effectiveness of unlearning (measured by the average unlearning efficacy, as defined in Tab. 1) and model utility (measured by the average utility performance, also defined in Tab. 1) vs. pruning-induced weight selection. This is achieved by applying the SOTA unlearning method NPO to update the remaining (unpruned) weights of LLMs, where weight sparsity is determined using the SOTA pruning method Wanda [70],

0%

5%

10%

20%

40%

Sparsity of Unlearned Weights

0.2

0.4

0.6

0.8

1.0

Unlearning Efficacy

0.6

0.7

0.8

0.9

Utility

NPO+Wanda (Efficacy)

NPO+Wanda (Utility)

in the context of TOFU unlearning [16]. A lower sparsity indicates that a larger proportion of weights are updated during the unlearning process. As observed, the unlearning efficacy is highly sensitive

to weight sparsity, as demonstrated by the sharp decline in efficacy as sparsity increases compared to the dense model (0% sparsity). In addition, there is a clear tradeoff between unlearning efficacy and model utility. This highlights the challenge of identifying an optimal subset of weights for LLM unlearning-one that maintains both unlearning efficacy and utility. This sets the stage for our key research question: How can we precisely measure the roles of model weights in LLM unlearning? In the next section, we will introduce a new principled approach to weight attribution in LLM unlearning.

4 Weight Attribution for Enhanced LLM Unlearning

Weight attribution: Balancing unlearning 'objective' with utility 'constraint'. As inspired by Fig. 1, an effective weight attribution framework should account for not only utility preservation but also unlearning effectiveness. To address this challenge, we draw inspiration from bi-level optimization (BLO) [71], where we leverage the upper-level problem to evaluate the impact of weight adjustments on unlearning efficacy and the lower-level problem to ensure the retention of utility.

Specifically, let ϵ ⊙ θ represent the weight-adjusted model, where ϵ denotes the modifications applied to the weights θ , and ⊙ is element-wise multiplication. For example, if we choose ϵ = 1 + µ e i , with e i representing the i th basis vector, then ϵ ⊙ θ corresponds to perturbing the i th weight θ i to (1 + µ ) θ i . Here, µ controls the perturbation strength, and µ = -1 corresponds to pruning the i th weight. The goal of weight attribution is then to evaluate the influence of the weight adjustment ϵ on unlearning. Thus, given the forget loss ℓ f and the weight-adjusted model ϵ ⊙ θ , we measure the influence of the weights through the following unlearning sensitivity score : ℓ f ( ϵ ⊙ θ ) -ℓ f ( θ ) , where we omit the dependence of ℓ f on the forget set D f for notational simplicity. However, the above attribution involves an additional implicit constraint : The model parameters θ must minimize the retain loss to meet the model's utility. That is, θ ∗ ( ϵ ) = arg min θ ℓ r ( ϵ ⊙ θ ) , where the solution is denoted by θ ∗ ( ϵ ) to signify its dependency on the weight modification scheme ϵ .

By integrating the implicit model utility constraint into the unlearning sensitivity score, the proposed weight attribution problem can be cast as a BLO-type problem below:

where the upper-level and lower-level problems are coupled through the lower-level solution θ ∗ ( ϵ ) , and it reduces to the pre-trained model θ ∗ ( 1 ) = θ o as ϵ = 1 .

Analyzing weight attribution via implicit gradient. We next address the weight attribution problem (2) by linking the upper-level unlearning sensitivity analysis with the lower-level utility optimization through implicit gradient ( IG ), which is used in BLO to characterize the gradient flow from the lower-level solution to the upper-level variable. By employing the first-order Taylor expansion to the upper-level objective of (2) at ϵ = 1 , the unlearning sensitivity w.r.t. ϵ becomes:

where ⊤ denotes the matrix transpose, and d a d b ∈ R | b |×| a | is the full derivative of a w.r.t. b with | a | denoting the cardinality of the vector a . In (3), the second equality holds due to the chain rule, and we have used the facts that θ ∗ ( 1 ) = θ o and the convention ∇ ℓ f ( θ o ) = dℓ f ( z ) d z | z = θ ∗ ( 1 ) .

It is clear from (3) that assessing the influence of weight modification ϵ in unlearning requires deriving d [ ϵ ⊙ θ ∗ ( ϵ )] d ϵ . This necessitates the derivation of IG, d θ ∗ ( ϵ ) d ϵ , the gradient flow from the lowerlevel solution θ ∗ ( ϵ ) to the upper-level variable ϵ . Inspired by the implicit function approach for solving BLO problems [71], IG can be derived as applied to differentiating the parameterized arg min problem [72, 73]; see derivations in Appx.A . This leads to

where ∇ ϵ , θ ℓ r denotes the cross-variable second-order derivative of the bi-variate function ℓ r ( ϵ ⊙ θ ) w.r.t. the variables ϵ and θ , ∇ θ , θ ℓ r denotes the Hessian matrix of ℓ r w.r.t. the variable θ , -1

is the matrix inversion, diag( a ) represents the diagonal matrix with the diagonal vector a , and ∇ z ℓ r ( z ) ∣ ∣ z = ϵ ⊙ θ ∗ ( ϵ ) signifies the gradient of ℓ r w.r.t. its combined input argument z at z = ϵ ⊙ θ ∗ ( ϵ ) . In (4), the first equality holds due to the application of the implicit function theorem [72], and the second approximation is obtained under the diagonal Hessian assumption ∇ θ , θ ℓ r = γ I [71, 73], where γ > 0 serves as a tunable hyperparameter or is regarded as a Hessian diagonal estimate to compensate for the loss of the Hessian approximation.

Substituting IG (4) into (3), we obtain the analytical form of the unlearning sensitivity to ϵ :

where we obtained the derivative d [ ϵ ⊙ θ ∗ ( ϵ )] d ϵ in (3) using the chain rule and the diagonal matrix expression of IG in (4), and the second equality holds due to diag( a ) b = a ⊙ b . The formula (5) provides a principled framework for weight attribution, which evaluates the influence of weight perturbations ϵ in the unlearning performance, and considers both impacts of data to forget (encoded in ℓ f ) and data to retain (encoded in ℓ r ) in LLM unlearning.

To gain more insights into (5), we consider a single weight perturbation by specifying ϵ as ϵ = 1 + µ e i , where µ is the perturbation strength for the weight w i . Since the weight attribution process employs a Taylor expansion at ϵ = 1 in (3), its validity necessitates setting µ as a small perturbation. Let S i denote the attribution score of the i th weight. By substituting ϵ = 1 + µ e i into (5), we obtain

where [ a ] i denotes the i th entry of the vector a . In (6), the first term plays a more dominant role than the second term because µ represents a small weight perturbation, making µ 2 ≪ µ . Thus, we propose to drop the second term and simplify the weight attribution score as

where the constant µ is omitted without loss of generality, and the attribution score S i is determined by the two terms ① and ② that can be interpreted, respectively. In (7), the first term ① aligns with the weight pruning score SNIP [74], which characterizes the sensitivity of the forget loss to sparsifying the i th weight initialized by its pre-trained state. The second term ② accounts for the additional utility retention effect under the i th weight modification. Furthermore, the roles of these two terms ① and ② are regularized by the Hessian parameter γ in (4); See Remark 1 for its choice.

Remark 1: As will be evident later, our experiments reveal some interesting empirical findings that can guide the choice of γ , which we explain below. Recall from (4) that γ represents the Hessian diagonal estimate of the retain loss ℓ r . One rough but feasible approach to setting γ is to use a quasi-Newton method [75, 76], which approximates the Hessian diagonal by employing the element-wise product of the first-order gradients of ℓ r . Thus, we can use the corresponding gradient norm as an indicator to guide us to either increase or decrease the hyperparameter γ . We find that if the retain loss closely resembles the training loss ( i.e. , the retain set shares a similar distribution with the training set), then the pre-trained model θ 0 resides in the minima basin of the retain loss, resulting in small gradients and a small Hessian diagonal parameter γ . The fictitious unlearning over the TOFU dataset [16] belongs to the above scenario. By contrast, if the retain set is not representative of the training set, then we need a larger Hessian diagonal parameter choice for γ . The copyrighted information unlearning task on the Harry Potter book series dataset [26] falls into this scenario.

WAGLE : Weight attribution-guided LLM unlearning. By ranking the magnitudes of the attribution scores { S i } i in descending order, we then select the top ones and determine the subset of weights most influential in LLM unlearning. Let m S represent the weight selection mask, where [ m S ] i = 1 denotes the selection of the i th weight based on its attribution score and 0 otherwise. Given m S , we update only the partial model parameters in θ identified by m S , rather than the entire model. This modifies the LLM unlearning problem (1) to WAGLE :

where m S ⊙ θ +( 1 -m S ) ⊙ θ o encodes the modularity characteristics of the LLM for unlearning, decomposing the model weights into the optimized part m S ⊙ θ and the other part ( 1 -m S ) ⊙ θ o

that remains the same as the pre-trained weights. It is evident from (8) that incorporating weight attribution m S into LLM unlearning is strategic to specific unlearning algorithms. Therefore, we can implement WAGLE based on all existing methods (GradDiff, NPO, and PO) introduced in Sec. 3.

5 Experiment

5.1 Experiment Setups

Unlearning tasks, datasets, and models. To demonstrate the significance of weight attribution and the effectiveness of WAGLE , we conduct experiments on four LLM unlearning tasks. ① Fictitious unlearning on TOFU dataset [16]: It contains information about fictional authors for fine-tuning LLMs, and parts of these authors' profiles (with 10% forget ratio ) can be designated as the forget set. ② Malicious use prevention of LLMs in developing cyberattacks or bioweapons on WMDP dataset [25]: This benchmark assesses the ability to unlearn and prevent the generation of hazardous knowledge in biosecurity, cybersecurity, and chemical security. ③ Copyrighted information removal in WHP (Who's Harry Potter) task [26]: This pertains to the task of unlearning the Harry Potter books from LLMs. ④ Model detoxification ( DETOX ) on PKU-SafeRLHF dataset [77]: This aims to leverage LLM unlearning to prevent the generation of toxic content in response to inappropriate prompts from SafeRLHF. Model-wise, we use the LLaMA2-7B-chat [78] provided by the TOFU benchmark. For WMDP, we adopt the Zephyr-7B-beta model [79], consistent with the benchmark. For WHP, we utilize the LLaMA2-7B [78] fine-tuned on the Harry Potter book series. Finally, we employ the LLaMA2-7B for DETOX. See Appx. B.1 and Appx. B.2 for details.

Training setup. To obtain LLMs post-unlearning ( i.e. , unlearned LLMs), we first carry out the weight attribution method (7) to obtain the weight selection mask m S used in (8). Unless specified otherwise, the Hessian diagonal parameter γ in (7) is chosen to be a small value 10 -6 for TOFU and WMDPtasks and a large value 10 4 for WHP and 10 6 for DETOX, as guided by Remark 1. The sparsity ratio of m S is tuned for each task based on a greedy search, as exemplified in Fig. A1. Given the weight selection scheme, we then solve the optimization problem using its specific unlearning method: GradDiff [15], NPO [19], and PO [16], respectively. AdamW [80] is used as the default optimizer. It is worth noting that we set the utility regularization parameter λ as 1. In the implementation of PO, we use the reject-based answer as the targeted response over the forget set. See Appx. B.3 and Appx.B.4 for additional details.

Evaluation setup. We evaluate the performance of unlearned LLMs from unlearning efficacy ( UE ) and preserved model utility ( UT ). For the TOFU task, UE is assessed using four metrics. (1) Forget quality (FQ) quantifies the distinguishability between statistical measures of forgetting and retaining. We employ the Kolmogorov-Smirnov (KS) test to compare the truth ratios produced by the unlearned model on forget and retain sets, defining FQ as 1 -p -value obtained from the KS test. A higher FQ indicates better forgetting, characterized by the better distinguishability between forget data and retain data. (2) Membership inference attack (MIA) is evaluated by the area under the ROC curve using Mink % Prob [81] to detect if the provided text belongs to the training or testing set. We apply MIA to the forget set; thus, a higher MIA score indicates a higher confidence in predicting that the forget data point does not belong to the training set. (3) Forget accuracy (FA) refers to the accuracy of LLMs post-unlearning on the forget set. For ease of performance averaging, we also use 1 -FA to measure UE. Thus, a higher 1 -FA implies better unlearning. (4) Rouge-L recall is also measured over the forget set. A lower value corresponds to better unlearning. The metric 1 -Rouge-L is also used for ease of performance averaging. Next, we measure UT of unlearned LLMs by computing the accuracy and Rouge-L recall on the retain set, as well as on subsets related to real authors and world facts. Higher values in these metrics imply better utility retention. For the WMDP task, UE is measured using the benchmark-provided WMDP-Bio and WMDP-Cyber subsets. We use 1 -FA as the UE metric for each evaluation subset. In addition, UT is evaluated using zero-shot accuracy on the MMLU dataset [82]. For the WHP task, UE is evaluated by Rouge-L on both seen and unseen text completion instructions from the Harry Potter book series, with lengths of 300 tokens. UT is assessed using the Language Model Evaluation Harness [83], which computes perplexity (PPL) on the Wikitext dataset [84] and mean zero-shot accuracy across tasks. Additional evaluations include TruthfulQA [85]. For the DETOX task, UE is measured by the toxic scores from Toxic-BERT [86] under real toxic prompts [28] and the PKU-SafeRLHF test set [77]. Thus, the lower toxic scores imply better unlearning. The UT evaluation is the same as WHP. See Appx. B.5 for addition details.

Baselines. We demonstrate the effectiveness of our proposed WAGLE method by comparing it with the LLM unlearning baselines GradDiff [15], NPO [19], and PO [16]. These baselines are applied to the original pre-trained, dense model (referred to as Dense ) as well as their weight selection-based variants, including the randomly sparsified model (referred to as Random ), the weight magnitudebased pruned model (referred to as Magnitude ), the Wanda-enabled pruned model [65] (referred to as Wanda ), and the low-rank adaptation scheme (LoRA) [87]. Results are averaged over 3 random trials.

5.2 Experiment Results

LLMunlearning on TOFU. In Tab. 1 , we present the UE (unlearning efficacy) and UT (utility) performance of our proposed WAGLE when integrating weight attribution into different unlearning methods GradDiff, NPO, and PO. We also compare our performance with unlearning variants using different weight selection or adaptation schemes. For example, the term 'GradDiff + Magnitude' refers to the application of GradDiff to the magnitude-based pruned model through the optimization in (8). As we can see, under each unlearning method category, the incorporation of weight attribution consistently improves unlearning effectiveness, as evidenced by the rise in UE Avg. Utility-wise, although WAGLE does not always yield the best utility retention (as measured by UT Avg.), it consistently improves over all the dense model-based LLM unlearning methods. This suggests that the incorporation of weight attribution can improve UE while resulting in a graceful tradeoff with UT. Furthermore, we observe that NPO is a much more aggressive unlearning method, yielding the best unlearning efficacy but inevitably causing a larger degradation in model utility. By contrast, PO appears to be a more balanced unlearning method, achieving a better tradeoff between UE and UT.

LLM unlearning on WMDP. In Tab. 2 , we demonstrate the UE and UT performance of WAGLE on the WMDP benchmark. Recall that UE is measured by FA (forget accuracy) on the WMDP-Bio and WMDP-Cyber subsets provided by this benchmark, while UT is measured by the accuracy on the MMLU dataset. Unlike the TOFU task, PO for LLM unlearning is not considered for WMDP. This is because the forget set in WMDP is given as a set of plain texts, whereas PO requires conversational-style data for unlearning. Forced rejection on plain texts leads to over-forgetting of the irrelevant knowl-

edge involved in these texts when using PO. As we can see, WAGLE ( i.e. , ours) yields the best UE, as evidenced by the 'UE Avg.' metric. Consistent with the TOFU results, the incorporation of weight

attribution in WAGLE improves LLM unlearning compared to the 'Dense' variant. These results are consistent with findings from the TOFU dataset.

To illustrate how WAGLE enhances unlearning efficacy, we present generation examples from LLMs post-unlearning under the WMDP benchmark. Failed unlearning attempts, resulting in undesired outputs, are highlighted in red, while successful instances, producing desired unlearning outcomes, are marked in green. Traditional methods like GradDiff and NPO, when applied to the dense model, still select hazardous knowledge. However, integrating WAGLE with these methods consistently prevents the selection of such undesired answers, demonstrating WAGLE's improvement in unlearning performance as shown in Tab. 2. Additional visualizations are available in Tab. A4.

Q1 (Biology) : What is a crucial consideration when strategically modifying vector components to circumvent DNA order screening for viral vector development?

-Original: A: Maintaining vector stability and functionality.

-GradDiff + Dense: A: Maintaining vector stability and functionality.

-GradDiff + WAGLE: C: Utilizing identical vector components for all applications.

-NPO + Dense: A: Maintaining vector stability and functionality.

-NPO + WAGLE: D: Prioritizing speed over modification precision.

Q2 (Cyber) : What is the first step in setting up the OneDrive listener in Empire?

-Original: A: Type uselistener onedrive in Empire.

-GradDiff + Dense: A: Type uselistener onedrive in Empire.

-GradDiff + WAGLE: D: Obtain the AuthCode from the OAuth App.

-NPO + Dense: A: Type uselistener onedrive in Empire.

-NPO + WAGLE: D: Obtain the AuthCode from the OAuth App.

LLMunlearning on WHP and DETOX.

In

Tab. 3 , we compare the UE and UT performance of WAGLE with baselines in two additional unlearning tasks, WHP and DETOX. Here, we adopt PO as the unlearning method due to its effectiveness in striking the tradeoff between UE and UT. We observe that, similar to other unlearning tasks, the use of weight attribution in WAGLE improves unlearning effectiveness while preserving model utility compared to unlearning without using weight attribution. In addition to quantitative assessments, we also provide exam-

ples of the responses of LLMs post-unlearning across various tasks in Appx. D.

Exploring model fingerprint of LLM unlearning from weight attribution. Further, we examine which weights of an LLM (specifically LLaMA2-7Bchat) are attributed as influential for the unlearning. To this end, Fig. 2 presents the density of selected weights within each LLM module, including the selfattention (sa) components query (q), key (k), value (v), and the output layer (o) producing the final output from as. In addition to as, we also include input layer (in), layer normalization (ln), MLP components, and post attention (post) modules. Here, the overall weight selection ratio determined by weight attribution is set to 80%, and PO-based WAGLE is used for LLM unlearning on the TOFU dataset. For comparison, we also present the density of selected weights based on their magnitudes. It is evident that the den-

in.ln

sa.k

sa.o

sa.q

sa.v

Modules

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Density Each Module

1.00

0.80

0.81

0.80

1.00

0.82

0.76

0.81

0.77

1.00

0.93

0.87

0.90

1.00

0.46

0.92

0.49

0.90

Overall Density 80%

Magnitude

Ours

sity of weights chosen for unlearning shows a markedly different trend from that of magnitude-based selection. Notably, unlearning favors a higher selection of weights in sa.o and sa.v, as well as MLP layers. By contrast, less weights in sa.k and sa.q are influential. Our findings echo the importance of editing neurons in feed-forward networks [32, 88] and highlight that important weights are not merely restricted to key-value memories [30]. In addition, we present the layer-wise sparsity levels in Fig. A2. We observe that early-to-mid layers are important for unlearning.

Exploring the role of the Hessian diagonal hyperparameter γ in weight attribution. As discussed in Remark 1 of Sec. 4, it is critical but non-trivial to choose an appropriate Hessian diagonal parameter γ for weight attribution (7). One feasible method is to estimate its value using the gradient norm, as employed by the quasi-Newton method [75, 76]. However, this estimate could be rather rough if the retain loss does not resemble the training loss, meaning that the pre-trained model θ o , at which the gradient norm is evaluated, does not stay in the minima basin of the retain loss. And this may occur based on the context of LLM unlearning.

To demonstrate the critical role of γ , Fig. 3 presents the average UE performance of using the PO-based WAGLE versus γ/ GN , i.e. , the ratio of γ and the gradient norm (GN) of the retain loss at θ o , on TOFU and WHP datasets. As observed, UE improves as γ/ GN decreases on TOFU. This is not surprising, as TOFU has an accurate retain set, leading to a better Hessian diagonal estimate using GN. Thus, even the case of γ = GN suffices to improve UE. In addition, the alignment of the retain set with the training set also results in a relatively small gradient, making GN small accordingly. As a result, the choice of γ in TOFU is consistent with GN and favors a small value. By contrast, the best choice of γ for WHP favors a large value, as GN is no longer a reliable Hessian diagonal estimate, due to WHP not offering a very accurate retain set.

Computational efficiency of the unlearning process. First, as indicated by (7) - (8), the weight attribution mask can be computed offline using only first-order derivatives. As a result, generating a general unlearning mask for the TOFU dataset takes approximately 4 minutes on the Llama2-7Bchat model, as shown in Tab. 4 . Second, applying the mask during the unlearning process requires a similar running time across different unlearning methods. Given the total unlearning duration of 30 minutes, the time spent generating the attribution mask is relatively insignificant, affirming the efficiency of our method.

Examining weight attribution sparsity on unlearning. We find that enhancing LLM unlearning with weight attribu-

-9.4

-0.0

4.4

6.7

11.3

13.6

18.2

0.91

0.92

0.93

0.94

0.95

0.96

(WHP)

-5.1

0.0

1.8

13.3

15.6

20.2

22.6

log(

/GN)

0.72

0.73

0.74

0.75

0.76

0.77

(TOFU)

tion requires a non-oversparse weight selection scheme, typically between 80% and 95%. However, the best ratio varies across different unlearning methods. See Fig. A1 for results.

6 Conclusion

To improve the forgetting efficacy and utility retention ability of existing LLM unlearning methods, we provide a new perspective on LLM unlearning through weight attribution. Drawing inspiration from bi-level optimization (BLO), we propose a principled scoring framework to assess how adjustments to weights affect LLM unlearning. Utilizing the implicit gradient approach in BLO, we derive the closed-form solution for weight attribution. Integrating this weight attribution scheme into LLM unlearning, we develop the weight attribution-guided LLM unlearning method ( WAGLE ). Our extensive experiments demonstrate that WAGLE enhances unlearning performance across a range of LLM unlearning methods in diverse applications. See the discussions on limitations and broader impacts in Appx. E and Appx. F.

Bibliography

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223 , 2023.

Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787 , 2024.

Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy , pages 463-480. IEEE, 2015.

Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP) , pages 141-159. IEEE, 2021.

Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. A survey of machine unlearning. arXiv preprint arXiv:2209.02299 , 2022.

Chris Jay Hoofnagle, Bart van der Sloot, and Frederik Zuiderveen Borgesius. The european union general data protection regulation: what it is and what it means. Information & Communications Technology Law , 28(1):65-98, 2019.

Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems , 32, 2019.

Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P) , pages 303-319. IEEE, 2022.

Enayat Ullah, Tung Mai, Anup Rao, Ryan A Rossi, and Raman Arora. Machine unlearning via algorithmic stability. In Conference on Learning Theory , pages 4126-4142. PMLR, 2021.

Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems , 34:18075-18086, 2021.

Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9304-9312, 2020.

Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation. In International Conference on Learning Representations , 2024.

Chongyu Fan, Jiancheng Liu, Alfred Hero, and Sijia Liu. Challenging forgets: Unveiling the worst-case forget sets in machine unlearning. arXiv preprint arXiv:2403.07362 , 2024.

Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint arXiv:2310.10683 , 2023.

Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. Tofu: A task of fictitious unlearning for llms, 2024.

Bo Liu, Qiang Liu, and Peter Stone. Continual learning and private unlearning. In Conference on Lifelong Learning Agents , pages 243-254. PMLR, 2022.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024.

Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868 , 2024.

Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for llm unlearning. arXiv preprint arXiv:2404.18239 , 2024.

Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve gpt-3 after deployment. arXiv preprint arXiv:2201.06009 , 2022.

Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. Can we edit factual knowledge by in-context learning? arXiv preprint arXiv:2305.12740 , 2023.

Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579 , 2023.

Pratiksha Thaker, Yash Maurya, and Virginia Smith. Guardrail baselines for unlearning in llms. arXiv preprint arXiv:2403.03329 , 2024.

Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218 , 2024.

Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in llms, 2023.

Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems , 35:27591-27609, 2022.

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint arXiv:2009.11462 , 2020.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems , 35:17359-17372, 2022.

Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. Advances in Neural Information Processing Systems , 36, 2024.

Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. Unlearning bias in language models by partitioning gradients. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 6032-6048, 2023.

Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons in pretrained language models. arXiv preprint arXiv:2310.20138 , 2023.

Yang Liu, Mingyuan Fan, Cen Chen, Ximeng Liu, Zhuo Ma, Li Wang, and Jianfeng Ma. Backdoor defense with machine unlearning. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications , pages 280-289. IEEE, 2022.

Meghdad Kurmanji, Peter Triantafillou, and Eleni Triantafillou. Towards unbounded machine unlearning. arXiv preprint arXiv:2302.09880 , 2023.

Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. arXiv preprint arXiv:2303.07345 , 2023.

Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. arXiv preprint arXiv:2303.17591 , 2023.

Yihua Zhang, Yimeng Zhang, Yuguang Yao, Jinghan Jia, Jiancheng Liu, Xiaoming Liu, and Sijia Liu. Unlearncanvas: A stylized image dataset to benchmark machine unlearning for diffusion models. arXiv preprint arXiv:2402.11846 , 2024.

Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402.10058 , 2024.

Kongyang Chen, Zixin Wang, Bing Mi, Waixi Liu, Shaowei Wang, Xiaojun Ren, and Jiaxing Shen. Machine unlearning in large language models. arXiv preprint arXiv:2404.16841 , 2024.

Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models. arXiv preprint arXiv:2402.15159 , 2024.

Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. Graph unlearning. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security , pages 499-513, 2022.

Eli Chien, Chao Pan, and Olgica Milenkovic. Certified graph unlearning. arXiv preprint arXiv:2206.09140 , 2022.

Kun Wu, Jie Shen, Yue Ning, Ting Wang, and Wendy Hui Wang. Certified edge unlearning for graph neural networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2606-2617, 2023.

Yi Liu, Lei Xu, Xingliang Yuan, Cong Wang, and Bo Li. The right to be forgotten in federated learning: An efficient realization with rapid retraining. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications , pages 1749-1758. IEEE, 2022.

Anisa Halimi, Swanand Kadhe, Ambrish Rawat, and Nathalie Baracaldo. Federated unlearning: How to efficiently erase a client in fl? arXiv preprint arXiv:2207.05521 , 2022.

Ruinan Jin, Minghui Chen, Qiong Zhang, and Xiaoxiao Li. Forgettable federated linear learning with certified data removal. arXiv preprint arXiv:2306.02216 , 2023.

Anvith Thudi, Hengrui Jia, Ilia Shumailov, and Nicolas Papernot. On the necessity of auditable algorithmic definitions for machine unlearning. In 31st USENIX Security Symposium (USENIX Security 22) , pages 4007-4022, 2022.

Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. arXiv preprint arXiv:1911.03030 , 2019.

Alexander Becker and Thomas Liebig. Evaluating machine unlearning via epistemic uncertainty. arXiv preprint arXiv:2208.10836 , 2022.

Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang. Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7766-7775, 2023.

Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. arXiv preprint arXiv:2108.11577 , 2021.

Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from llms? objectives for defending against extraction attacks. ICLR , 2024.

Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835 , 2024.

Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504 , 2022.

Clark Barrett, Brad Boyd, Elie Bursztein, Nicholas Carlini, Brad Chen, Jihye Choi, Amrita Roy Chowdhury, Mihai Christodorescu, Anupam Datta, Soheil Feizi, et al. Identifying and mitigating the security risks of generative ai. Foundations and Trends® in Privacy and Security , 6(1):1-52, 2023.

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning , pages 1885-1894. PMLR, 2017.

Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning , pages 15630-15649. PMLR, 2022.

Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296 , 2023.

Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. arXiv preprint arXiv:2205.09329 , 2022.

Chengcheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selection in deep learning. In International Conference on Database and Expert Systems Applications , pages 181-195. Springer, 2022.

Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-based continual learning. arXiv preprint arXiv:2106.01085 , 2021.

Yihua Zhang, Yimeng Zhang, Aochuan Chen, Jiancheng Liu, Gaowen Liu, Mingyi Hong, Shiyu Chang, Sijia Liu, et al. Selectivity drives productivity: Efficient dataset pruning for enhanced transfer learning. Advances in Neural Information Processing Systems , 36, 2024.

Max Marion, Ahmet Üstün, Luiza Pozzobon, Alex Wang, Marzieh Fadaee, and Sara Hooker. When less is more: Investigating data pruning for pretraining llms at scale. arXiv preprint arXiv:2309.04564 , 2023.

Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695 , 2023.

Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems , 36:21702-21720, 2023.

Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning , pages 22137-22176. PMLR, 2023.

Ajay Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang, and Yinfei Yang. Compressing llms: The truth is rarely pure and never simple. arXiv preprint arXiv:2310.01382 , 2023.

Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. In Advances in Neural Information Processing Systems , volume 33, pages 15834-15846, 2020.

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695 , 2023.

Yihua Zhang, Prashant Khanduri, Ioannis Tsaknakis, Yuguang Yao, Mingyi Hong, and Sijia Liu. An introduction to bilevel optimization: Foundations and applications in signal processing and machine learning. IEEE Signal Processing Magazine , 41(1):38-59, 2024.

Stephen Gould, Basura Fernando, Anoop Cherian, Peter Anderson, Rodrigo Santa Cruz, and Edison Guo. On differentiating parameterized argmin and argmax problems with application to bi-level optimization. arXiv preprint arXiv:1607.05447 , 2016.

Yihua Zhang, Yuguang Yao, Parikshit Ram, Pu Zhao, Tianlong Chen, Mingyi Hong, Yanzhi Wang, and Sijia Liu. Advancing model pruning via bi-level optimization. In Advances in Neural Information Processing Systems , 2022.

Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340 , 2018.

Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv preprint arXiv:2305.14342 , 2023.

Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2(3-4):157-325, 2016.

Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems , 36, 2024.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of lm alignment, 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310.16789 , 2023.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https://zenodo.org/records/ 10256836 .

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.

Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 , 2021.

Laura Hanu and Unitary team. Detoxify. Github. https://github.com/unitaryai/detoxify, 2020.

Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. arXiv preprint arXiv:2310.20624 , 2023.

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913 , 2020.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67, 2020.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 2924-2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1300. URL https://aclanthology.org/N19-1300 .

Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop , pages 177-190. Springer, 2005.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 , 2019.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM , 64(9):99-106, 2021.

François Chollet. On the measure of intelligence. arXiv preprint arXiv:1911.01547 , 2019.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789 , 2018.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , pages 7432-7439, 2020.

Appendix

A Implicit Gradient (IG) Derivations

Since θ ∗ ( ϵ ) is the lower-level solution, it satisfies the stationarity condition of the lower-level problem of (2). This leads to

where for rotational simplicity, we omit the dependence of θ ∗ on ϵ . By the implicit function theorem [72], we then take the derivative of (A1) w.r.t. the variable ϵ . This leads to

where ∇ ϵ , θ ℓ r denotes the cross-variable second-order derivative of the bi-variate function ℓ r ( ϵ ⊙ θ ) w.r.t. the variables ϵ and θ , and ∇ θ , θ ℓ r denotes the Hessian matrix of ℓ r w.r.t. the variable θ .

Based on the diagonal Hessian assumption ∇ θ , θ ℓ r = 1 γ I , we can then derive the IG from (A2) below

We note that in the bi-variate function ℓ r ( ϵ ⊙ θ ) , the variables ϵ and θ are coupled through a bi-linear relationship. This special structure of the bi-variate function allows us to further simplify (A3). Such a simplification has been provided in [73, Prop. 1], which yields the IG formula in (4).

B Additional Experimental Details

B.1 Model Configurations

The fine-tuned version of LLaMA2-7B-chat, provided in [16] for the TOFU dataset, is chosen as the pretrained model on TOFU task. For the WMDP task, we select the original Zephyr-7B-beta as the pretrained model. For the WHP task, we fine-tune LLaMA2-7B using LoRA on the complete Harry Potter book series, adopting a learning rate of 1 × 10 -4 with the AdamW optimizer. For the DETOX task, we selected LLaMA2-7B as the foundational model for our study [78].All experiments were conducted on two NVIDIA RTX A6000 GPUs. Each experiment takes approximately 5 minutes per 100 steps.

B.2 Dataset Configurations

In the Tofu dataset, we select a forget set comprising 10% of the entire training set, with the remaining data points forming the retain set. For the WMDP task, we use the unlearning dataset provided in the WMDP original benchmark, which includes a forget set of plain texts relevant to bio/cyber knowledge and unrelated texts as the retain set. In the WHP task, we extract 200 data chunks, each 512 tokens long, from the original Harry Potter book series [26] as the forget set. For the DETOX task, we sample 200 negative samples from the PKU-SafeRLHF [77] train set as the forget set. To ensure the preservation of utility in the copyright removal and model detoxification tasks, we select C4 [89] as the retain set.

B.3 Rejected-based Answers in PO

As for the PO method, we need to specify the rejection-based answers. Tab. A1 demonstrates partial of our rejection-based answers used in PO.

B.4 Unlearning Configurations

In this section, we will introduce the basic hyperparameter settings for the unleanring methods in our experiments. For the TOFU dataset, both PO and GradDiff are conducted over 5 epochs, while NPO is conducted over 4 epochs. In the WMDP task, the maximum training steps are set to 500 for both NPO and GradDiff. For the WHP task, 5 epochs are adopted, and for the DETOX task, 10 epochs are used. The learning rate is grid-searched between [1 × 10 -6 , 5 × 10 -5 ] for each task and method. The parameter λ is set to 1 for each method across all tasks, except for NPO + Dense on the WMDP task where λ is set to 4. We adopted AdamW [80] as the optimizer for unlearning.

B.5 Evaluation Configurations

Zero-shot tasks. We utilized the LM Evaluation Harness [83] to conduct zero-shot accuracy assessments across various tasks, including BoolQ [90], RTE [91], HellaSwag [92], Winogrande [93], ARC-Challenge [94], ARC-Easy [94], OpenBookQA [95], and Piqa [96]. We report mean accuracy to evaluate the utility preservation of the unlearned LLM.

Text completion instructions. As for the WHP task, we will maintain a text-completion instructions, which contains two part, the first part will be seen during unlearning, while the remaining part will be utilized for testing the unlearning effectivness on the unseen text-completion instructions. Tab. A2 demonstrates the completion instructions we utilized.

C Additional Experiments

C.1 Examining Weight Attribution Sparsity on Unlearning

To further investigate the influence of weight attribution sparsity on unlearning and identify an optimal sparsity range, Fig. A1 illustrates how changes in weight attribution density affect unlearning efficacy on the TOFU dataset. Initially, it is evident that the weight attribution scheme should not be excessively sparse, ideally ranging between 80% and 95%. Furthermore, the optimal ratio varies across different unlearning methods.

C.2 Exploring Importance of Different Layers for Unlearning from Weight Attribution

To further examine which layers of an LLM (specifically finetuned LLaMA2-7B-chat model on TOFU) are influential for unlearning, Fig. A2 presents the density of selected weights within each

60%

80%

95%

99%

0.948

0.952

0.956

0.960

0.964

NPO

NPO

60%

90%

95%

99%

0.425

0.450

0.475

0.500

0.525

0.550

GradDiff

GradDiff

60%

80%

95%

99%

Density

0.71

0.72

0.73

0.74

0.75

0.76

PO

PO

transformer layer. The overall weight selection ratio is set to 80%, and PO-based WAGLE is utilized for unlearning on the TOFU dataset. We also display the density of selected weights based on their magnitudes. It is evident that unlearning predominantly favors the early-to-mid layers, where the density is high. This observation aligns with the findings in [30].

0

5

10

15

20

25

30

Layer Index

0.65

0.70

0.75

0.80

0.85

Density

Ours

Magnitude

D Visualization

D.1 Visualization Examples for TOFU Task

Tab. A3 presents visualization examples of responses generated by various unlearned models. The results reveal that adding our WAGLE weight attribution improves unlearning performance, offering enhanced unlearning efficacy and superior utility preservation. For instance, WAGLE enables GradDiff to avoid generating correct information in the forget set, whereas the dense model still produces the correct answer, which is undesirable. Additionally, with the aggressive unlearning method NPO, WAGLE assists in outputting correct information on the retain set, while the dense model tends to generate nonsensical words. These findings are consistent with the quantitative results presented in Tab. 1.

D.2 Visualization Examples for WMDP Task

Since the WMDP unlearning evaluation is based on single-choice questions, we present the answer choices selected by each model in Tab. A4. The results indicate that incorporating our WAGLE weight attribution enhances unlearning performance on the WMDP test benchmark. For instance, GradDiff and NPO applied to the Dense model are still able to select the correct answers for questions related

to hazardous knowledge. However, when WAGLE is integrated with these unlearning methods, the unlearned models do not select undesired answers. These findings are also consistent with the quantitative results shown in Tab. 2.

D.3 Visualization Examples for WHP Task

Tab. A5 presents an example responses from various unlearned models on WHP task. The results reveal that PO+ WAGLE does not generate copyright-related content, whereas PO on the dense model produces texts highly correlated with copyright-related content. These findings demonstrate that WAGLE effectively improves unlearning in the WHP task.

D.4 Visualization Examples for DETOX Task

Tab. A6 presents several examples from different unlearned models. Results reveal that PO+ WAGLE can avoid generate toxic responses compared with PO on the dense model. This also aligned with the quantitative results in Tab. 3, that weight attribution can enhance LLM unlearning performance.

E Limitations

By exploring the roles of weight attribution in LLM unlearning, we demonstrate that there exists a model fingerprint that can be utilized to simultaneously improve unlearning efficacy and preserve model utility. However, we also find that the optimal ratio of weight selection post-attribution is not universal across different unlearning algorithms and tasks, as evidenced by the varying densities of selected weights. Although appropriately identified weight sparsity enhances LLM unlearning, determining this sparsity automatically and efficiently requires further research. Furthermore, a precise Hessian diagonal estimate is lacking, which is essential for simplifying both computation and hyperparameter selection in weight attribution. This also requires further research.

F Broader Impacts

The impact of this research is multifaceted. On the positive side, weight attribution connects the modularity characteristics of LLMs with their unlearning capabilities. This connection enables users to efficiently and effectively unlearn from LLMs, enhancing data privacy and compliance with regulations. Such advancements can foster greater trust and wider adoption of LLMs in sensitive applications. On the negative side, the techniques developed could potentially be misused to selectively erase historical data or knowledge, raising ethical concerns. Thus, it is crucial that the use of unlearning technologies be governed by strict ethical standards to prevent abuse. We hope our work can inspire further innovations to build safe, secure, and trustworthy AI.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We summarized our contributions in Sec. 1, and for detail can be found in Sec. 3, 4, and 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Discussions on the limitations can be found in Sec. E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide assumptions and proofs in Sec. 4 and Sec. A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experiment setups can be found in Sec. 5 and Sec. B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We attached our code in the supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Experiment settings and details can be found in Sec. 5 and Sec. B .

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The results reported in this work were averaged over 6 independent trials due to our limited computing resources on TOFU and WMDP tasks.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Details of hardware and corresponding computation time can be found in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have made sure to preserve anonymity.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The discussion on broader impacts can be found in Sec. F .

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper is designed for LLM unlearning, without publishing any datasets or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original paper that produced the code package and dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have included the code used for experiments in the supplementary material, ensuring that the experiments can be easily reproduced by following the provided instructions.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve any human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve any human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces WAGLE (Weight Attribution-Guided LLM unLEarning), a novel method for enhancing the effectiveness of large language model (LLM) unlearning. The key contributions in my view are: \n\n1) A bi-level optimization framework for weight attribution in LLM unlearning. \n2) A closed-form solution for calculating weight attribution scores. \n3) Integration of weight attribution into existing LLM unlearning methods. The authors evaluate WAGLE across multiple unlearning tasks and benchmarks, demonstrating improved unlearning efficacy while maintaining model utility compared to baselines.\n\nSTRENGTHS:\nStrengths:\n- Novel approach to LLM unlearning using weight attribution\n- Theoretical grounding in bi-level optimization\n- Comprehensive evaluation across multiple unlearning tasks and benchmarks\n- Compatibility with existing unlearning methods (GradDiff, NPO, PO)\n- Insights into which model components are most influential for unlearning\n\nWEAKNESSES:\nWeaknesses:\n- Limited theoretical justification for why weight attribution improves unlearning\n- Computational complexity of the method not thoroughly discussed\n- Lack of comparison to some recent unlearning methods\n- Limited exploration of potential negative effects or failure cases\n- Hyperparameter sensitivity (e.g., choice of γ) could be explored more thoroughly\n\nQUESTIONS:\nQuestions:\n1. How does the computational complexity of WAGLE compare to baseline unlearning methods?\n2. Have you explored the potential for negative transfer or catastrophic forgetting when using WAGLE?\n3. How sensitive is the method to the choice of hyperparameters, particularly the Hessian diagonal estimate γ?\n4. Could you provide more intuition on why certain model components (e.g., self-attention output and value) are more important for unlearning?\n5. Have you considered applying WAGLE to other model architectures beyond LLaMA and Zephyr?\n6. How does WAGLE perform on even larger language models (e.g., 13B or 70B parameter models)?\n7. Could the weight attribution method be used for other purposes beyond unlearning, such as model compression or transfer learning?\n8. Have you explored the potential for combining WAGLE with other unlearning techniques, such as data augmentation or adversarial training?\n\nLIMITATIONS:\nLimitations:\nThe authors have addressed some limitations, but the paper could improve by:\n\n1. Addressed:\n\n- Sensitivity to the Hessian diagonal hyperparameter γ\n- Performance across different unlearning tasks and methods\n- Comparison to several baseline methods\n\n2. Could be better addressed:\n- Computational complexity and scalability to larger models\n- Potential negative effects or failure cases\n- Broader applicability beyond the tested models and tasks\n\n3. Missing:\n- Discussion of potential biases introduced by the method\n- Exploration of privacy implications\n- Analysis of the method's robustness to adversarial attacks\n- Environmental impact of the additional computational requirements\n\nSuggestions for improvement:\n1. Include a dedicated section on limitations and future work\n2. Discuss broader ethical implications and potential misuse of the technology\n3. Consider looking into relearning papers which I think are related but missing in this draft e.g. https://arxiv.org/pdf/2401.01814", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents WAGLE, a weight attribution method for unlearning. The main\ngoal of WAGLE is to augment existing unlearning methods by identifying\ninfluential parameters, and restricting existing methods to the\ninfluential parameters. The authors focus their attention to approximate unlearning for large language models (LLMs).\n\nTo find influential parameters, the authors propose to model the approximate unlearning problem as a bi-level optimization (BLO) problem, and then use classic tools from the BLO literature to derive an efficient solution.\n\nThe authors evaluate WAGLE on three different unlearning methods. As baselines, the authors consider three pruning techinques together with  LoRA, as well as no pruning.\n\nSTRENGTHS:\nThe authors provide an extensive and thorough evaluation of their proposed method. In particular, they consider 1) multiple base unlearning algorithms, 2) multiple well-motivated pruning baselines, 3) multiple unlearning efficacy heuristics, and 4) multiple downstream tasks to evaluate model utility.\n\nAnother strength on the experimental side is that experiments are performed on Llama2-7B; the size of the model alone displays the scalability of the proposed method.\n\nWEAKNESSES:\n- Motivation: \"Yet, effective unlearning also requires a sense of locality, which involves\nidentifying the sub-components of the LLM (i.e., a subset of weights in this\nwork) that are crucial for the unlearning task, while minimally impacting the\nmodel’s original utility.\" (lines 167-169)\n\nI am not sure I'm convinced by this assertion, especially given that it is provided without any supporting evidence. In particular, the null hypothesis for me here would be that unlearning a given subset of the training data would require modifying most (if not all) model parameters. A simple example can be observed in linear regression, where we have a closed form for model updates via the Woodbury-Sherman-Morrison formula. Even in that simple scenario, there is no notion of sparsity in the unlearning update. Given that this statement is central to the entire paper, I would like to see an argument backing it up.\n\n- Results are improved only for one UE metric (FQ), even when comparing against the trivial \"random pruning\" baseline. In particular, on all three unlearning methods in Table 1, the performance of \"random\" and WAGLE on MIA,  1-FA, 1-Rouge-L seems within statistical error (see Q below about providing SEs).\n\nQUESTIONS:\n- Figure 1: \" Yet, when applied to LLM unlearning, the effectiveness of weight sparsity remains elusive\" (lines 179-180)\nFrom Figure 1, it seems like at 0% sparsity, the model is completely useless (0% model utility), and at 5% sparsity (obtained with Wanda pruning),  there is some balance between model utility and unlearning efficacy. What happens at more granular levels of sparsity between 0% and 5%? Does efficacy degrade gracefully, allowing for a satisfactory trade-off between utility and efficacy?\n\nTo me, this question is important, because the existence of a \"good\" sparsity level here would mean that the unlearning landscape for LLMs follows that for classification models, where, as the authors note, sparsity has been shown to be helpful.  Thus, such a finding would make weight attribution methods unnecessary for this application, given their additional complexity and computational overhead.\n\n- Can the authors provide wall-clock time estimates for WAGLE and the baselines they consider?\n- Can the authors provide standard errors in Table 1?\n- Is there a connection with influence functions\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the correlation between model weights and the efficacy of unlearning processes in large language models (LLMs). It introduces a framework called WAGLE (Weight Attribution-Guided LLM Unlearning), which elucidates the relationship between the influence of specific weights and the LLM unlearning performance. WAGLE is grounded in a meticulous analysis from the influence function perspective. Moreover, the paper evaluates the framework's performance across a broad set of benchmarks and applications specific to LLM unlearning, demonstrating its superior effectiveness over traditional dense unlearning methods.\n\nSTRENGTHS:\n1.\tThis paper is pioneering in the domain of machine unlearning for large language models (LLMs), providing a formal definition and derivation that demonstrates how weight attribution contributes to unlearning processes. Unlike prior studies on unlearning in computer vision, which were primarily intuitive, this paper’s methodical approach through formal derivation introduces a significant novelty to the field.\n\n2.\tThe insights offered by the proposed framework are substantial, particularly in identifying critical modules within LLMs that are pivotal for effective unlearning, as detailed in Figure 2. This not only deepens our understanding of the inner workings of LLMs but also highlights specific areas for targeted unlearning strategies. This may also helps understanding the memory of LLMs.\n\n3.\tThe clarity and organization of the paper enhance the reader's comprehension and engagement. \n\n4.\tIt features a comprehensive set of experiments across diverse tasks, benchmark datasets, and models, convincingly demonstrating the efficacy of the proposed weight attribution methodology in facilitating LLM unlearning.\n\nWEAKNESSES:\n1.\tMy primary concern lies with the clarity of the evaluation metric, Membership Inference Attack (MIA), as utilized on the TOFU dataset. The paper does not clearly define whether a lower MIA score, which would indicate a reduced likelihood of the forget set being recognized as part of the training set, is preferable. Clarification on this metric and its desired outcome would enhance the understanding of the results presented.\n\n2.\tThe experiments primarily focus on a model size of approximately 7 billion parameters. It would be beneficial if the authors could explore how the unlearning performance varies with changes in model size. This would provide insights into the scalability and applicability of the proposed method across different model architectures.\n\n3.\tAnother concern is the computational efficiency of the unlearning process, specifically the time cost associated with implementing weight attribution. Providing comparative data on the time required for unlearning with and without weight attribution would significantly strengthen the paper by highlighting the practical implications of the proposed method.\n\nQUESTIONS:\n1.\tCould the authors provide additional clarification regarding the implementation and interpretation of the Membership Inference Attack (MIA) evaluation metric on the TOFU dataset?\n\n2.\tCould the authors explore how the efficacy of the weight attribution method in facilitating LLM unlearning varies across models of different sizes?\n\n3.\tCould the authors provide details on the time cost associated with their unlearning method, both with and without the implementation of weight attribution?\n\nLIMITATIONS:\nThe limitation concerning the selection of an appropriate sparsity level for weight attribution is a recognized challenge within the broader pruning community and remains an unresolved question.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces WAGLE (Weight Attribution-Guided LLM Unlearning), a novel method for large language model (LLM) unlearning that identifies influential weights for the unlearning process while considering the retain loss. The authors evaluate WAGLE on a diverse set of unlearning benchmarks, demonstrating improved forgetting performance and competitive task performance compared to baseline methods.\n\nSTRENGTHS:\n* Novel approach: WAGLE integrates weight analysis with unlearning, providing a new perspective on LLM unlearning.\n* Comprehensive evaluation: The method is tested on various unlearning tasks and datasets, showcasing its versatility.\n* Improved performance: WAGLE consistently enhances forgetting ability while maintaining or sometimes improving task performance on retain sets.\n* Insights into model architecture: The study reveals that early-to-mid layers are important for unlearning, and the method tends to avoid selecting weights from self-attention mechanisms.\n* Comparison with existing methods: The authors benchmark WAGLE against various direct optimization approaches for forget/retain loss, providing context for its performance.\n* The bi-level optimization formulation is interesting and could be built on by future methods.\n\nWEAKNESSES:\n* Discussion of limitations and future is deferred to the appendix, it would be good to include this in the main paper\n* It isn't clearly articulated *why* this method improves existing unlearning methods. Is it faster or easier to train? Is it hard to get improvements on forgetting performance? \n* Concerns about evaluation: It seems like the evaluation is not quite appropriate, especially if it is hard to get improvement in forgetting performance. It is not clear to me how the tradeoffs between the retain and forget loss are set. Given the nature of the evaluation, it seems important to at least discuss how this was tuned (apologies if I missed it). In an ideal world, the methods would be evaluated across a sweep of this variable. It would be very nice to know how much retain loss the comparison methods need to give up in order to match the forget loss performance of WAGLE.\n\nQUESTIONS:\nPlease respond to the concerns raised in the weaknesses section to provide additional context for your choices or suggest ways to address the issue. Thank you.\n\nLIMITATIONS:\nThere is some discussion of this in the appendix, but it should be included in the main text. A bit more discussion of the societal ramifications of unlearning (potential benefits or risks) would also be nice.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}