Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond

Authors: Qizhou Wang, Jin Peng Zhou, Zhanke Zhou, Saebyeol Shin, Bo Han, Kilian Q Weinberger
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: LLM Unlearning

Abstract:
Large language models (LLMs) should undergo rigorous audits to identify potential risks, such as copyright and privacy infringements. Once these risks emerge, timely updates are crucial to remove undesirable responses, ensuring legal and safe model usage. It has spurred recent research into LLM unlearning, focusing on erasing targeted undesirable knowledge without compromising the integrity of other, non-targeted responses. Existing studies have introduced various unlearning objectives to pursue LLM unlearning without necessitating complete retraining. However, each of these objectives has unique properties, and no unified framework is currently available to comprehend them thoroughly. To fill the gap, we propose the metric of the G-effect, quantifying the impacts of unlearning objectives on model performance from a gradient lens. A significant advantage of our metric is its broad ability to detail the unlearning impacts from various aspects across instances, updating steps, and LLM layers. Accordingly, the G-effect offers new insights into identifying drawbacks of existing unlearning objectives, further motivating us to explore a series of candidate solutions for their mitigation and improvements. Finally, we outline promising directions that merit further studies, aiming at contributing to the community to advance this critical field.

Decision: accept (poster)
Reviewer scores: 8, 6, 3, 8, 5, 6

Reviews and discussion:
<thread>
creation date:
1730445388464

thread_type:
official_review

<thread_object>
creator: reviewer_okm6

<content>
summary:
This paper focus on analysis of existing unlearning methods in the scope of LLMs. The authors propose a general toolkit for analysis of existing methods, which is named G-effect. G-effect analyzes unlearning methods from both forgetting and retaining. Experiments show the rationality of the proposed G-effect, and the effectiveness of several proposed variants.

strengths:
- The paper is well structured.
- The authors propose G-effect to analyze existing unlearning methods from the perspectives of both forgetting and retaining effects.
- Analysis of G-effect somehow accords with the experimental results. Experiments show the effectiveness of the proposed variants.

weaknesses:
- What is the rationale of designing WGA? The authors did not clearly state the reasons of choosing such format.
- I do not clearly see the challenges of coming up such a general toolkit for analysis of various unlearning methods.
- The claim in Line 081 is somehow too strong. How did the authors conclude that KL is the optimal choice?
- The presentations of Figure 2,3,4 are somehow hard to read. Putting the legends near the figures might be better.

questions:
no
</content>
</thread_object>

</thread>
<thread>
creation date:
1730459376671

thread_type:
official_review

<thread_object>
creator: reviewer_uxuc

<content>
summary:
This paper tackles the concept of unlearning in Large Language Models (LLMs), focusing on the removal of learned knowledge while preserving the overall model integrity. The authors propose a new metric, the G-effect, to quantify the impact of unlearning objectives on model performance.

strengths:
- Investigating unlearning dynamics is an interesting and understudied area.

- Empirical results back up some of the authors claims.

weaknesses:
See questions and comments.

questions:
Legal compliance has been an oft touted reason for unlearning. I have yet to see any compelling argument that unlearning passes any kind of legal bar for data removal. I’m not even sure what the requirements are for meeting e.g. GDPR requirements. I am concerned that the unlearning community may be operating in a vacuum, failing to actively engage with the legal community to determine the practical applications and implications of unlearning. 

– 

“It is common the cases where shallow layers are more affected than deeper layers during unlearning. It suggests that general knowledge, predominantly encoded in shallow layers (Patil et al., 2023), undergoes substantial alterations“ Why not freeze shallow layers during unlearning using methods like [1]?

[1] Goel, Shashwat, et al. "Towards adversarial evaluations for inexact machine unlearning." arXiv preprint arXiv:2201.06640 (2022).

– 

“Unlearning compromises retention. Although conceptually existing (cf., Section 3), current
unlearning objectives all fail to retain the overall model performance when unlearning.“ This is not a new contribution. Many prior works have made this discovery. I also do not understand the difference between this statement and the next contribution: “ Excessive unlearning is harmful. An excessive extent of unlearning has severe impacts such that the deterioration in common model responses can outweigh improvements in unlearning.”

–

I do not believe the contributions highlighted are particularly interesting. The first three listed are already well known phenomena (the first being the motivation for prior unlearning techniques in [1]). The fifth contribution has already been studied. I do think the fourth contribution: “Risk weighting is powerful. Prioritizing certain beneficial points is justified to be effective for unlearning. However, there still exists a large space to further refine risk weighting mechanisms.” is interesting, but I’m not sure this amounts to a significant contribution to the field.

–

The unlearning objectives designed for concept removal assumes that one can pinpoint and incorporate the specific data requiring removal into the unlearning dataset denoted as D_u. I do not believe this is realistic in general.

–

“Removal. The performance on the unlearning dataset Du should significantly deteriorate, i.e., R(Du; θu) ≫ R(Du; θo), revealing effective unlearning on data targeted to be erased.“ Throughout this paper, privacy is highlighted as a use case for unlearning. This is not a good removal metric for anything to do with private information. For example, imagine I want to unlearn “Alice’s phone number is 12345”, and I do this by gradient ascent, up to the point where loss(“Alice’s phone number is 12345”) >> loss(“Alice’s phone number is *any other number*”), then this becomes an oracle, and reconstruction or identification of private information becomes easy. Multiple prior works have discussed this and how to define unlearning for privacy.
Similarly, “We consider the practical objective of erasing targeted knowledge as much as possible (Liu et al., 2024), diverging from the classical definition of machine unlearning (Bourtoule et al., 2021) that seeks to make models behave as if they were trained without the targeted data. Our goal is more suitable for LLM unlearning, driven by the need to eliminate content that poses privacy and copyright concerns, with the understanding that more thorough elimination leads to more favorable behaviors.“ I believe any unlearning definition for privacy that does not try to align with a model that never trained on that data is bad for privacy. It also not clear what kind of privacy we should be concerned about. Reconstruction? Identification of membership?

—

“Sadly, merely comparing performance provides limited insights into understanding the underlying mechanisms.“ What's the definition of “understand” here? Can you formalize it?

—

“Generally speaking, the G-effect compares the gradients of the unlearning objective Lu and the risk metric R. If the gradients of Lu align in similar directions to R, model updating based on Lu
is capable to enhance model performance measured by R, an obvious alternative of R(D; θu) −
R(D; θo) to measure the performance change“ Apologies, I’m a bit confused here. Why not directly optimize R then if it is differentiable?

–

In Figure 1, it’s not clear how the intersection actually maps to successful unlearning. It would be really useful to give some examples (or quantitative results) for various points on the sphere, showing that unlearning is most successful at the intersection.

–

“Due to the high costs in fully computing the G-effects, we focus on experiments based on 5% TOFU fictitious unlearning (Maini et al., 2024) with Llama-2-7B (Touvron et al., 2023a) (cf. Appendix B). All the methods will run for 5 epochs, totaling about 60 steps.
“ This seems like a significant barrier for using G-effect. Also does this mean you only use a dataset of 40 examples? This seems quite small.
–

“The G-Effects across Unlearning Steps” I don’t see what the useful insights are here. I believe we could have instead measured the NLL of examples and gotten an equally useful signal. Overall I found it difficult to assess if the G-Effect is a more useful metric than directly measuring losses over unlearning.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730696636044

thread_type:
official_review

<thread_object>
creator: reviewer_zecj

<content>
summary:
The paper beings with a bird's eye view of unlearning, highlighting its importance and the main drawbacks of the current approach. The paper suggests G-effect, a method to better examine the behavior and properties of unlearning during learning. This is done by taking into account the gradients with respect to unlearning two objectives and computing their inner-product with the gradients taken for unlearning. This method has the potential to serve as an evaluation criteria of existing and new unlearning approaches, while also providing insights to explain the success and failure of unlearning approaches, beyond the simple black-box evaluation.

This method is interesting and powerful, as it was able to generate five observations on unlearning: Unlearning affects shallow layers more, Unlearning compromises retention, Excessive unlearning is harmful, Risk weighting is powerful, Regularization is important.

The paper then explores multiple unlearning approaches; GA, NPO, PO and RMU using the G-effect.

The paper address the current limitation of the approach and suggests new promising directions.

strengths:
The paper is insightful and well written. It provides a lot of context to the reader, explains the main drawbacks of the current approach and the importance of G-effect in addressing and examining unlearning from a scientific point-of-view, as opposed to blackbox trial and error which stems from intuition. The paper provide a very thorough analysis of G-effects on multiple unlearning approaches.

weaknesses:
Not sure

questions:
Is something off with Figure-3 or its caption?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722140673

thread_type:
official_review

<thread_object>
creator: reviewer_sbc5

<content>
summary:
Various approaches have been proposed in the literature to perform LLM unlearning. Existing unlearning evaluation metrics compare LLM performance before and after unlearning. To provide more insights into understanding the underlying mechanisms, this paper aims to quantify the impacts of unlearning objectives on model performance from a gradient perspective. To do that, this paper proposes the toolkit of the G-effect: 
1. unlearning G-effect: the capability to decrease model performance on unlearning data 
2. retaining G-effect: the capability to maintain/enhance performance on other data 

G-effect compares the gradient of the unlearning objective and the risk metric that assesses the LLM performance:
 - If the gradients of the unlearning objective align in opposite directions to the risk metric, model updating based on the unlearning objective is capable of decreasing model performance measured by the risk
 - If the gradients of the unlearning objective align in similar directions to the risk metric, model updating based on the unlearning objective is capable of enhancing model performance measured by the risk metric

This paper quantifies the degree of such similarity between unlearning objective gradients and the risk metric gradients using their dot products.

strengths:
1. Gradient-based analysis of the G-effect enables a better understanding of unlearning approaches including
	- examining the dynamics of unlearning procedures
	- explore the impacts of particular layers or data points involved during unlearning
2. Using G-effects to assess Gradient Ascent, Weighted Gradient Ascent, Negative Preference Optimization, Preference Optimization, Representation Misdirection for Unlearning and Regularization. Their study concludes several interesting findings. For example among 3 representative regularization terms, namely gradient difference, KL divergence and representation retention, KL  is superior for retention.

weaknesses:
1. This paper (Section 4) examines G-effects of each unlearning objective independently and in isolation to other learning objectives. Results are also shown and discussed in separate figures and parts of the paper. Studying G-effect of each learning objective in isolation, raises the concern regarding the comparability of G-effect values across various unlearning objectives and approaches. 
 	- Why empirical analysis of each unlearning approach is shown and discussed in separate parts of the paper?
	- Are G-effect values comparable across different unlearning approaches? are values comparable and why?
	- Can the proposed G-effect rank unlearning approaches?
 2. Section 5 and its Table 1 provide a comprehensive comparison of various unlearning approaches using TOFU unlearning dataset for the removal of fictitious author profiles from LLMs finetuned on them. However, this comparison uses only existing metrics: forget quality, model utility, and PS-scores, and does not report the proposed G-effects.  
 	- Why G-effects are missing in this section?
 	- How do G-effect values correlate with metrics presented in Table 1?
 	- Why are the order and ranking of unlearning objectives different across different removal and retention metrics?

3. G-effects need access to intermediate checkpoints during unlearning, especially given the pattern of values in for example Figure 3 (i.e., a peak and then flat close to zero). How does this limit the applicability of the proposed metric? 
 		
4. The G-effect definition uses model checkpoints at different time steps and does not directly take into account the risk and unlearning of the initial model. 	
	- Why does this make sense?
	- Is this why you need to do accumulative?
	- what does the G-effect at each unlearning step mean?
	- what does accumulation across unlearning steps mean?
	- What does pick mean in Figure 3? Should we stop after that step to have an effective unlearning? what would be the benefit of continuing? is 0 G-effect value the limitation of your method?

5. Some of the claims are not completely supported. For example, the claim "In terms of the unlearning G-effects, it indicates that the unlearning strength of NPO is weaker; however, for the retaining G-effects, it suggests that NPO better preserves the model integrity." As an initial step, I would link it to numbers in Table 1.
6. Membership inference attacks are a common approach in the literature for evaluating the removal capability of unlearning approaches [MUSE]. However, this paper does not report the success of membership inference attacks. How the unlearning G-effect is compared to the success of MIA? Are they aligned?

[MUSE] Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models, 2024.

questions:
I have outlined questions for each weakness above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730753403863

thread_type:
official_review

<thread_object>
creator: reviewer_43ov

<content>
summary:
Large language models (LLMs) need thorough audits to uncover risks like copyright or privacy violations. When such risks are identified, prompt updates are essential to filter out inappropriate outputs, ensuring that models remain lawful and safe for use. This concern has fueled new research into LLM unlearning, aimed at selectively erasing specific unwanted knowledge while maintaining the reliability of other, unaffected responses. In this context, the authors of this paper introduce the G-effect toolkit, which measures how unlearning objectives influence model performance through the analysis of gradients.

strengths:
The paper is clearly written and straightforward to understand.

weaknesses:
It is uncertain if the findings discussed in this paper can be applied to scenarios where LLM unlearning aims to eliminate the effects of contaminated data.

questions:
Overall, this paper is quite intriguing. The authors introduce an innovative framework called G-effect, designed to measure the influence of unlearning objectives on model performance through gradient analysis. However, there are some points for the authors to consider:

1) Unlearning is typically applied in two scenarios: first, for privacy reasons, where users seek to remove content related to privacy. This is the scenario explored in this paper. However, unlearning can also be driven by security concerns, such as when an LLM is compromised by data poisoning, and the system needs to mitigate the impact of various poisoning attacks. It remains unclear if the conclusions drawn in this paper can be extended to this second scenario.

2) G-effect appears to share similarities with influence functions and Shapley value methods. The authors should clarify these connections.

3) Research, such as [A], suggests that LLM unlearning may not always fully eliminate the influence of data that users wish to remove. It is not evident whether the proposed G-effect accounts for this limitation.

[A] Machine Unlearning Fails to Remove Data Poisoning Attacks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731024895403

thread_type:
official_review

<thread_object>
creator: reviewer_mskc

<content>
summary:
This paper discusses the importance of auditing large language models (LLMs) to identify potential risks and the need for timely updates to remove bad responses (unlearning being the solution in consideration here). The authors propose a unified framework to comprehend various unlearning objectives, introducing the concept of G-effect to analyze and compare different unlearning methods. Insights from G-effect analysis prompt focus on a two-step approach to update original LLM parameters to get unlearned one- using a removal and retention step to deteriorate performance on the unlearning dataset while maintaining performance on the rest of the data.

strengths:
-   The paper tackles an important topic in the field of natural language processing, specifically the need for LLM unlearning to remove targeted information without destroying model integrity.
-   The authors propose a novel framework to analyze and compare different unlearning objectives, introducing the concept of G-effect to measure the impact of unlearning objectives on targeted or common data. G effect combines the influence of both goals of a good unlearning objective- removal and retention into a single metric.
-   The paper provides a comprehensive discussion of different unlearning objectives, including gradient ascent, negative preference optimization, PO, and representation misdirection for unlearning. The paper introduces advanced unlearning objectives, such as WGA and WTNPO, which set new state-of-the-art results in unlearning objectives.
- Comparison between objectives Gradient Ascent, NPO, etc. and effect of regularization is thoroughly studied.
The paper introduces advanced unlearning objectives, such as WGA and WTNPO, which set new state-of-the-art results in unlearning objectives.

weaknesses:
- Ablation study on the effect of size of data to be forgotten on the effectiveness of G-effect. 
- Effect of 'harder' or 'easier' examples to forget on G-effect. 
- Figures plotting the G-effect have their axes not labeled.

questions:
- Is it possible to see the effect of choice parameters like number of samples to be forgotten on the G-effect ? 
- Other ablation studies are also appreciated- for instance choice of network architecture, sensitivity to optimizer params, etc. It wasn't clear if these results are averaged over multiple runs. 
- I am curious to see how G-effect varies for samples that are harder to forget. This may even help explaining the relationship between influence functions and data values with ability to unlearn.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732439526128

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We would like to express our deep gratitude to all the reviewers for their time and insightful feedback. We appreciate the recognition of our analysis as **comprehensive** and **insightful**  (MskC, SBC5, zecJ, and oKM6). We are also grateful for the comments that our empirical results **support** our claims (MskC, uXUc, oKM6) and that our paper is **clearly written** (43oV, zecJ, oKM6).

Endorsed by most reviewers, our work contributes meaningfully to the field by introducing a general toolkit named the G-effect, which facilitates the in-depth study of unlearning dynamics. Our contributions, leveraging the corresponding analysis, can be broadly summarized in twofold.  

- We carefully analyze several popular unlearning objectives, including GA and NPO, revealing their **key unlearning mechanism** as well as their **failure modes**.

- We mitigate the drawbacks of existing unlearning objectives by **designing their improved versions**, supported by extensive experimental results that demonstrate their practical effectiveness.

We also thank the reviewers for their comments to conduct more experiments and improve clarity. Additionally, we are pleased to further clarify two points that may have caused slight confusion for some reviewers (SBC5 and uXUc). 

- The G-effect **extends beyond the role of a metric** for performance assessment. It is intended for researchers seeking an in-depth understanding of unlearning dynamics, beyond what standard unlearning metrics like PS and MU provide.

- This paper particularly focuses on the unlearning goal of full removal. However, we acknowledge **its similarity with another representative goal** named influence removal, where our findings could benefit such a broader community. We also recognize the unique challenges posed by influence removal and intend to further adapt our G-effect for it in future work. 

We have responded to each reviewer's valuable critiques in our individual responses and look forward to continuing this valuable discussion during the conference discussion period!
</content>
</thread_object>

</thread>


Metareview:
metareview:
- This paper introduces G-effect, a gradient-based toolkit for evaluating unlearning methods by analyzing their dual impact: deteriorating performance on targeted undesirable data (unlearning G-effect) and maintaining performance on other data (retaining G-effect). The G-effect enables a detailed examination of various unlearning algorithms and generates five general observations: unlearning affects shallow layers more, unlearning compromises retention, excessive unlearning is harmful, risk weighting is powerful, and regularization is important.

- This paper is well-structured and easy to follow. It conducts insightful analyses, considering both unlearning and retention, and provides a comprehensive exploration of various unlearning objectives (e.g., GA, NPO, PO, RMU). To address the limitations of existing unlearning methods, the authors further propose advanced objectives such as WGA and WTNPO. Experiments demonstrate the effectiveness of the proposed variants.

- However, it is uncertain whether the proposed framework, general observations, and unlearning variants are applicable to other scenarios (e.g., data poisoning) or align with practical legal compliance requirements. The empirical analysis is limited to small datasets (~40 samples) without results averaged over multiple runs. Additionally, there is a lack of clarity regarding the relationship between G-effect and mechanisms like influence functions or Shapley value methods, as well as insufficient analysis of gradient behaviors that could provide insights beyond performance changes.