Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs

Authors: Sungmin Cha, Sungjun Cho, Dasol Hwang, Moontae Lee
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Machine Unlearning, Large Language Models, Low-rank Adaptation

Abstract:
Large Language Models (LLMs) have demonstrated strong reasoning and memorization capabilities via pretraining on massive textual corpora. However, this poses risk of privacy and copyright violations, highlighting the need for efficient machine unlearning methods that remove sensitive data without retraining from scratch. While Gradient Ascent (GA) is commonly used to unlearn by reducing the likelihood of generating unwanted content, it leads to unstable optimization and catastrophic forgetting of retrained knowledge. We find that combining GA with low-rank adaptation results in poor trade-offs between computational cost and generative performance. To address these challenges, we propose Low-rank Knowledge Unlearning (LoKU), a novel framework that enables robust and efficient unlearning for LLMs. First, we introduce Inverted Hinge Loss, which suppresses unwanted tokens while maintaining fluency by boosting the probability of the next most likely token. Second, we develop a data-adaptive initialization for LoRA adapters via low-rank approximation weighted with relative Fisher information, thereby focusing updates on parameters critical for removing targeted knowledge. Experiments on the Training Data Extraction Challenge dataset using GPT-Neo models as well as on the TOFU benchmark with Phi-1.5B and Llama2-7B models demonstrate that our approach effectively removes sensitive information while maintaining reasoning and generative capabilities with minimal impact. Our implementation can be found in https://github.com/csm9493/efficient-llm-unlearning.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729495701045

thread_type:
official_review

<thread_object>
creator: reviewer_aaxs

<content>
summary:
The paper focus on the problem of unstable optimization, catastrophic forgetting, and computational cost from Gradient Ascent for LLM unlearning, and propose two novel techniques, including the Inverted Hinge Loss and Fisher Information weighted initialization of LoRA adapters, for robust and efficient unlearning for LLMs. Experiments on two tasks with different LLMs show that the proposed methods enable faster and more stable LoRA-based LLM unlearning.

strengths:
1. The paper makes a good contribution to knowledge unlearning of LLMs through improving tuning stability and efficiency with Inverted Hinge Loss and Fisher-Initialization of Low-rank Adapters, respectively. The proposed method is valuable of managing unnecessary forgetting and unbounded optimization in typical Gradient Ascent strategies.
2. The paper is generally well-written, particularly in formula derivation and clear explanation about IHL and FILA solve the weaknesses of GA in Section 3.3 and Section 3.4.
3. The experiments and analysis of the paper is comprehensive, with great illustration on performance evaluation using high quality charts.

weaknesses:
1. In Introduction Section (Line 51-53), you mention GA and its shortcomings, I think a better way of writing here would be providing a brief overview of 2-3 other key knowledge unlearning approaches beyond GA, and summarize 1-2 common shortcomings across these methods that motivate the proposed approach. GA should be only one of those existing typical methods.

2. In Introduction Section (Line 76), you mention the application of LoRA to LLM unlearning remains unexplored, however, there are some existing studies using LoRA for LLM unlearning, including Practical Unlearning for Large Language Models (https://arxiv.org/abs/2407.10223) and Machine Unlearning in Large Language Models (https://arxiv.org/abs/2405.15152v1). It would be better to briefly summarize (in 1-2 sentences each) how LoRA was used for unlearning in these two papers, and then explain how their proposed approach differs or improves upon these methods.

3. Some important content is missing. In Introduction Section, a lack of clear summarization of contributions in the paper, making readers difficult to capture the important points of the study. Besides, in Related Work Section, a brief comparison between your proposed method and other relevant studies should be presented to better emphasize the advantages of your work. 

4. In Section 3.3 (Line 223-227), you should provide a more detailed explanation of how you arrived at these hypotheses. There is still a gap between GA motivation and its weaknesses. To make the illustration more convincible here, a better way would be providing a specific example or mathematical derivation showing how the GA loss function leads to one of the stated problems (e.g., unbounded optimization or inefficient gradient updates).

5. In Section 3.4 (Line 265), a basic and brief description of Fisher Information is necessary here for better understanding of the reason you employ it to address the importance quantification you mentioned before.

6. In Table 1, to make the table more readable, the data of important results should be highlighted via bolding the best result in each row or using color coding to show relative performance between methods, in order to show either your FILA LoRA performs much better than traditional LoRA, or it can approach the performance of full fine-tuning.

7. There are some writing flaws:
* Some sentences are too long to follow and comprehend, such as Line 89-93.
* In Line 419-421, there are two "Second" in these two sentences, making them difficult to be understood.
* The capitalization in the text should be more consistent. For instance, you use lowercase "l" in "Inverted Hinge loss" at Line 20 and Line 161, but uppercase "L" in "Inverted Hinge Loss" at Line 82. All uppercase would be better.

questions:
1. In Introduction Section (Line 72-74), can you explain more about the reason that low-rankness can be beneficial in stabilizing optimization and preventing catastrophic forgetting? Clearer illustration would be better here.

2. In Table 1, why you record results from running different epochs? Does it mean the method reaches the optimal with these epochs?

3. In Experiments Section, why different LLMs are used for those two tasks? Have you evaluated more popular and larger LLMs such as Llama3.1-8B? I suggest giving explanation of the strategy and purpose of model selection.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729503275023

thread_type:
official_review

<thread_object>
creator: reviewer_sxtu

<content>
summary:
The authors identify the limitations of current unlearning methods (e.g., Gradient Ascent (GA)), which can lead to unstable optimization and catastrophic forgetting of retrained knowledge. To overcome these challenges, the paper introduces two novel techniques for robust and efficient unlearning:

1. **Inverted Hinge loss (HIL):** This new loss function suppresses unwanted tokens while maintaining fluency by boosting the probability of the next most likely token.

2. **Fisher-Initialization of Low-rank Adapters (FILA):** Developed through low-rank approximation weighted with relative Fisher information, this method focuses updates on parameters critical for removing targeted knowledge.

The paper demonstrates the effectiveness of these techniques through experiments on the Training Data Extraction Challenge dataset using GPT-Neo models and on the TOFU benchmark with Phi-1.5B and Llama2-7B models. The proposed approach successfully removes sensitive information while maintaining the reasoning and generative capabilities of the models with minimal impact on performance.

In summary, this paper provides innovative solutions to the drawback of GA and demonstrates the effectiveness of the solutions.

strengths:
### Originality
- This paper points out the shortcomings of Gradient Ascent (GA) by analyzing its inverse. 
- This paper proposes two strategies to improve these shortcomings. 
- This paper demonstrates the effectiveness of their improvements on two datasets.

### Clarity
- The structure of this paper is clear, and most of the content is explained clearly.

### Significance
- This paper provides insights into knowledge unlearning through the analysis of Gradient Ascent (GA).

weaknesses:
- This paper lacks the state-of-the-art knowledge unlearning baselines (such as [1][2]). Although the main goal of the paper is to address the shortcomings of GA, incorporating the state-of-the-art knowledge unlearning for comparison would make it more convincing.

- Some descriptions are not clear enough. For example, lines 221-223 should include more explanation for the reasons. The authors should explain in detail why GA increases the prediction score for all other tokens $v \neq x_t$ in the vocabulary.

- From the experimental results, when only IHL is used, the performance is worse than the original GA. Does this contradict the paper's claim that IHL is designed to address the shortcomings of GA and the analysis of derivatives of GA?

- The paper devotes too much content to the background of knowledge unlearning in the Abstract and in the first paragraph of the Introduction. Since knowledge unlearning is a common problem, I believe it is unnecessary to describe it in such detail. The main content should focus on describing the research conducted in this paper. Specifically, Figure 1 should illustrate the proposed approach rather than the knowledge unlearning problem.

[1] Zhang R, Lin L, Bai Y, et al. Negative preference optimization: From catastrophic collapse to effective unlearning[J]. arXiv preprint arXiv:2404.05868, 2024.

[2] Gao C, Wang L, Weng C, et al. Practical unlearning for large language models[J]. arXiv preprint arXiv:2407.10223, 2024.

questions:
What is the reason for deriving the unlearning mechanism of GA from the formulas in lines 217-220?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730033374383

thread_type:
official_review

<thread_object>
creator: reviewer_t4et

<content>
summary:
The paper works on machine unlearning in LLMs, particularly focusing on the challenges of removing specific data instances from a model's memory without retraining from scratch. The authors propose two strategies: Inverted Hinge Loss (IHL) and Fisher-Initialization of Low-rank Adapters (FILA). IHL is designed to replace the unbounded negative cross-entropy loss in gradient ascent with a more stable and efficient loss function. FILA aims to initialize low-rank adapters in a way that prioritizes the removal of unwanted information and accelerates the unlearning process. Extensive experiments validates that the proposed methods significantly outperform existing baselines in efficiency and post-unlearning performance.

strengths:
1. Authors analyze the derivatives of GA and highlight its shortcomings, the motivation is clear and the theoretical foundation strengthens the rationale for the proposed methods.
2. The introduction of IHL addresses the instability issues of GA by focusing gradient updates on a minimal number of viable replacements for the ground-truth token. This results in a more controlled and stable unlearning process.
3. The proposed strategies are effective. The authors evaluate the methods on multiple datasets and multiple model sizes. This comprehensive evaluation demonstrates the robustness and generalizability of the proposed methods.

weaknesses:
The intuition and connection between the proposed methods, IHL and Fisher-Initialization of FILA, appear somewhat weak. This makes the paper feel like it is stacking two separate tricks rather than offering a unified and coherent approach. A more systematic linkage between these methods would enhance the overall coherence and impact of the paper.

questions:
How robust are the proposed methods to changes in the data distribution of the forget set? For instance, if the forget set contains highly diverse or outlier data, would the unlearning process still be effective?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730447764056

thread_type:
official_review

<thread_object>
creator: reviewer_ehhm

<content>
summary:
The paper proposes a framework to remove sensitive information from LLMs without retraining them from scratch. Recognizing the limitations of common unlearning methods like Gradient Ascent (GA), which risks instability and unintended forgetting, the authors introduce two new techniques. The Inverted Hinge Loss (IHL) method enhances stability by suppressing unwanted tokens with the next most likely alternative, while the Fisher-weighted Initialization of Low-rank Adapters (FILA) uses Fisher information to initialize LoRA adapters, selectively targeting parameters associated with unwanted information to optimize unlearning. This dual approach was evaluated on the Training Data Extraction Challenge and TOFU benchmark with models such as GPT-Neo, Phi-1.5B, and Llama2-7B, achieving efficient unlearning with minimal loss to the model’s reasoning and generative capabilities, and demonstrating improved performance over existing methods.

strengths:
1. The motivation is clearly explained. 
2. Extensive experiments have been conducted to prove the effectiveness of the proposed methods. 
3. The theoretical analysis strengthens the rationale of the proposed methods.

weaknesses:
1. One of the most significant contributions of this paper is the proposal of Inverse Hard Loss (IHL), which claims to increase the probability of the second-best token only. However, it is not clear why IHL does not affect the probability of other tokens. Based on the definition of IHL in Lines 233, the probability of all other tokens is impacted. As such, IHL can only address problem 1 (Line 224) but cannot address problems 2 and 3 of GA (Lines 224 ~ 226).
2. In Figures 3 and 5, the unlearning performance of employing only IHL (represented in green) does not outperform the GD baseline (depicted in blue), which undermines the effectiveness of IHL. 
3. The main results only use GPT-neo models, which are old models. It is better to use more recent models like Llama and Mistral models to make it more practically useful. It is also inconsistent to use different models for main results and analysis. 
4. There are no ablations studies for the following settings: 1) full parameter fine-tuning with IHL; 2) LoRA + FILA only; 3) GD + LoRA + FILA.

questions:
1. In Figure 3, why are some data points missing? 
2. It is better to add legends in Figures 3 and 4 to improve the clarity.
3. It is better to define “Model Utility” within the paper instead of referring readers to other papers.
4. For the Hinge loss equation in Line 233, since the probability p(.) is in the range of (0,1), the second item within max() function is always larger than 0, right? If so, IHL is to reduce the probability of true tokens but to increase the probability of other tokens, right?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732056464206

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response to All Reviewers

comment:
We thank all reviewers for your time and commitment made into reviewing our work. We are deeply encouraged by the overall positive feedback on our valuable motivation towards stable and cost-efficient LLM unlearning [EhHm, T4eT, sxtU, AAXs], strong theoretical foundation [EhHm, T4eT, sxtU], and comprehensive experimentation with promising results [EhHm, T4eT, AAXs].

In light of reviewer-specific questions, our responses and clarifications can be found below in each respective comment. We are in the process of revising our manuscript to reflect the reviewers' comments, and will upload the revised version with a reminder soon. Should any additional questions arise, please share with us and we would be happy to discuss further. 

Thank you again for your service.

Sincerely,

Authors of Submission11013.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732349614871

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Manuscript Available for Further Review

comment:
We would like to inform the reviewers that we have updated our manuscript for further review. As our discussion phase is closing within a few days, we respectfully ask the reviewers to review our revised manuscript, and let us know in case of any additional questions or concerns.

For your reference, the revised portions within the manuscript can be found in blue-colored text. Below is our summary of revisions:

1. Writing
    - **Lines 50-75 and 83-87** [sxtU, AAXs]: We have added references to other LLM unlearning methods and brief discussions on them in the Introduction section. 
    - **Lines 102-107** [AAXs]: We have added a summary of the main contributions of our paper.
    - **Lines 159-161** [AAXs]: We have added a brief comparison between our study and other relevant studies.
    - **Lines 216-227 and 242-255** [EhHm, sxtU, AAXs]: We have added further details on how the gradients of GA reflects its issues, and how our IHL formulation resolves those issues from a probabilistic perspective.
    - **Lines 258-269** [T4eT]: We have revised the Motivation paragraph of Section 3.4 to clarify the linkage between IHL and FILA.
    - **Lines 270-273** [AAXs]: We have added explanations on the mathematical and intuitive definition of Fisher information.
    - **Lines 420-424** [EhHm, sxtU]: We have clarified the observation that even without FILA, IHL outperforms GD in many cases with both full-parameter and LoRA-based tuning.
    - **Lines 462-465** [EhHm]: We have added details on how the Model Utility metric is measured in TOFU. 
    - **Writing flaws** [AAXs]: We thank the reviewer again for spotting these errors. We have corrected the errors in the revision. Additionally, overly long or verbose sentences have been revised for clarity and conciseness, particularly in the Introduction and Related Work sections.
2. Figures and Tables
    - **Figure 1** [sxtU]: We have revised Figure 1 to contain specific illustrations of our IHL and FILA methods in addition to the overall LLM unlearning pipeline.
    - **Table 1** [AAXs]: We have denoted the best performance as bold for each column, and also included color-coded changes in performance after unlearning.
    - **Figures 3 and 4** [EhHm]: We have added legends to both figures and made the $\times$ markers denoting cases where unlearning was unsuccessful larger for better visibility.
3. Additional Experimental Results
    - **IHL with full-finetuning on TDEC (Table 1)** [EhHm]: We have added results from running full-parameter unlearning with IHL on the TDEC dataset. We can see that while GA and GD leads to significant loss in generative performance especially with smaller models (GPT-Neo-125M), IHL exhibits superior stability, minimizing the performance gap with the base model consistently across all GPT-Neo models.
    - **GD+FILA Results on TOFU (Figure 5)** [EhHm]: For completeness, we have added results using FILA with GD to our TOFU experiments, and made the color-coding consistent with Figures 3 and 4 for better readability. Results indicate that the model utility degrades quickly with GD+FILA, as observed in our TDEC experiments.
    - **Comparison vs. KL, DPO, and NPO in TOFU (Figure 7 in Appendix)** [sxtU]: We have run additional experiments on TOFU with three existing baselines: KL [A], DPO [B], and NPO [C]. We found that our IHL+FILA consistently outperforms all three baselines, as all three methods lead to significant degradation in model utility.

[A] [Maini et al., TOFU: A Task of Fictitious Unlearning for LLMs. arXiv 2024.](https://arxiv.org/abs/2401.06121)

[B] [Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023](https://arxiv.org/abs/2305.18290)

[C] [Zhang et al., Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning. arXiv 2024.](https://arxiv.org/abs/2404.05868)
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper addresses the challenge of removing sensitive information from Large Language Models (LLMs) without retraining, proposing two techniques: Inverted Hinge Loss (IHL) and Fisher-weighted Initialization of Low-rank Adapters (FILA). IHL aims to stabilize unlearning by focusing on the next most likely token, while FILA uses Fisher information to initialize LoRA adapters, selectively targeting parameters associated with unwanted information to optimize unlearning. The methods were tested on GPT-Neo, Phi-1.5B, and Llama2-7B models, showing efficient unlearning with minimal performance loss. All reviewers agree on the importance of the work and mention that the provided methodology is clearly justified theoretically and empirically. Moreover, the authors have put the effort to modify the manuscript based on the provided reviews and address all their comments. There are suggestions for improvement including clearer explanations (i.e., connection between the two methods are unclear), more recent model evaluations, and better scaling analysis.