Unified Parameter-Efficient Unlearning for LLMs

Authors: Chenlu Ding, Jiancan Wu, Yancheng Yuan, Jinda Lu, Kai Zhang, Alex Su, Xiang Wang, Xiangnan He
Primary area: interpretability and explainable AI
Keywords: Large Language Model Unlearning; Machine Unlearning; Influence Function

Abstract:
The advent of Large Language Models (LLMs) has revolutionized natural language processing, enabling advanced understanding and reasoning capabilities across a variety of tasks. Fine-tuning these models for specific domains, particularly through Parameter-Efficient Fine-Tuning (PEFT) strategies like LoRA, has become a prevalent practice due to its efficiency. However, this raises significant privacy and security concerns, as models may inadvertently retain and disseminate sensitive or undesirable information. To address these issues, we introduce a novel instance-wise unlearning framework, LLMEraser, which systematically categorizes unlearning tasks and applies precise parameter adjustments using influence functions. Unlike traditional unlearning techniques that are often limited in scope and require extensive retraining, LLMEraser is designed to handle a broad spectrum of unlearning tasks without compromising model performance. Extensive experiments on benchmark datasets demonstrate that LLMEraser excels in efficiently managing various unlearning scenarios while maintaining the overall integrity and efficacy of the models.

Decision: accept (poster)
Reviewer scores: 8, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730524530641

thread_type:
official_review

<thread_object>
creator: reviewer_nwjj

<content>
summary:
This paper introduces LLMEraser, a unified framework for parameter-efficient unlearning in large language models (LLMs), specifically tailored to address privacy and security concerns in domain-specific fine-tuning. The framework utilizes influence functions to perform instance-wise unlearning tasks such as instance removal, query modification, and response correction. LLMEraser allows unlearning without requiring full model retraining, and has demonstrated efficacy in preserving model performance across various unlearning tasks.

strengths:
1. The paper is written clearly and is easy to follow, with clearly presented formulations.
2. The taxonomy of unlearning tasks is well-defined, providing clarity on the types of unlearning scenarios addressed. 
3. Extensive experiments are conducted across diverse unlearning tasks with both LLMs and MLLMs, covering recommendation and multimodal relation mining applications. 
4. LLMEraser accelerates the computation of the inverse Hessian-vector-product in the influence function, enabling efficient implementations. This improvement is valuable for LLM applications, where computational efficiency is a growing concern.

weaknesses:
1. While LLMEraser reduces the need for retraining, the memory-intensive computation of inverse Hessian-vector products remains demanding. This requirement may limit scalability, particularly for very large models or environments with limited GPU memory.
2. The method’s reliance on first-order Taylor expansion in influence functions can introduce estimation errors. A lack of detailed error analysis makes it difficult to assess the impact of these errors, especially in tasks requiring high unlearning precision.
3. Limited generalizability is a concern, as the LLM experiments primarily focus on LLMs for recommendation (LLM4Rec). Broader evaluation across more diverse LLM unlearning tasks would strengthen the claims.
4. Key experimental details, such as PEFT configurations and the number of trainable parameters, are not fully provided. These details are essential for evaluating the method's complexity and efficacy across different settings.

questions:
1. Could the authors provide a detailed analysis of estimation errors to demonstrate their impact on performance? Given the importance of precision in unlearning tasks, such an analysis would help establish whether the errors introduced by the method are indeed within an acceptable range and do not significantly detract from the efficacy of LLMEraser in high-precision unlearning scenarios.

2. The LLM4Rec unlearning tasks are different from those unlearning tasks used in the baseline paper. Could the authors provide performance results on additional unlearning tasks more closely aligned with those proposed in other baselines? Additionally, how do the LLM4Rec tasks reflect the generalizability of LLMEraser across diverse unlearning scenarios? 

3. While LoRA was used in the experiments, it would be helpful to know the specific experimental setup, including the number of trainable parameters. Since computational complexity is directly related to the number of trainable parameters and rank values in LoRA, could the authors clarify these settings and present results across different ranks? Additionally, could they specify the sample sizes used in unlearning experiments, as sample size likely affects the overall complexity?

4. While the method demonstrates computational efficiency, I am concerned about memory efficiency, particularly for broader applications. Could the authors provide an analysis of the memory cost and specify whether LLMEraser could run on a lower-memory GPU than an Nvidia A100? If the method is limited to high-memory GPUs, this could hinder its practical application. A comparison of memory usage with baseline methods is also important to clarify this point.

I would be happy to engage with the authors to help improve the presentation of the method and evaluation, but my concerns are not insignificant. Clarifications would need to resolve my questions in order for my score to improve.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730589044355

thread_type:
official_review

<thread_object>
creator: reviewer_8xjc

<content>
summary:
This paper studies machine unlearning for parameter-efficient fine-tuning (PEFT) of large language models (LLM). The authors first propose a framework called LLMEraser for three different unlearning tasks: instance removal, query modification, and response correction. The core of LLMEraser is based on the influence function, based on which the algorithm can minimize the effect of the removed or modified samples. Some experiment results are presented to show the effectiveness of the proposed algorithm.

strengths:
1) The paper shows the motivation and derivation of the proposed algorithm/framework.
2) The proposed algorithm demonstrates that it can effectively maintain good performance on different tasks, including recommendation and relation mining tasks.
3) The proposed algorithm is also shown to have high efficiency compared to the existing algorithms.

weaknesses:
1) All the experiments show how well the models can perform after unlearning a certain amount of training samples. However, it seems the paper does not present or compare how effectively the algorithms can let the model "forget" those training samples, which is the original goal of the unlearning algorithm.
2) As PEFT is fine-tuning the model, it is unclear how to distinguish the influence of a sample when it is in the fine-tuning training set or the pre-training training set. When the sample was also used in the pre-training set, it is also unclear whether it is reasonable to require the PEFT to eliminate those 
3) The experiment settings in the paper do not include the generative tasks of LLMs.
4) Some minor typos, such as "handling handle" near the end of page 9.

questions:
1) How effectively can the proposed algorithm let the model forget the samples?
2) How effectively can the proposed algorithm maintain the model performance on generative tasks?
3) If a sample is also used in the pre-training, can it still be unlearned? 
4) How can we tell whether the unlearning (or failure to unlearn) is on the PEFT module but not on the pre-trained model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701317485

thread_type:
official_review

<thread_object>
creator: reviewer_8gyq

<content>
summary:
This paper introduce a novel instance-wise unlearning method that utilizes influence functions to adjusting parameters for unlearning tasks. Specifically, the method can remove specific sample from the training dataset, adjust input tokens in user queries, as well as correct model response. Experimental results show that the method can perform unlearning tasks with high accuracy while ensuring efficiency.

strengths:
1. The problem is important and interesting. The three unlearning tasks are practical and meaningful in real-world settings.

2. The unlearning approach edits model parameters based on influence functions, which avoids finetuning or re-training the models

3. The evaluations are comprehensive. 

4. The writing is clear and easy to follow.

weaknesses:
My concern is mainly about evaluation datasets. 

1. Although the paper claims to address privacy concerns in LLMs, the authors primarily evaluated the method using tabular datasets for recommendation tasks and a multimodal dataset for a classification task. While these tasks are representative, they are relatively simple and straightforward. Although the authors applied prompt engineering to those data, the training data are still very similar. 

2. In real AI applications, LLMs are used for text generation more often, and the training data consists of documents from various sources and may contain sensitive information, such as bank accounts, addresses, and SSNs. Thus, this paper lacks evaluations in more practical and meaningful scenarios that would better demonstrate its effectiveness in addressing privacy concerns in real-world settings.

questions:
see weakness. Also, how is the performance of the method on text-generation tasks? Could you evaluate your approach on some QA datasets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719675643

thread_type:
official_review

<thread_object>
creator: reviewer_w27p

<content>
summary:
This paper suggest a unified and feasible way to address multiple unlearning scenarios under a single framework. It first generalizes three unlearning objectives. Then, proposes to use influence-functions to approximate the influence of datapoints requested to be unlearned. It then proposed to use PEFT to avoid full finetuning of the model, by training each adapter separately.

strengths:
The paper unifies three unlearning objective under a single simple framework.
The paper directly addresses the complexity concerns the framework seems to have in two simple ways: using fast Hessian-vector product, and using mini-batches. It then provides a convincing argument for why such problem formulation would converge using SGD.

weaknesses:
The paper is a bit hard to read. I would suggest putting most equations in the appendix and focus on high-level explanation in the body of the paper. It is unclear exactly what exactly how adapters are used in Algorithm 1. The evaluation is a bit limited, consider comparing this method performance to other unlearning approaches. Evaluating efficiency in seconds can be a nice addition but the main evaluation should compare an asymptotic running-time complexity.

questions:
In the limitation section is mentioned a limited due to "first-order Taylor expansion", but I thought influence functions rely on second order expansion, which one is it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733221696886

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely appreciate the reviewers' efforts and valuable suggestions in reviewing our paper. We are pleased that all reviewers reached a positive consensus regarding the motivation, presentation, novelty, and experimental effectiveness of our work. In this response, we summarize the strengths of our work as recognized by the reviewers, highlight its key contributions, and detail the revisions and responses we have made to address all the concerns raised.

>### **Strengths**

- Motivation and Novelty: **novel instance-wise** unlearning method (`Reviewer 8gYQ`), address multiple unlearning scenarios (`Reviewer W27P`), addresses the complexity concerns (`Reviewer W27P`), **important and interesting** (`Reviewer 8gYQ`).

- Presentation: written clearly (`Reviewer nWJj`), easy to follow (`Reviewer 8gYQ`), with clearly presented formulations (`Reviewer nWJj`)

- Experimental effectiveness: **effectively** maintain good performance (`Reviewer 8XJc`), evaluations are **comprehensive** (`Reviewer 8gYQ`), **high accuracy** (`Reviewer 8gYQ`), **high efficiency** (`Reviewer 8XJc`), extensive experiments are conducted (`Reviewer nWJj`)

>### **Contributions of our work**

- **Unified Framework for Unlearning Tasks**: LLMEraser introduces a unified and efficient framework for parameter-efficient unlearning in LLMs. It generalizes three unlearning tasks—instance removal, query modification, and response correction—under a single, well-defined taxonomy.

- **Influence Function-Based Approach**: The framework leverages influence functions to approximate the impact of data points to be unlearned. This avoids the need for full model retraining or fine-tuning, significantly enhancing computational efficiency.

- **Efficiency Enhancements**: LLMEraser accelerates the computation of inverse Hessian-vector products required in influence functions by using fast Hessian-vector products and mini-batch processing. These optimizations make the framework suitable for large-scale LLM applications.

- **Comprehensive Evaluation and Practicality**: The proposed framework is demonstrated to perform effectively across diverse unlearning tasks. It also addresses practical privacy and security concerns in domain-specific fine-tuning scenarios.

>### **Responses**
- Reviewer ` W27P ` :
  - More experimental evaluation: We have supplemented the experimental results on generative tasks and included these results in Appendix G.
  - Evaluating efficiency and running-time complexity: We have added a theoretical analysis of the time and space complexity in Appendix F.
  - How adapters are used in Algorithm 1: We have revised the description of Algorithm 1 and added an introduction to the workflow of LLMEraser.

- Reviewer `8gYQ` :
  - Model performance on generative tasks: We have supplemented the experimental results on generative tasks and included these results in Appendix G.
  
- Reviewer `8XJc` :
  - Model performance on generative tasks: We have supplemented the experimental results on generative tasks and included these results in Appendix G.
  - Unlearning effectiveness: We have added explanations regarding unlearning effectiveness along with the corresponding experimental results.
  - Unlearning setting of data used in both pretraining and PEFT: We have added a discussion on scenarios where data is used simultaneously by both pretraining and PEFT, along with the corresponding experimental results.
  
- Reviewer `nWJj` :
  - Model performance on generative tasks: We have supplemented the experimental results on generative tasks and included these results in Appendix G.
  - Estimation errors analysis: We have included a theoretical analysis of approximation errors in Appendix E.
  - Discussion about memory efficiency: We have added a theoretical analysis of the time and space complexity in Appendix F.

We deeply appreciate the time and effort you have devoted to evaluating our work. Your insightful feedback has greatly motivated us to further improve and advance this research for the benefit of the broader academic community. If you have any additional suggestions or questions, we would be pleased to offer further explanations.

Best regards,

Authors of Paper 6232
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper provides a unified view of different unlearning scenarios. The reviews agree that the approach provided by the authors is both novel and well motivated. The area of unlearning is interesting to many researchers, and the paper provides a practical solution. The reviews appreciated the deep analysis and promising empirical results. That said, the reviews raised concern that there is a need for more thorough experiments, and minor concerns about the clarity of the paper.  

In the rebuttal, the authors added some experiments mitigating the concerns related to the empirical section and explained the vague or unclear parts of the paper. The provided material extends the existing content rather than requiring a new discussion, hence integrating it into the paper should be a simple task. Given this, the above mentioned strength and that the modified scores being unanimously towards accepting, I recommend accepting the paper, and urge the authors to carefully review the discussion and integrate the necessary parts into the final version of the paper.