Unlearn What You Want to Forget: Efficient Unlearning for LLMs
Authors: Jiaao Chen
Diyi Yang
Venue: EMNLP 2023 Main (2023)
Keywords: Efficient Unlearning
Large Language Model

Abstract: Large language models (LLMs) have achieved significant progress from pre-training on and memorizing a wide range of textual data, however, this process might suffer from privacy issues and violations of data protection regulations. As a result, the ability to easily remove data related to individual users from such models while not deteriorating their predictive quality after the removal becomes increasingly important. To address these issues, in this work, we propose an efficient unlearning framework that could efficiently update LLMs without having to retrain the whole model after data removals, by introducing lightweight unlearning layers learned with a selective teacher-student objective into the transformers. In addition, we introduce a fusion mechanism to effectively combine different unlearning layers that learns to forget different sets of data to handle a sequence of forgetting operations. Experiments on classification and generation tasks demonstrate the effectiveness of our proposed methods compared to the state-of-the-art baselines.

Unlearn What You Want to Forget: Efficient Unlearning for LLMs
Jiaao Chen
Diyi Yang Stanford University
Georgia Institute of Technology
jiaaochengattech. edu
diyiyêcs stanford.edu
Abstract
Kumar et al., 2022). studies showed that LLMs could generate private information such as the Editor-in-Chief of MIT Technology Review in- cluding his family members, work address, and phone number (Carlini et al., 2022). Recently, the HU's General Data Protection Regulation (GDPR) and US's California Consumer Privacy Act (CCPA) have also required the right to be forgotten, intro- ducing new regulations that require applications to support the deletion of user-generated content when requested by users (Sekhari et al., 2021; Ku- mar et al., 2022). In light of this, it is essential to provide LLMs with an efficient and effective way to unlearn the information requested by users.
Large language models (LLMs) have achieved significant progress from pre-training on and memorizing wide range of textual data, how- ever, this process might suffer from privacy issues and violations of data protection regu- lations. As result, the ability to easily re- move data related to individual users from such models while not deteriorating their predictive quality after the removal becomes increasingly important. To address these issues, in this work, we propose efficient unlearning framework that could efficiently update L.LMs without hav- ing retrain the whole model after data re- movals, by introducing lightweight unlcarning layers learned with a selective teacher-student objective into the transformers. In addition, we introduce a fusion mechanism to effectively combine different unlearning layers that learns forget different of data handle quence of forgetting operations. Experiments on classification and generation tasks demon- strate the effectiveness of proposed meth- ods compared to the state-of-the-art baselines!.
Recent attention has been paid to the handling of such unlearning requests for LLMs through retrain- ing and data pre-processing like SISA (Bourtoule et al., 2021; Kumar et al., 2022) where training data is stored in different isolated slices and each check- point saved after training each slice. When deletion request is received, the respective data point will be removed from the slice, and the model checkpoint up to the data point will be used to fur- ther retrain the model. The effect of unlearning is often reflected by the model errors on the deleted data (models cannot predict the deleted data) (Kur- manji et al.. 2023; Jang et al., 2022). Other works have also explored the design of algorithms that ensure differential privacy (DP) (Yu et al., 2021; Li et al., 2021; Anil al., 2021). However, ma- chine unlearning approaches like SISA (Bourtoule et al., 2021) usually require a significantly large amount of storage space (Bourtoule et al., 2021), and DP methods could result in slow convergence and significant deterioration in model performance (Nguyen et al., 2022). In addition, both of them require retraining the whole model, which is ex- tremely expensive and time-consuming consider- ing the model scales of the current I.LMs. These limitations also make them unable to dynamically deal with a sequence of unlearning requests which
1 Introduction
Utilizing Large Language Models (LLMs) has be- come the dominant paradigm for various NLP applications (Brown et al., 2020; Chowdhery et al., 2022a; Kojima et al., 2022; Ouyang et al., 2022: Brown et al., 2020; Radford et al., 2019; Lewkowycz et al., 2022: Qin et al., 2023; Touvron al., 2023) LLMs memorize a vast amount of knowledge during pre-training or fine-tuning on a wide range textual data (Brown et al., 2020; Rad- ford et al., 2019; Hoffmann et al., 2022; Webson and Pavlick, 2022; Min et al., 2022; Liang et al., 2022; Carlini et al., 2022). However, these data could contain sensitive information such names, phone numbers, email addresses, and private clini- cal notes (Jang et al., 2022; Kurmanji et al., 2023;
Thee codes are avaiable here: https://gtttub.comm thub. SALT- NLF /E.f ff ficiiee _nt_een__ee___eeeeeeeeeeeeeeeeee Unlearning/

---

Add&Norm
Add&Norm
Fused Unlearning
Unlearning Layer
Unlearning Layer
Layer
Unlea ming Layer
Feed Forward
Feed Forward
Add&Norm
Add&Norm
Unlearning Layer
Self Attn
Self Attn
Learning unlearning layer
Single Unlearning Layers
Fused Unlearning Transformers
Figure 1: Overall process of our BUL framework. The unlearning layers are plugged into transformer layers after the red-forwarr networks. During training, only the unlearning layers are learned forget requested data while the original LLMs remain unchanged. For every deletion request, an unlearning layer is learned first and then merged with other unlearning layers via our designed fusion mechanism to form the fused unlearning transformer which satisfies series of deletion requests.
is often the need in real-world scenarios (Jang et al., 2022: Nguyen et al., 2022).
We conduct experiments on classification and generation tasks with backbone models of dif- ferent scales in different settings, to illustrate the effectiveness of EUL.
To fill in these gaps, in this work, we propose an Efficient Unlearning method for LLMs (EUL) to ef- ficiently unlearn what needs to be forgotten without completely retraining the whole model while retain- ing the performances of the models. Specifically, we propose a lightweight approach learning the unlearning layer that plugged into transform- ers through selective teacher-student formulation (Kurmanji et al., 2023) within several updates, with- out tuning the large language models. Additionally, we introduce a fusion mechanism to effectively combine the weights of different unlearning layers that learn forget different sets of data a single unified unlearning layer by minimizing a regression objective. This allows EUI. efficiently address a sequence of deletion operations. To demonstrate the effectiveness of our proposed EUL, we per- form experiments on IMDB (Maas et al., 2011) and SAMSum (Gliwa et al., 2019) in different set- tings compared to the state-of the-tartttttttt unlearning or model editing baselines. To summarize, main contributions are threefold:
2 Related Work
2.1 Large Language Models
Large language models have witnessed extensive progress recently (Brown al., 2020; Radford et al., 2019; Smith al., 2022: Rae et al., 2021; Chowdhery et al., 2022b; Touvron et al., 2023), especially terms of scaling up LLMs such as LLAMA (Touvron et al., 2023), Megatron-tune turing NLG (Smith et al., 2022), Gopher (Rac et al., 2021), and PaL.M Chowdhery et al. (2022b). Other works have also achieved better performance with smaller models through longer training (Hoffmann et al., 2022), instruction tuning (Wang et al.. 2022: Zhou et al., 2023) and human feedback (Ouyang et al., 2022). However, recent studies have shown that training data, such as personally identifiable in- formation like names, phone numbers, email ad- dresses, and even bank account numbers (Carlini et al., 2021; Lee et al., 2021; Carlini et al., 2022; Jagielski et al., 2022), can be casily extracted from LLMs because LLMS memorize the training data in billions of parameters (Carlini al., 2022). Our work is proposed to alleviate such issues by allow- ing efficient unlcarning of the requested or private data from the learned parameters in LLMs.
We introduce an efficient unlcarning method to remove the effect of required data in lightweight way via selective teacher- student formulation.
We design fusion mechanism to merge un- learning layers that are learned forget differ- ent sets of data into a single unlcarning layer to deal with sequence of removal operations.

---

2.2 Machine Unlearning for Privacy
where D = D DJ = 1(x",, ,y") refers the data we would like retain, and (.) is the mutual information. Intuitively, we will update F(.) with F(.) to generate similar output for the data we want to retain while losing all information about making predictions on the data we want to forget.
To mitigate the privacy risks for LLMs, machine un- learning methods have been introduced to remove the contributions of training examples that users re- quest to be erased by users (Bourtoule al., 2021; Chien et al., 2023) including exact unlearning that retrains deep learning models on new datasets after removal (Bourtoule al., 2021) and approximate unlearning (Izzo et al., 2021; Golatkur et al., 2020; Kurmanji et al., 2023; Jang et al., 2022) which aims to modify the weights of trained models to produce new set of weights that approximate the weights from retraining. The effect of unlearning is often reflected by the model errors on the deleted data (models cannot predict the deleted data) (Kur- manji et al., 2023; Jang et al., 2022). Another line of work has focused on Differential Privacy (DP) which ensures that user information in training data cannot be inferred (Dwork, 2008; Yu et al., 2021;
3.1 Learning to Forget via Unlearning Layers As the scales of current LLMs and the size of train- ing data are usually large, updating all the param- eters in the model F.) (e.g.. re-training F(.) on Dj) becomes extremely expensive. Inspired by recent advances in parameter-effickee fine-tuning (Houlsby al.. 2019; Chien al., 2023), we model F'(.) by F(f(-)) where f-; W) is an adapter with significant smaller amount parameters W com- pared to F(.). And we would only update f(.) to fulfill the unlearning requests.
To effectively achieve the unlearning goals in equation 1, we minimize a selective teacher-student objective where the student model F* F'(.) F(f(-)
Li et al., 2021; Anil et al., 2021; Abadi et al., 201 However, both types of methods require retraining the whole model, which is extremely expensive and time-consuming, especially for large language models and even impacts the task performances (Anil et al., 2021). And thus they can not dynami- cally tackle sequences of deletion (Jang et al., 2022; Nguyen et al., 2022). To overcome these limita- tions, we introduce an efficient unlearning method well as fusion mechanism to efficiently and dynamically unlearn sequence of user data.
is learned follow the teacher model F() on Dr while disobeyed F.) on DJ:
LKL =a KLEE"IIFU(E))
(2)
E (LFFE")FFUL
where is pyyeer-paameeee to balance the trade- off between forgetting xf and retaining x". Intu- itively, during training. /() is leaned to minimize the KL-divergence between the output from the updated model and the original model on the data to retain while maximizing the KL-divergence be- tween the output from them on the data to forget.
Our work also related model editing (Mitchell et al., 2021; Belinkov et al., 2017; Dai et al., 2021; Wang et al., 2020) while they usually focus on editing the model output based on several given linguistic structures or facts about the world instead of forgetting the required data.
To maintain the task performance, we optimize f()) for the task loss on the retain data:
3 Efficient Unlearning for LLMs
LTASK E((F(F())))))
(3)
This section presents our designed Efficient Unlearning method for LLMs (EUL) which could efficiently and dynamically handle a sequence of deletion requests. The overall process is shown in Figure! 1. Formally, for a large language model F() that is trained on dataset D = {xx y}} where x is textual data and y is the corresponding label, and a deletion request to forget DJ {(2 / y'), }. our goal is to learn an updated model (.) that satisfies the following (Kurmanji al., 2023):
where ((.) is the task-related loss, for example, cross-entropy loss. lo ((F(f(z")) for clas- sification tasks.
Furthermore, we also negate the original training objectives used in LLMs (e.g., masked language modeling objective (Raffel et al., 2020)) to forget the knowledge related to the data, in order to for- get in pre-trained parameters and ensure that the information in the forgotten data cannot be casily extracted from F()):
I(((");F(D))
LLM = Σ [(F((((()))
(I)
(4)
(((")

---

where 1(.) is the language model loss used when pre-training F(), ), for example, masked language model loss, - log P( x i) (& are the randomly masked tokens). In our experiments, we utilize T5 models (Raffel et al., 2020). Thus we add an extra "Predict the masked word"' at the beginning of the input for this loss term.
Dataset
Task
Train
Dev
Test
IMDB
Classification
20000
2000
25000
SAMSum
Summarization
14732
818
819
Table I: Dataset statistics for IMDB and SUMSum.
Our final training objective is then the following:
utilize the IMDB (ats et al., 20, which is a sentiment classification dataset consisting of users' reviews of movies, directors, actors, etc. For the generation task, we use SAMSum (Gliwa et al., 2019), which is recent popular conversation sum- marization dataset consisting of conversations be- tween different speakers. The dataset statistics are shown in Table 1.
(5) where À and 7 are hyper-parameters. In practice, following Kurmanji al. (2023), we alternate the updates for the data be forgotten and the data to be retained to optimize min-max terms in LEUL more stably. Specifically, we iteratively perform an epoch of updates on the data to be retained and then an epoch of updates on the data to be forgotten.
LEUL = LKL + ALTASH + +
We choose these two datasets because they are widely used (Wang et al., 2021; Yang et al., 2019; Qin et al., 2023; Ji et al., 2023; Wei et al., 2021; Sanh et al., 2021; Chen al., 2022) to evaluate large language models and both datasets are related to cases where the user might require remove their data, for example, removing all the reviews of a specific movie or removing all the conversations from one specific speaker.
3.2 Fusing Unlearning Layers
To dynamically handle a sequence of unlearning requests and derive unified model that could for- get all of the requested data, we then introduce fusion mechanism that could merge different un- learning layers fi(:; W4) which are learned to forget DJ = =(x?, yy in the previous section into sin- gle (.; Wm).. Namely, we would like the output of (.) on Df. being close to fí(-):
In experiments, we use the pre-trained NER models from AllenNLP2 to extract all the entities (names) in IMDB and directly use the speakers' names in SAMSum and simulate the unlearning re- quests to remove all the data from or related to cer- tain names. Moreover, we substitute all the names in the dev and test set with special tokens.
Σ EIWTX wFx/11
min W
(6)
which is a linear regression problem and has a closed-form solution:
4.2 Evaluation Metries
Wm Σ Ex!"xx E(x!" x(W.)
(7)
To evaluate the performances, following Kurmanji et al. (2023), measure several metrics: (1) Per- formance on the test set: The task-related perfor- mance on the test set, namely, accuracy for IMDB and ROUGE for SAMSum. This measures whether the unlearning algorithms affect the model perfor- mance or not. (2) Performance on the retained set: The task-related performance on the data to be retained. This measures whether the unlearn- ing algorithms forget the data that need to be tained. Higher performance means that the model remembers the data that not to be forgotten. (3) Performance on the forgot set: The task-related performance on the data to be forgotten. This mea- sures whether the unlearning algorithms effectively forget the data requested to be forgotten. Lower performance means that the model is better at for-
Specifically, to derive the weights W., for the merged unlearning layer f' we would use the pre- computed inner product matrix of the hidden repre- sentations before the unlearning layers LLMs of the forgotten data X!" X and then compute W., following Equation
The fusion mechanism ensures efficiency and privacy as it could be performed without any extra training and only requires storing the inner prod- uct matrix of the representations of the data to be forgotten instead of the data itself.
4 Experiments
4.1 Datasets
We conduct experiments on both classification and generation tasks. For the classification task, we
Pnttps://demo, a cannlp.org.

---

Methods
Forgot Data
Test Set f
Retained Set
Forgot Set
MLM Loss
Time (8)
75-base
Original
93.2
100
100
1.46
Re-train
92.8
100
92.5
1.52
6685
Fine-tune
93.0
100
96.5
1.47
4200
SISA
92.4
98.2
91.5
1.54
1580
0.5%
Reverse- Reverse-Gradient
92.0
68.6
1.56
4400
MEND
92.2
98.5
73.5
1.60
34
EUL1
93.0
100
65.7
1.78
1200
Re-train
92.7
100
91.6
1.55
6610
Fine-tune
92.8
100
96.2
1.48
3950
SISA
92.2
98.1
90.4
1.55
2930
1%
91.5
96.4
67.4
1,59
4166
MEND
91.3
95.5
74,6
1.62
62
FUL,
93.0
100
64.4
1.84
1526
Re-train
92.1
90.2
1.56
6026
Fine-tune
92.0
100
95.8
1.52
3133
SISA
91.6
98.2
88.4
1.55
2010
10%
Reverse- Reverse-Gradient Gradient
91.0
96.5
65.4
1.62
3228
MEND
90.8
94.8
76.2
1.66
328
BUL
92.2
99.0
$7.2
2.01
1828
T5-3b
Original
97.0
100
100
1.28
Re-train
96.6
94.8
1.30
26855
Fine-tune
96.7
100
96.2
1.28
20465
SISA
95.0
97.2
94.1
1.33
16503
0.5%
Deverse-Gradien
93.3
96.5
78.9
1.42
21826
MEND
93.0
95.8
89.5
1.30
4980
BUL1
96.5
100
70.2
1.66
9240
Re-train
96.3
100
94.2
1.30
25280
Fine-tune
96.5
100
96.0
1,28
18466
SISA
93.8
96.8
92.7
1.35
15680
Reverse-Grudient
1%
92.5
96.0
80.1
1.46
18800
MEND
92.8
95.0
84.4
1.48
6600
EUL1
96.5
100
67.5
1.72
9840
Re-train
96.0
100
93.5
1.31
22140
Fine-tune
96.2
100
94.0
1.30
16752
SISA
93.0
95.5
92.2
1.35
14180
10%
Reverse- Gradient
91.9
95.2
68.4
1.46
17850
MEND
92.0
94.2
78.5
1.50
12072
EUL+
96.0
100
60.8
1.92
10460
Table Performances IMDB for TS-base and T5-3B after unlearlinng different number of privacy-related data. i refers to our model. All the results are averaged over 5 random runs.
getting the data. (4) MLM Loss: The masked language model loses on the data be forgotten where related entities or actions are masked. This is achieved by adding "Predict the masked word" in the beginning. This measure whether the infor- mation in the data that needs to be forgotten can be extracted from the LLMs. Higher MLM loss means that it is harder to extract such information from the models. (5) Updating time: The time to update the original model in the forgetting process.
4.3 Baselines
We compare our EUL with several baseline meth- ods: Re-train (Kumar et al., 2022): Re-training the model from scratch on the data to be retained with- out any forgotten data. Fine-tune (Kurmanji et al., 2023): Fine-tuning the original model on the data to be retained without any forgotten data. SISA (Kumar al., 2022): Sharded, Isolated, Sliced. and Aggregated training where multiple models are trained independently on disjoined shards, and its slices and model checkpoints are saved for each

---

Methods
Forgot Data
Test Set
Retained Set f
Forgot
MLM Loss f
Time
75-base
Original
47.2/23.5/39.6
71.4/42.6/62.7
70.2/42.2/62.7
1.37
Re-train
46.8/23.0/38
8/62.4
12.4/23.2/422
1.40
28000
Fine-tune
46.6/23.2/38.
72.5/44.7/6..2
58,8/34.1/6 1/54.1
1.38
27120
SISA
0.5%
44.2/22.0V37.4
70.5/41. 66/60.5
41.4/23.0/40.8
1.48
22582
Geverse-Gradien
43.2/2009/35..
68. 2/58.2
42.3/21 4/338.
1.64
28800
FULL
46.8/23.0/38.5
71.5/42.4/6.33
38,4/20.2/37.2
1.88
17060
Re-train
45.4/22.8/37.5
72,4/43.016 0/62.8
42,2/22.8/41 8/41.
1,44
26855
Fine-tune
46.4/23.2/38.1
72.9/43.6/644.
56,4/31 8/52.7
1,40
27210
SISA
1%
43.1/21. 1/36.8
69 8/40 2/60.0
4//4/33
4/20000/400.8 0/40,8
1.50
22420
Reverse- Ceverse-Gradient
42,0/20.0/34, 0/34,
68, 15.8/40.2/55 2/58.5
42.3/114/38.1
1.64
27700
EULÍ
46.5/22.8/38.0
71.5142.463.3
35.8/19.0/36.2
1.95
16820
Re-train
44.2/21.2/35.
20.4/41.2/60.33
1.48
26155
Fine-tune
45.2/22. 1/36.6
71.1/42.6/62.9
51.5/28, 6/50.0
1.43
27510
SISA
10%
41.8/19.6/33.
68.3/38. 8/58.8
40.2/20. 1/38.9
1.55
20790
Reverse-Gree Gransint
40.8/18.4/333 4//333.
66.6/38.3/55. 3/55.5
38.0/19 4/36.6
1.71
27240
EULÍ
45.8/22.4/37.8
70.9/42.0/62.3
33.0/18.3/33.00
2.23
15000
T5-3b
Original
53.6/29.6/45.1
78.5/47. 6/66.1
74.2/43.5/64.9
1.30
Re-train
52.8/28. 8/44.0
8/44.0
77.4/46.1 1/65.4
50.4/27.2/43
1.34
84480
Fine-tune
53.3/29.0/44 0/44.4
78.0/47.1/65.8
10.2//6.1/555.7
1.30
83600
SISA
0.5%
31.7/27.2/40.
74.8/44880 8/6..6///665
49.4/26. 8/42.2
1.33
75000
Reverse-Grading
50.6/25.9/30 9/39.9
72. 8/42.0//66..
44.3/23. 1/39.0
83200
EULt
53.6/29.4/44.8
77.5/46./66.66
41.0/218//8.22
1.67
60430
Re-train
52.0/28. 2/42.8
76.7/45
49 49,6/26, 6/42.
1.35
82440
Fine-tune
52.5/28.5/43.6
16.2/45.5/64.2
66.8/32.2/52.44
1.32
81135
SISA
1%
50.0/26. 1/38.9
72. 1/61.1
49,0/25 8/41.
1.38
73550
Reverse-Gradient
3/.372
70.6/41
42.2/22. 0/37.7
1.45
82485
EULi
53.3/29.0/44.4
76.4/45 3/64.3
38.4/19.9/36.0
1.74
60880
Re-train
50.8/26.4/40.5
74.2/45. 0/63.2
48. 48.2/25.8 5/41.4
1.38
81010
Fine-tune
51.4/27.2/41.9
75.2/45.3/644.0
1/.1129..8/4 8/499 9
1.35
81800
SISA
10%
48. 5/3260
70.4/40. 5/59.6
11.2/23.5/40..
1,40
70400
Reverse-Gradion
44.7/22.0V34.2
9/5888
40,9/21
10/./.5 00/36.5
1.49
82070
EULÍ
52.0/28.4/42.6
74.9/45.0/63.6
36.2/18.6/34.7
1.78
59900
Table 3: Performances on SAMSum for T5-base and T5-3B after unlearnling different number of privacy-related data. refers model. All the results averaged over 3 random runs. The performance Test, Retained and Forgot Set are ROUGE- 1/2/L scores.
slice. When forgetting certain data, the correspond- ing data point is deleted from its slice, and the model checkpoint up to the data point used to further retrain the model. Reverse-Gradient (Liu et al., 2022): Fine-tuning the original model on both retained data and forgot data while negating the gradient for the forgot data. MEND (Mitchell et al., 2021): Editing the model to generate out- put following the given examples. To adapt the model in the unlearning setting, we reverse the labels for data in classification tasks as input to MEND. However, it is infcasible to apply MEND to summarization tasks as it is hard to design the new output to perform the editing.
4.4 Model Settings
For all the experiments, we T5 models (T5-base and T5-3b) (Raffel al., 2020) as the backbone models. For SISA, we follow Kumar et al. (2022) to split the dataset. For our unlearning layers, we only tune 0.5% (Chen et al., 2023) of the param- eters. The = 0.8, À - 1.0 and - 0.2 are se- lected from grid searching (0.1, 0.2, 0.5, 0.8, 1.0). We set the linear decay scheduler with warmup ratio of 0.06 for training. The maximum sequence length 128 for IMDB and 800 for SAMSum. The batch size was 256 for base models and 128 for 3b models on IMDB and for base models and for 3b models on SAMSum. The maximum learning

---

Methods
Test Set f
Retained Set f
Forgot Set |
Updating Time (s)
Original
91.8
100
91.2
Re-train
92.5
100
12.6
6026
Fine-tune
92.3
100
26.8
3133
SISA
92.2
98.2
12.6
1510
Graniennt
92.8
98.6
9.0
3228
MEND
92.2
97.8
16.8
328
EULi
93.0
99.0
5.0
1828
Table Performances IMDB for T5-base after unlearnling 10% wrong-labeled data. f refers to our model. All the results are averaged over 5 random runs.
Rai-triin
Fine-tune
Reverse-Cradient
FUL
FUL-fuse
90
Test Sest
92
Accuracy on Forgat Sel
80
Accuracy
75
65
87
Figure Sequentially unlearnling 1,2,3,4,5 different sets data bS-base on IMDB. The results are accuracy on the test set and the accuracy on the forgot averaging across different orderings. Every single set contains 1% of the training data.
Metric
EUL
-KL
-TASK
-LM
Models
Sct 2
Set 2,
Set 2, 1,3
Test Set f
93.0
91.4
91.0
92.4
Re-train
92.7/91.4
92.5/90.8
91.3/90.0
Retained Set
100
100
97.4
99.0
Fine-tune
92.8/96.0
92. 1/940
91.0/93.3
Forgot Sct 4
65.7
90.8
67.4
69.0
SISA
92.2/90.4
92.0/8.88
91.2/85.5
MLM Loss
1.78
1.75
1.78
1.50
Reverse-Gradien
91.5/67.9
90.5/67.2
89.8/66.0
EUL.
93.0/64.6
92. 9/64.6
91.0/64.2
Table 5: Performances on IMDB for T5-base after re- moving 0.5% privacy-related data. We remove ob- jective at time from our EUL methods.
BUL-fuse
93.0/64.6
92.8/62.2
92.4/60.8
Table Accuracy on the test/retained set of after un- learning sets of data following a sequence (set 2 set set 3).
was 5e and the maximum number of train- ing epochs was set to 5. All the experiments were performed using A.000 GPUs.
cases where people/companies request to remove all the data related to them. The performances are displayed in Table 2.
4.5 Results
Unlearning Privacy-related Data on IMDB We request the T5-base and T5-3b models that are fine- tuned the IMDB dataset to unlearn 0.5%, 1 1% and 10% of the training data. The data be for- gotten is randomly selected based on the names of movies, actors, actresses, directors, etc. For ex-
A After unlearning the requested data from T5- base models, the re-training method hurts the accu- racy (c.g., 1.1 accuracy drop when forgetting 10% data) the test because there fewer data for training, and the accuracy the retained set keeps unchanged (100%) probably because the model memorizes the retained data. The accuracy on the forgot set drops after re-training (e.g., 92.5 com-
ample, the model might need to forget all the data points related to "Lena Numan". This simulates the

---

pared to 100 when unlearning 0.5% of the data), showing that the model is forgetting the requested data, and the masked language model loss increases (e.g., increasing 0.06 when unlearning 0.5% of the data), indicating that it is harder to extract the in- formation of the forgot data after re-training. The fine-tuning method shows better test accuracy with less updating time, however, it is worse in terms of forgetring the data. Even though SISA takes sig- nificantly less time (only costing around 1/3 of the time compared re-training) derive the updated model that forgets the requested data, receives lower accuracy on the test and retained set, which means that the model prediction abilities get worse because of failing to remember the retained data. When reversing the gradients for the data to be forgotten, the updated model gets better at forget- ting with lower test accuracy. The model editing method, MEND, shows better overall performance on nearly all the metries but it requires extra data to train a model editing module to edit the original model, making the method hard to be generalized to new models and settings. Our EUL approach boosts all the metrics with faster speed to update the model compared to previous unlearning baselines after removing different numbers of privacy-related data (e.g., achieving the lowest accuracy (65.6%) on forgot set while keeping the best test accuracy (93.0%) and 100% retained accuracy with 1/6 of the updating time compared to re-training when forgetting 0.5% of the data), suggesting that our designed unlearning layers that are learned with tai- lored objectives could efficiently update the LLMs to forget the required data and remain the abilities perform the tasks. When the size of the back- bone model scales up to 3b, the improvements of our EUL are consistent, indicating that our methods could still forget what the user requests even for larger models that are better at memorizing data.
membering the retained data and keeping the test ROUGE scores with significantly less amount of training time. This indicates that our objectives could also be generalized to generation tasks.
Unlearning Mislabeled Data on IMDB We also test setting where the data to be forgotten is those with wrong labels. In experiments, we randomly change the labels for 10% of the training data and then request the model unlearn their impact. This simulates the cases where we improve the models that are trained on noisy data by unlearning the mislabeled data (Kumar et al., 2022). We report the performances with T5-base models in Table 4. We observe that the accuracy of the test set of the original model is affected by the mislabeled data, And our EUL is the most effective approach to unlearn and remove the negative impact of those mislabeled data achieve the best test accuracy.
Sequence of Removals We test baseline and our methods in a setting where a sequence of unlearn requests are received, i.c., the models need to forget different sets of data sequentially. In experiments, we sequentially unlearn 1,2,3,4,5 sets of data from T5-base model on IMDB dataset. For every unlearn length, test with all the possible sequences and average the accuracy on the test set and the forgot set. For example, when the length of the forgetting requests are (set 1, set 2), we test on the sequence (set 1 -> set 2) and sequence (set 2-> set 1) and av- erage the final performances. We show the results (accuracy on the test/retained set) of one possible sequence whose length is 3 (set 2 -> set -> set 3) in Table 6 as example. Averaged performances over different sequence lengths Figure 2. EUL means that we keep one unlearning layer to sequentially unlearn different sets of data and BUL-fuse means that for every set of forgot data we learn separate unlcarming layers and then merge them into a single unlearning layer via our proposed fusion mechanism. The results demon- strate that our proposed fusion method that com- bines different unlearning layers could effectively handle the sequence of deletion (achieving higher accuracy on the test set and lower accuracy on the forgot set.) especially when the sequence length gets longer compared to baseline models.
visualized in
Unlearning Privacy-related Data on SAMSum We unlearn 0.5%, 1% and 10% training data from T5-base and T5-3B models that are fine-tuned on the SAMSum dataset. The data to be forgotten is randomly selected based on the speaker names. For example, the model might need to forget all the conversations from "Jack", This simulates the cases where people request to remove all the data generated by them. The performances are shown in Table 3. Similarly, our EUL, method consis- tently achieves the best overall performances by effectively forgetting the requested data while re-
4.6 Ablation Studies
Removal of Objectives We perform ablation studies to show the effectiveness of each designed objective in EUL by removing each of them when

---

Models
LMDB
SAMSum
ing unlearning layers through the selective teacher- student objective. We further introduce fusion mechanism that could merge different unlearning layers into one unified layer to dynamically un- learn a sequence of data. Experiments on different settings (different datasets, different model sizes, different forget set sizes) demonstrated the effec- tiveness of our proposed EUL method compared to state-of-the-ard baselines.
Original
0.542
0.510
Re-train
0.550
0.522
Fine-tune
0.568
0.525
SISA 0.585
0.530
Reverse- Reverse-Gradient 0.626
0.588
EUL
0.566
0.530
Table 7: Accuracy from trained binary classifier predict whether an input data belongs to the retained set or the forgot set.
Limitations
In this work, we mainly perform experiments on T5-hase/3b models with fine-tuned tasks. We en- courage future work to explore how to update dif- ferent backbone models with larger sizes such LLAMA models
learning the unlearning layers in Table 5. Com- pared to EUL which utilizes all of the learning objectives, removing each of them would result in performance drop, which demonstrates every component contributes to the final performance. Specifically, removing LKL would increase the ac- curacy of the forgot set, indicating that LąL is the main factor to forget the requested data. Remov- ing LTASK from EUL would drop the accuracy on the test set, suggesting that LTASK is essen- tial maintain task performance. Removing LLM decreases the MLM Loss, showing that LLM is the main objective to avoid the extraction of the requested information.
close-sourced models like ChatGPT forget the requested data such asemn privacy-related data, toxic data, or misinformation in the pre-training corpus. Also, we mainly fol- low the previous work to mcasure the unlearning through performance on the test set, retained set, and forgot set, together with the MLM loss. Fu- ture work might explore how to evaluate unlearning methods more comprehensively, such as whether the model could recall forgotten content whether methods would make forgotten data identifiable. In addition, we perform all the experiments simu- lated settings. Future work might apply our meth- ods to real-world applications to deal with actual use cases or introduce new benchmarks for evaluat- ing unlearning methods.
Member Inference Attack We further perform Member Inference Attack (MIA) (Kurmanji et al., 2023) on IMDB and SAMSum when unlearn 1 1% privacy-related data for T5-base models. Specif- ically, we test the accuracy of a binary classifier which is trained to predict whether the input data belong to the forgotten set the retained set based their representations after the final layer of the T5 model. An accuracy closer 0.5 means that is hard for the classifier to predict the groups of the input data. The accuracies are shown in Table We found that the classifiers could not converge so well on the training set and always had a low accu- racy on the test set both before and after unlcarning (e.g., 0.542 before unlearning and 0.566 after our BUL unlearning on IMDB). These showed that the randomly deleted data could not be easily inferred both before and after our EUL unlcarning.
Acknowledgment
We would like thank all reviewers and the SALT Lab for their valuable feedback. This work was partially sponsored by NSF grant IIS-2247357 and IIS 2308944.
References
Martin Abadi. Andy Chu. Ian Goodfellow, H Bren- dan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential pri- vacy. In Proceedings of the 2016 ACM SIGSAC con- ference on computer and communications security, pages 308-318.
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. 2021. Large-scale differen- tially private bert. arXiv preprint 1rXiv:2108.011 002444
5 Conclusion
In this work, we propose EUI., an efficient unlearn- ing method for LLMs that could efficiently and ef- fectively unlearn the useu-teee data via learn-
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Has- san Sajjad, and James Glass. 2017. What do neural

---

machine translation models learn about morphology? arXiv preprint arXiv:1704.03477
David Dohan, Shivami Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022a. Palm: Scaling language modeling with pathways.
Lucas Bourtoule, Varun Chandrasckaran, Christopher A Choquette-Chee Hengrui Jia, Adelin Travers. Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Ma- chine unlcarning. In 202/ IEEE Symposium on Secu- rity and Privacy (SP), pages 141-159. IEEE
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Se- bastian Gehrmann, al. 2022b. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:220.002311
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2021. Knowledge neu- rons in pretrained transformers. arXiv preprint Sifi:20044086966
Cynthia Dwork. 2008 Differential privacy: A survey of results. In Theory and Applications of Models of Computation: Sth International Conference, TAMC 2008, Xi'an, China, April 25-29, 2008, Proceedings 5, pages 1-19. Springer.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural lan- guage models. arXiv preprint www.:::222.00444..
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. Samsum corpus: A human- annotated dialogue dataset for abstractive summa- rization. arXiv preprint arXiv: 1911::99111.22
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel HerberttVosss Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song. UI- far Erlingsson, al. 2021. Extracting training data from large language models. In USENIX Security Symposium, volume 6.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2020. Fternal sunshine of the spotless net: Se- lective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9304-9312.
Jiaao Chen, Mohan Dodda, and Diyi Yang. 2022. Human-in-the-loop abstractive dialogue summariza- tion.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya. Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 225.:255.....6
Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alex Smola, and Diyi Yang. 2023. Parameter- efficient fine-tuning design spaces. arXiv preprint 201.022301.01821.
Eli Chien, Chao Pan, and Olgica Milenkovic. 2023. Ef- ficient model updates for approximate unlearning of graph-structured data. In The Eleventh International Conference on Learning Representations.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019, Parameter-efficient transfer learning for nip. In In- ternational Conference on Machine Learning, pages 2790-2799. PMLR.
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi,
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. 2021. Approximate data deletion from machine learning models. In International Con- ference on Artficial Intelligence and Statistics, pages 2008 2008-2016.12 PMLR.
Matthew Jagielski, Om Thakkar, Florian Tramer, Daphne Ippolito. Katherine Lee, Nicholas Carlini, Fric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, et al. 2022. Measuring forget- ting of memorized training examples. arXiv preprint arXiv:2207.00099

---

Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2022. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021. Fast model editing at scale. arXiv preprint arXiv: 10. 11309.
Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. 2022. A survey of machine unlearning. arXiv preprint 220.02200...299.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang. Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput- ing Surveys, 55(12):1-38.
Long Ouyang, Jeff Wu, Xu Jiang. Diogo Almeida, Car- roll L Wainwright, Pamela Míshkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow in- structions with human feedback. arXiv preprint 22X.::223.022155
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In Thirty-sixth Conference on Neural Information Processing Sys- tems (NeurIPS 2022).
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaso Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt general-purporn natural language process- ing task solver? arXiv preprint arXiv:2302.064
Vinayshekhar Bannihatti Kumar, Rashmi Gangadhara- iah, and Dan Roth. 2022. Privacy adhering machine un-learning in nlp. arXiv preprint arXiv:2212.09572
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, page 9.
Meghdad Kurmanji, Peter Triantafillou, and Eleni Tri- antafillou. 2023. Towards unbounded machine un- learning. arXiv preprint arXiv:2302.09888
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Bure Burch, and Nicholas Carlini. 2021. Deduplicating training data makes language models better. arXiv preprint
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.1 11446.
arXiv:2107.06499.
Aitor Lewkowycz. Anders Andreassen, David Dohan, Ethan Dyer, Hemryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. Solving quantitative reasoning problems with language models. arXiv preprint arXiv:2206.14S:
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. JMLR, 21(10))
Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2021. Large language models can be strong differentially private learners. arXiv preprint 1rXiv:2110.0569.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. 2021. Multitask prompted training en- ables zero-sho task generalization. arXiv preprint 2rXiv:2110.08207. 10.08207.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models. ArXiv preprint, abs/221 11 11.091 10.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. 2021. Remember what you want to forget: Algorithms for machine unlcarning. Advances in Neural Information Processing Systems. 34:18075 04l18875-18886.
Bo Liu, Qiang Liu, and Peter Stone. 2022. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pages 243-254. PMLR.
Shaden Smith, Mostofa Patwary, Brandon Norick. Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. 2022. Using deep- speed and megatron to train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint arXiv:2201.I 19900.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the associ- ation for computational linguistics: Human language technologies, pages 142-150.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 2302:2300.11977
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstra- tions: What makes in in-context learning work? arXiv preprint a2x: 12837.

---

Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei. Xuanjing Huang, Guihong Cao, Daxin Jiang. Ming 7hou, et al. 2020. K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv preprint arXiv:2002.01800
Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021. Entailment as few-shot learner. arXiv preprint arXiv:2104. 14690
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- nanch Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprint arXiv:2212 10560.
Albert Webson and Ellie Pavlick. 2022. Do prompt- based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tèch- nologies. pages
Seattle, United States. Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew Dai, and Quoc V Le. 2021. Finetuned guage models are zero-shot learners. arXiv preprint ar9iv:2109.01652.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc Lc. 2019, Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems, 32.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulka- mi, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. 2021. Differentially private fine-tuning of lan- guage models. arXiv preprint arXiv:211000631
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Tyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. arXiv preprini arXiv arXiv:2305.11200 12206.


Reviews:
[{"review_id": "q75BYCaTb4", "reviewer": "Reviewer_wd9K", "summary": "", "questions": "In Table5, it is shown that different losses contribute to different aspect of the performance. Does this finding still hold in the summarization dataset?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive assessment and constructive feedback.\n\n## On the concern about effectiveness for larger LLMs\nWe performed experiments on T5-base and T5-3b as working examples to illustrate the effectiveness of our introduced unlearning methods. We did not test on recent larger LLMs because of the computation limit. However, from our experiments on smaller (base) and larger (3b) models, we could observe consistent forgetting ratio (e.g., 36\\% on T5-base and 34.9\\% on T5-3b on classification tasks when forgetting 10\\% of the data.). As a result, we think our method would still work for recent proposed LLMs.\n\n## On the concern about evaluations about the forgetting\nOur work is a starting point on the important issues for deploying large language models about how to unlearn the knowledge/data in large LLMs. While there are no well-defined test sets and evaluation metrics, we evaluate our methods with metrics used in all of the previous work about unlearning LLMs on both classic classification and generation datasets which contain privacy information to simulate the user cases.  In terms of the metrics, besides the  performance on different sets used in previous studies, we include MLM loss to further reflect whether generative LLMs could still predict user information that needs to be forgotten. And our results demonstrate that our methods could forget the knowledge effectively and efficiently even in a sequential setting which previous methods fail to handle. \nWhat’s more, in our early studies, we also evaluated the MIA attack when performing unlearning on the IMDB dataset where we tried if we could learn a binary classifier to predict whether a data belongs to the forgotten set or retained set based on the representations. And we found that the classifiers could not converge so well on the training set and always had a low accuracy on the test set both before and after unlearning (e.g., 0.542 before unlearning and 0.566 after unlearning). The results are shown in the table below. These show that the randomly deleted data could not be easily inferred both before and after the unlearning. As a result, we did not present the studies in the main paper. We would add this study and results to the revised version.\n\n| Models | Original Model | Re-train | Fine-tune | SISA  | Reverse-Gradient | EUL   |\n|---------|--------------|----------|-----------|-------|------------------|-------|\n| MIA    | 0.542          | 0.550    | 0.568     | 0.585 | 0.626            | 0.566 |\n\nWe hope our preliminary study could raise awareness of the community on this important problem about forgetting the required information in LLMs and facilitate future research on unlearning LLMs.\n\n## On the question about the ablation studies on summarization dataset.\nYes, the findings still hold for summarization dataset. We would add the additional results in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. After trained on huge amount of data, deleting certain information that might be hurtful, toxic, or sensitive without re-training the model is very important. The authors propose a simple but effective method to teach the model unlearn without any retrain steps would be very beneficial for future LLM development.\n2.The fused learning mechanism allows the model to remove the unneeded information in a sequence. Figure 2 shows the effectiveness of this component compared with the original version.\n3.The authors conduct several experiments to test the model on both classification and generation tasks under two base models. Experiment results show that the EUL method helps to save the time and assist the model unlearn certain information.\n4. The motivation and method parts are clear and the experiments are reasonable.\n", "reasons_to_reject": "1.The main concern of this paper is that they did not perform experiments on LLMs such as LLAMA but only adopt T5 as the base model. However, as shown in Table2 and Table3, it seems the performance on the forget set increases when the base model grows larger (T5-3b v.s. T5-base). This result shows that, as the language model grows larger, it gets harder for them to forget. This is reasonable because the larger the language model is, the more the model knows. Therefore, it would provide the essential insight if the proposed method would still be beneficial in the recent proposed LLMs, and how much will it forget under this method.\n\n2. Even though the model has performance drop on the forgot data, it’s still very hard to say whether it truly ‘forgets’ the data, considering there always remain some common knowledge overlap between forgot data and the training portion. As in the limitation part states, the experiments are performed in a simulated setting, it would be more convincing if the users can provide some real cases to analyse which data is easily to be forgotten and which is not.\n", "questions_to_authors": "In Table5, it is shown that different losses contribute to different aspect of the performance. Does this finding still hold in the summarization dataset?\n", "paper_topic_and_main_contributions": "This paper addresses the problem of effectively forgetting some information in large language models, in order to prevent some privacy leak or bias. The authors propose an unlearning layer to be trained without re-training on the entire model. Experiments are performed on both generative and discriminative tasks to show the effectiveness of the model.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mWIZ975bnW", "reviewer": "Reviewer_sR7i", "summary": "", "questions": "1. [Section 3.1] For the language modeling part, what is the percentage of masked tokens? Are they randomly masked?\n2. [section 4.4] What is the architecture of the unlearning layers? \n3. [Section 4.5] According to Table 1, EUL has a much lower accuracy than the re-train model on the forgot set (e.g. 65.7 vs 92.5 for T5-base). From my understanding, an ideal unlearning model should mimic the re-train model's behavior and make it hard to tell the difference between the forgotten data and unseen data. Will the extremely low accuracy make the forgotten data easily detectable?\n4. [Section 4.6] In Table 5, why does removing LM have such a large impact on the retained set? The accuracy drops from 100 to 69.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments. We humbly think that some concerns might be caused by misunderstanding, which we will explain in detail below. We hope that our response can clarify the misunderstandings and you can consider our work more favorably in the ratings.\n\n### On the Membership Inference Attack (MIA)\nWe actually have evaluated the MIA attack in our preliminary studies when performing unlearning on the IMDB dataset where we tried if we could learn a binary classifier to predict whether a data belongs to the forgotten set or retained set based on the representations. And we found that the classifiers could not converge so well on the training set and always had a low accuracy on the test set both before and after unlearning (e.g., 0.542 before unlearning and 0.566 after unlearning). The results are shown in the table below. These showed that the randomly deleted data could not be easily inferred both before and after the unlearning. As a result, we did not present the studies in the main paper. We would add this study and results to the revised version.\n| Models | Original Model | Re-train | Fine-tune | SISA  | Reverse-Gradient | EUL   |\n|--------|----------------|----------|-----------|-------|------------------|-------|\n| MIA    | 0.542          | 0.550    | 0.568     | 0.585 | 0.626            | 0.566 |\n\n### On the evaluation metrics\nAs stated in the limitation section in the paper, we mainly follow the previous work to measure the unlearning through performances on the test set, retained set, and forgot set, together with the MLM loss. While there does not exist  a well-defined evaluation metric, we hope our study could facilitate future research on evaluating unlearning LLMs. The reason why we are utilizing MLM Loss as an additional metric is that we want to measure whether the information in the data that needs to be forgotten can be extracted from the LLMs through the token predictions. This factor has not been considered before while being raised up as a risk issue for LLMs where identifiable information like names, phone numbers, email addresses, and even bank account numbers can be easily extracted from LLMs through token predictions in a given context [1]. And we optimize such objectives during training. This is similar to MSE Loss when training regression models where L2 distance is used in both optimization and evaluation. \n[1] Carlini, Nicholas, et al. \"Extracting training data from large language models.\" 30th USENIX Security Symposium (USENIX Security 21). 2021.\n\n### On the concern about the selection of tasks\nWe utilize two representative tasks (classification and summarization) to evaluate the effectiveness of the unlearning methods. And we utilize the MLM Loss to measure whether the deleted personal information (such as names, address) could still be inferred in a given masked sequence. Our unlearning methods could achieve a higher MLM loss which means that it is harder to extract deleted information from the models. For example, after unlearning an utterance (“John: I will pick Bob up on Main Street.”)  In the conversation summarization dataset, the model after our EUL could not predict the “Bob” for the masked sequence (“John: I will pick [MASK] up on Main Street.”) which prevents the personal information from being extracted from LLMs. Again, as a starting point, we perform all the experiments in simulated settings. However, our methods could be easily applied to real-world applications to deal with actual use cases.\n\n### On the question about the masked tokens during the training\nThe tokens are randomly selected to be masked and the ration is 15\\%.\n\n### On the question about the  architecture of the unlearning layers\nWe utilize the two feed-forward linear layers in one unlearning layer.\n\n### On the question about detecting forgot data\nAs stated in the response to the first concern, we have performed the MIA attack in our early experiments and we found that they can not be easily inferred both before and after the unlearning. We would add the studies in our revised version.\n\n### On the performance on the retained set after removing the LM loss\nWe made a typo for the number and the performance on the retained set should be 99.0. We would correct the typo in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Machine unlearning is a very interesting and underexplored area. The proposed method is promising when removing mislabeled data is needed.\n2. The proposed method is straightforward and easy to implement. Also, the method can deal with a sequence of forgetting data requests.", "reasons_to_reject": "I have three main concerns:\n1. I doubt the proposed method can work as well as they claim in the paper under the privacy protection scenario. The proposed method encourages getting a high error on the forgotten data. However, it also makes the forgotten data identifiable and vulnerable to attacks. I think adding metrics like Membership Inference Attack (MIA) is necessary. \n2. The metrics used in the paper are not sufficient and some of them are inappropriate. As I mentioned in 1, the paper lacks metrics like MIA to show the model's performance on unlearning privacy-related data. Moreover, I do not understand why the MLM loss on the forgotten data is used as both an optimization objective and an evaluation metric. \n3. The paper only reports results on two datasets. I hope to see the results on some text completion tasks to check if the model can really avoid generating user-related private information.", "questions_to_authors": "1. [Section 3.1] For the language modeling part, what is the percentage of masked tokens? Are they randomly masked?\n2. [section 4.4] What is the architecture of the unlearning layers? \n3. [Section 4.5] According to Table 1, EUL has a much lower accuracy than the re-train model on the forgot set (e.g. 65.7 vs 92.5 for T5-base). From my understanding, an ideal unlearning model should mimic the re-train model's behavior and make it hard to tell the difference between the forgotten data and unseen data. Will the extremely low accuracy make the forgotten data easily detectable?\n4. [Section 4.6] In Table 5, why does removing LM have such a large impact on the retained set? The accuracy drops from 100 to 69.", "paper_topic_and_main_contributions": "This paper proposes to unlearn certain data by adding unlearning layers to the original model. During training, only the unlearning layers are tuned to minimize the final objective which contains three parts: L_KL, L_TASK, and L_LM. Moreover, the paper also studies fusing different unlearning layers to handle multiple forgetting requests. The method is tested on one classification task and one generation task.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. [Line 002] on and => remove \"on\"\n2. [Line 191] update F(.) with F(.) => update F'(.) with F(.)\n3. forgot data, forgotten data => please make them consistent"}, "extra_text": {}}, {"review_id": "RpFGd0SIP8", "reviewer": "Reviewer_41eo", "summary": "", "questions": "1. Can you show the time series experimental results of the sequence of forgotten requests?\n2. With the rapid of the LLMs' size, can unlearning layers fit well during training and remain the performances?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your reviews. We hope our response below could solve your concerns.\n\n### On the evaluation on the fusion mechanism\nThe fusion mechanism is proposed to handle the sequence of unlearning operations. We evaluate the effectiveness of our proposed fusion mechanism through the experiments where we gradually forget a set of data in an sequential way (sequence of removals in Section 4.5) and the results are shown in Figure 2. The results demonstrate that our proposed fusion method could effectively handle the sequence of deletion (achieving higher accuracy on the test set and lower accuracy on the forgot set compared to baseline models) especially when the sequence length gets longer compared to baseline models.\n\n### The results for sequence of removals\n\nFor every length of the removals, we test all the possible sequences and report the average performance. For example, when removing 2 sets (set 1, set 2), we test the performances of different removing sequences: set 1->set 2 and set 2->set 1. And this kind of evaluation is used in previous continual learning work [1]. We would add all the performances of every sequence in the Appendix in the revised version. \n\n[1] Huang, Yufan, et al. \"Continual Learning for Text Classification with Information Disentanglement Based Regularization.\" Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.\n\n### On the concern and question about effectiveness on different backbone and larger models\n\nWe mainly performed experiments on T5-base/3b to illustrate the effectiveness of our introduced unlearning methods. We did not test on larger LLMs because of the computation limit. Actually, the designs of our ELU architectures and methods are agnostic to different types of pre-trained models. In our early experiments, we have tested on RoBERTa-base models on the classification tasks (IMDB) and we observed the gains of our methods (after unlearning 10\\% of the IMDB data, the accuracy on the test set, retained set and forgot set are 0.946, 1.0, 0.428). \n\nWhat’s more, from our experiments on models with different sizes, we could observe a consistent forgetting ratio (e.g., 36\\% on T5-base and 34.9\\% on T5-3b on classification tasks when forgetting 10\\% of the data.). \n\nNote that, as our model architectures are similar to adapters [1] which have been shown to be applicable to most recent larger models like LLAMA models, we think our EUL would fit well for larger models.\n\n[1] Zhang, Renrui, et al. \"Llama-adapter: Efficient fine-tuning of language models with zero-init attention.\" arXiv preprint arXiv:2303.16199 (2023).\n\n### On the questions about the time series results of the sequence of forgetting requests\nIn our experiments, we have actually tested on all the possible sequences for a given sequence length. For example, when the length of the forgetting requests are 2 (set 1, set 2), we have tested on the sequence (set 1->set 2) and sequence (set 2->set 1) and averaged the final performances. We would add all the results on different sequences in the revised version. \n\nHere, we show the results (accuracy on the test/retained set) of one possible sequence whose length is 3 (set 2 -> set 1 -> set 3) below as an example. \n| Methods          | set 2     | set 2,1   | set 2,1,3 |\n|------------------|-----------|-----------|-----------|\n| Re-train         | 92.7/91.4 | 92.5/90.8 | 91.3/90.0 |\n| Fine-tune        | 92.8/96.0 | 92.1/94.0 | 91.0/93.3 |\n| SISA             | 92.2/90.4 | 92.0/87.8 | 91.2/85.8 |\n| Reverse-Gradient | 91.5/67.9 | 90.5/67.2 | 89.8/66.0 |\n| EUL              | 93.0/64.6 | 92.1/64.8 | 91.0/64.2 |\n| EUL-fuse         | 93.0/64.6 | 92.8/62.2 | 92.4/60.8 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Due to the rapid development of large-scale language models, this topic needs to be focused on.\n2. This paper proposed an efficient unlearning method for LLMs to forget data while retaining the performances of the model.\n3. The experiments show the obvious improvement achieved by the EUL.", "reasons_to_reject": "1. The effective of fusion mechanism has not been evaluated in the experiments.\n2. For the sequence of forgotten requests, the results of experiments during the temporal process should be presented, the same as the continual learning, in order to verify that the model will not recall previously forgotten content.\n3. The proposed EUL is validated only on specific LLMs (T5-base and T5 3b), and more comparisons with different LLMs in different sizes and backbones need to be conducted. Especially, with the rapid of the LLMs' size, can unlearning layers fit well during training and remain the performances?\n4. Technical writing skill needs to be improved.", "questions_to_authors": "1. Can you show the time series experimental results of the sequence of forgotten requests?\n2. With the rapid of the LLMs' size, can unlearning layers fit well during training and remain the performances?", "paper_topic_and_main_contributions": "This paper mainly focuses on handling unlearning requests for LLMs. An efficient unlearn method, that freezes the LLMs' parameters while inserting trainable unlearning layers into transformer layers to forget requested data,  is proposed combined with a fusion mechanism for fusing the multiple unlearning layers trained by obtained by a sequence of unlearning requests. The experiments show the obvious improvement achieved by the proposed method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of effectively forgetting some information in large language models

The authors propose a simple but effective method to teach the model unlearn without any retrain steps.
Several experiments are conducted to test the model on both classification and generation tasks under two base models.
The motivation and method parts are clear and the experiments are reasonable.
Machine unlearning is a an interesting and underexplored area. The proposed method is promising when removing mislabeled data is needed.

Experiments should have been performed on larger LLMs such as Llama.
The contribution of individual components could be evaluated better.
Several important questions are not considered - whether the model could recall previously forgotten content, and whether the method makes forgotten data identifiable.
Decision: Accept-Main