On Large Language Model Continual Unlearning
Authors: Chongyang Gao
Lixu Wang
Kaize Ding
Chenkai Weng
Xiao Wang
Qi Zhu
Venue: ICLR 2025 Poster (2025)
Keywords: Continual Unlearning
Large Language Models
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: While large language models have demonstrated impressive performance across various domains and tasks, their security issues have become increasingly severe. Machine unlearning has emerged as a representative approach for model safety and security by removing the influence of undesired data on the target model. 
However, these methods do not sufficiently consider that unlearning requests in real-world scenarios are continuously emerging, especially in the context of LLMs, which may lead to accumulated model utility loss that eventually becomes unacceptable.
Moreover, existing LLM unlearning methods often ignore previous data access limitations due to privacy concerns and copyright protection. Without previous data, the utility preservation during unlearning is much harder.  
To overcome these challenges, we propose the \OOO{} framework that includes an \underline{\textit{O}}rthogonal low-rank adapter (LoRA) for continually unlearning requested data and an \underline{\textit{O}}ut-\underline{\textit{O}}f-Distribution (OOD) detector to measure the similarity between input and unlearning data. The orthogonal LoRA achieves parameter disentanglement among continual unlearning requests. The OOD detector is trained with a novel contrastive entropy loss and utilizes a glocal-aware scoring mechanism. During inference, our \OOO{} framework can decide whether and to what extent to load the unlearning LoRA based on the OOD detector's predicted similarity between the input and the unlearned knowledge. Notably, \OOO{}'s effectiveness does not rely on any retained data. We conducted extensive experiments on \OOO{} and state-of-the-art LLM unlearning methods across three tasks and seven datasets. The results indicate that \OOO{} consistently achieves the best unlearning effectiveness and utility preservation, especially when facing continuous unlearning requests. The source codes can be found at \url{https://github.com/GCYZSL/O3-LLM-UNLEARNING}.

Published as a conference paper at ICLR 2025
ON LARGE LANGUAGE MODEL CONTINUAL UNLEARNING
Chongyang Gao Lixu Wang!~; Kaize Ding', Chenkai Weng", Boong"XXnn Xiao Wang', Qi Zhu'
Worthwestern University, "Arizona State University
{cygao, 1ixuwang2025 )u. wwwthwestern edu, chenkai, wengdasu. [kaize. ding, wangxiao, getu///norttwwwsterr.cmmmm
ABSTRACT
While large language models have demonstrated impressive performance across various domains and tasks, their security issues have become increasingly severe. Machine unlearning has emerged as a representative approach for model safety and security by removing the influence of undesired data the target model. However, these methods do not sufficiently consider that unlearning requests in real-world scenarios are continuously emerging, especially in the context of LLMs, which may lead to accumulated model utillity loss that eventually becomes unacceptable. Moreover, existing LLM unlearning methods often ignore previous data access limitations due to privacy concerns and copyright protection. Without previous data, the utility preservation during unlearning is much harder. To over- come these challenges, we propose the Oa framework that includes an Orthogonal low-rank adapter (LoRA) for continually unlcarning requested data and an Out- Of-Distribution (OOD) detector to measure the similarity between input and un- learning data. The orthogonal LoRA achieves parameter disentanglement among continual unlearning requests. The OOD detector is trained with novel con- trastive entropy loss and utilizes glocal-aware scoring mechanism. During infer- ence, our O framework can decide whether and to what extent load the unlearn- ing LoRA based on the OOD detector's predicted similarity between the input and the unlearned knowledge. Notably, o effectiveness does not rely on any re- tained data. We conducted extensive experiments on O and tfate-of-the-aee art LLM unlearning methods across three tasks and seven datasets. The results indicate that 0 consistently achieves the best unlearning effectiveness and utility preservation, especially when facing continuous unlearning requests. The source codes can be found at https::::/
http://ccc//////////// LLM UNLEARNING
INTRODUCTION
Recently, bolstered by scaling laws (Kaplan et al., 2020), the size of language models has grown tremendously, demonstrating excellent performance across various tasks (Wang et al., 2024). How- ever, concerns about large language models (LLMs) have also increased, particularly regarding how to eliminate undesirable data influence (e.g., privacy information (Pan et al. 2020)). To address this issue, machine unlearning (Bourtoule et al. 2021 is applied in LLMs to remove private, toxic, or illegal data. Current methods for LLM un anlearning
be primarily categorized into parameter optimization (Chen Yang 2023 Eldan & Russinovich 2023 Iia et al.] 2024) Zhang et al.] [024]] Meng et al., 2022;] Li et al. 2024), and in-context unlearning (Thaker al. 2024] Pawelczyk al., 2024). The parameter optimization methods involve directly fine-tuning the I.MM, with the objective typically being to maximize the task loss on the unlearning data or to minimize the random label loss. Some methods identify the related parameters and then make appropriate modifications. In- context learning- based methods modify the LLM input prompts make the LLM refuse to output content related to the unlearning data. Regarding unlearning effectiveness, parameter optimization typically much more effective than in-context learning.
However, these methods still often poorly maintain the model utility outside the unlearned knowl- edge, especially in real-world continual settings. The challenges are two-fold: (i): First, in addition
"Equal contributions (ordered aicteredalphaate alphabetically); 'Corresponding author.

---

Published as a conference paper at ICLR 2025
to the data that needs to be unlearned, existing unlearning methods also require large dataset called the retained dataset to maintain the model utility. This retained dataset often consists of the original training dataset (Bourtoule et al. 202T) portion of it, but as LLMs are trained on massive datasets (Wang et al., (l,22) assuming access to the complete training data is typically unrealistic (Liu al.] 2024). Moreover, time goes on, the original training data of LLMSs may become inaccessible due to expired access authorization, data privacy, and intellectual property pro- tection (Sun et al. 2024). If the retained dataset only contains incomplete training data distribution, the model utility of the missing parts significantly declines after unlearning. Although some studies shrink the range of the retained data to the distribution most susceptible to unlearning, this distri- bution itself is hard to characterize and its data may be limited due to intrinsic rarity and privacy protection (Chang et al. 2024 Huang et al. 2024). (ii): The second challenge is that existing LLM unlearning methods only consider single operations and cannot perform effective continual unlearn- ing. LLM unlcarning is often not a one-off operation but a continual process, as unlcarning requests continuously emerge in the real world (Liu et al. 2024). As the number of unlearning operations increases, the aforementioned decline in model utility will also have a cumulative effect, even with the retained dataset, meaning that the model's general capabilities will significantly decrease Gu iet al. (2024); Gupta et al. 2024) over time.
In this work, achieve more effective continual unlearning for LLMs, propose the Oa frame- work, which can balance unlearning effectiveness and model utility preservation in continuous scenarios without using any retained data. At high level, the O framework mainly includes an Orthogonal Low-rank adapter (LoRA) (Hu et al. 202I) for continuously unlearning requested data and an Out-Or- -f--st--iitittiionn (OOD) detection module assess the similarity between input data and unlearning data. Specifically, the orthogonal LoRA in o enables the disentanglement of parameter space across different unlearning requests, ensuring that the unlearning effectiveness of different requests does not interfere with each other. Then the OOD detector in O is trained with novel contrastive entropy loss as its backbone and supplemented with a glocal-aware scoring mechanism. The 6 framework can balance unlearning and utility because it smartly leverages the data similarity determined by the OOD detector to decide whether and to what extent to load the unlearning LoRA during inference. In summary, the main contributions of this work include:
We study the underexplored problem of LLM continual unlearning and tackle the challenge of balancing unlearning effectiveness and model utility preservation when LLM faces the continuous arrival of unlearning requests, without using any retained data.
We propose a novel 0 framework that includes an orthogonal unlearning LoRA and an OOD de- tector. The orthogonal design of LoRA prevents interference among different unlearning requests, achieving better unlearning effectiveness in continuous scenarios. The OOD detector measures the similarity between input and unlearning data, allowing Oa to decide whether and to what extent to load the unlearning LoRA during inference.
We conduct extensive experiments on multiple benchmark tasks that comprehensively test the LLM continual unlearning on discriminative, generative, and reasoning tasks. The experiment results demonstrate that O3 consistently achieves the best balance between unlearning effective- ness and utility preservation without using any retained data, compared with many state-of-the -f-th-o baseline methods when facing continuous unlearning requests.
PRELIMINARY
Language Model Unlearning. Machine unlearning (Bourtoule et al. 2021 is proposed protect data privacy and ensure authorized usage (Liu et al.
et al. (023). There have been
approaches achieving unlearning through parameter optimization (Jang et al. 2023 Golatkar
et al.
2020 Yao et al.] [023 Eldan & Russinovich] 2023 Zhang et al. 2024 et al.) 2024| Meng
et al.
2022
Yu et al., 2023 Wu et al. Li et al. (024)) and in-context learning (Thaker a
2024
Pawelczyk et al. 2024). The iitimization-baseee unlearning is to employ GradAsc (Golatkar
ct al.
2020; Yao et al. 2023) on the unlearned data. The following approaches, like PO (Eldan &
Russinovich 2023 Zhang et al. 2024 Jia et al. 2024), notice that unconstrained GradAsc hurts the model's 's utility, thus crafting task labels through shuffling or rejection. Yu et al. (2023) localizes the model parameters related to unlearning data and updates them through merging or subtracting (Ding jet al., 2023). The in-context learning-baat based methods adjust input prompts to reject unwanted con- tent generation. Although these approaches can achieve unlearning in certain cases, they neglect that
2

---

Published as a conference paper at ICLR 2025
unlearning requests always appear continuously (Liu et al. 2024). The challenge is that the accu- mulative impact of conducting unlearning sequentially induces catastrophic model utility loss (Gu et al. 2024; Guppt al. 2024). Besides, their success in single unlearning operation relies on sufficient retained data with accurate labels, which are scarce in real-world L.M applications. Our
0 addresses the above challenges via orthogonal unlearning LoRA and designed OOD detector.
Detection, Unlike traditional OOD detection based on one-class SVM (Erfani ct al. 2016), random forest (Primartha & Tamaa 2017). and Gaussian mixture modeling (Laxhammar et al., 2009). deep learning-basee OOD detection (Yang et al., 202T) has become mainstream, focus- ing on the classification task. These approaches assume the availability of category- labeled ID data. However, only few studies explore unsupervised OOD detection in language models with only un- labeled ID data. DAGMM (Zong et al. 2018) uses deep auto-encoder to generate low-dimensional representations to estimate OOD scores, only working multivariate time-series data. Xu et al. (202T) leverage the feature extracted from pre-trained language model and then an OC-SVM for detection. More recent related works search ways like ensemble learning (Zhou et al. 2023),
Out-of-distribution
pseudo labeling (Lang et al. 2022), outlier exposure (Cao et al. 2024). and prefix tuning (Ouyang et al. 2023) achieve OOD text intent detection. Although these works achieve OOD detection over non-classification tasks, their effectiveness still relies on labels of specific tasks. Our proposed 0 does require any types of labels.
Problem Definition. In the context of LLM,, we consider the prevalent causal language mod- els, which take textual token sequences with variable lengths as the input. As autoregressive model, the target LLM Me parameterized on & calculates the probability of each token in a text with the previous tokens. For the problem of unlearning, we suppose sequence of contin- ually arriving unlearning requests with U, data samples, i.e., [DU. where, for 1-th request, DU. = {ax:] al; " and Pu is the input marginal distribution. T is index of the latest request. Each unlearning request corresponds to a task with the training data characterized by the input P4 and label distributions Pj. Traditional unlearning approaches always assume retained dataset drawn from a distribution x
RR. disjoint from the unlearning one U.t to retain the model performance on the original training distribution. Then, on one hand, the straightforward objective of continual unlearning is:
T
min I(M 6
py (2, θ); M uup
u.. (x, O²)),
Σ
max I(M 6
pxpp (2, θ); M p.
R. (xx, O²)),
(1)
where M' parameterized on O' represents the target model during and after unlearning on the t-th unlearning set DU,,, and I(::-) calculates the mutual information between two random variables. On the other hand, the model utility preservation on other distributions Po distinct from the unlearning one, is another important objective for unlearning.
max I(M (Manpo θ); M 0¹
xupo po O²))
3 METHODOLOGY
Proposed Approach Overview. To handle continual unlcarning requests for LLM, especially con- sidering the constraints in acquiring the retained dataset, we propose framework called O that contains two major modules: an Orthogonal LoRA module to unlearn requested knowledge con- tinually and an OOD detector-like module to detect unlearning knowledge, detailed Sections 3.1 and 3.2. respectively. 0 continuously unlearns the requested data using LoRA with an orthogonal regularization loss that can maintain continuous unlearning performance. O obtains the unlearn- ing knowledge detection module by the contrastive entropy minimization and local-global layer- aggregated scoring techniques, which can predict the probability that the input data sample belongs to the unlearning distribution P These two major modules only use the unlearning dataset of each unlearning request and do not require the access retained data. After unlearning, O works with an effective inference mechanism (Section 3.3). in which the unlearning LoRA is loaded with soft weights originating from the probability predicted by the OOD module to produce distinct outputs for different data.
3

---

Published as a conference paper at ICLR 2025
Continuel
Unlearning Request
Unlearning Requests
Unleorn The Lord of the Rings
Unseen Date Related to The Lord of the Rings
Ramdom Label
Mark Twain
000 Score
Huckieberry Finn
Lax T
Pre pretrrinedd
OOD
Pre-trained
LUM
Lovn
A
Detector
LLM
Who the writer The Lord the Rings?
Who is main character The Lord of the Rings?
Training Pipeline
Soft-weighted Inference
The overview of o framework to handle continual unlearning requests for LLM without using any retained data. 0 includes major components: Orthogonal optimization process for unlearning requested knowledge, and an OOD detector used to detect whether the input contains the unlearning knowledge. The unlearning knowledge optimization uses the orthogonal loss (Cort) to prevent interference among different unlearning requests. The OOD detector is trained by novel contrastive entropy loss (LCEl) and works with a layer-aggregated scoring mechanism that leverages cosine similarity (dcos) and Mahalanobis distance (dMaha). In the inference phase, the OOD detector decides whether and what extent load the unlearning LoRA.
Figure I:
3.1 LLM UNLEARNING WITH ORTHOGONAL-REGUUAAAOO REGULARIZED LORA
As the model size of LLM is quite large, the associated costs of fine-tuning- unlearning are extremely massive. Moreover, continual fine-tuning of the entire LLM also poses challenges of preserving the general capabilities and sustaining the efficacy of previous unlearning efforts (Gu al. 2024). To address these issues, we use LoRA and incorporate an orthogonal regularization by keeping the original LLM frozen and focusing updating previously neglected LoRA parameters.
LoRA (Hu et al., 202T) employs low-rank matrix decomposition reduce the trainable parameters. A transformer's s attention or linear layers are parameterized on weight matrices, W ER LoRA introduces two low-rank trainable matrices := {A, B} where ensures that the dimension of ABh matches that of W h where h is the input of the weight matrix,
RU*K and RXVV This
(3) where h' is the output of the weight matrix. During LoRA fine-tuning, W is frozen, and only A and B are updated. To achieve effective unlearning, we leverage preference optimization to update the model for random task labels or reject-based answers such as 'I don't know", termed as ÿ. We adopt CrossEntropy Loss of the unlearning set DU, for training.
h' Wh + WW Wh h + ABh,
U..
Σ
U. log Me(
(4)
NU,
To further reduce the cost during unlearning, we leverage single LoRA for all unlearning tasks rather than employing multiple LoRAs. However, this adoption compromises the effectiveness of previously unlearned requests due to task entanglement. To address such issues, inspired by Wang et al. (2023), we propose to adopt an orthogonal regularization loss to disentangle unlearning quests in the LoRA parameter space. Specifically, the I-th unlearning request, we initialize The matrix A can be further denoted as "1_ where each a 1-1, is column vector with dimensions. Note that the matrix B can be regarded as the linear weights of matrix A. Thus, we don't 't consider its interference among different requests. We view the spanned space that consists of column vectors A' as the t-th task parameter space 4t. In this case, if the LoRA parameter space of different requests achieves orthogonal, their spanned space will be:
the LoRA with the latest ,d-1 {A'-1 B' ('
[a 1,1
Yu 4², Vu E - --===--
(5)
This design guides the LoRA tuning focus more previously overlooked parameters, striking balance between different unlearning tasks. Therefore, we can approximate this correlation with an orthogonal loss Lorth and combine it with the preference optimization form the total loss of the LoRA tuning LLORA.
Forth [[AA-11TA\11 and CLORA C' + XCo
4

---

Published as a conference paper at ICLR 2025
where À A= 0.1 is factor used to balance the optimization guidance, and please refer its sensitivity analysis in Section 4.3 Initially, when t= 1, there is no need to apply the orthogonal regularization. As the unlearning continues, the orthogonal loss is incorporated.
3.2 UNLEARNED KNOWLEDGE DETECTION
By viewing the unlearning dataset as in-distribution (ID) data, our task of unlearning knowledge detection becomes an OÔD detection problem (Yang et al. 202T). However, nearly existing OOD detection works are built upon the classification problem, which is not the mainstream task of language models. Besides, their representation learning and scoring mechanisms rely on semantic category labels that are inaccessible in our problem. To tackle these shortcomings, propose new designs for the representation learning and scoring mechanism for LLM continual unlearning.
OOD Representation Learning with Contrastive Entropy Minimization. In unsupervised sce- narios, self-supervised learning (SSL), such as contrastive learning, has been demonstrated effec- tively for OOD detection on visual data (Tack al. 2020). However, such contrastive learning unsuitable for language OOD detection, as it is hard to achieve semantically equivalent data aug- mentation on texts. Moreover, token-level SSL tasks, like Masked Language Modeling (MLM) and SimCSE (Gao et al. 2021: Jian et al., 2022a b), are far less effective. To address these challenges, we propose a novel contrastive entropy loss to learn text representations for OOD detection.
Our contrastive entropy also starts with the augmentation view generation. Inspired by MLM, leverage random masking to generate the first view type. Specifically, for a particular text instance with the token length n, randomly select p% (p 15 in implementation) tokens and replace them with the token of [MASK]. The instance with the random masking is denoted as x². For the second contrastive view, we follow MoCo (He et al.] 2020) to maintain a key encoder Fiues initialized from the original OOD module backbone Pn that is a transformer consisting of L attention layers, i.e., F :- fuz o... fux .... Swz. Then the second view is generated from Fakey by forwarding the original text instance
Our contrastive entropy loss shares similar intuition with the standard contrastive learning, i.c., aligning positive pairs as close as possible while pushing negative pairs far away, but converges much faster owing to weighting by cosine similarity- Softmax probability and being conducted at every model layer. In the mini-batch training, our contrastive entropy loss shapes like
N L R
exp(funaa (xx)) f (xx,))
LCRL.
(o. where A(i ΣN exp(Junny (x²) key (xx))
"(1i))
where fui (xt) means the token averaging representation of the 1-th layer. N B is the sample quan- tity of a mini-batch. The convergence condition of entropy loss is known to be that one dimension holds the probability of I, corresponding the state that the maximum similarity between positive pairs and minimum similarity between negative pairs. Considering the forgetting of the pre-trained knowledge, we also supplement LCEL with the MLM loss form the total loss LOOD.
N
LOOD LCEL. + LMLM, where LMLM
Σ
vi log Fa
(8)
NB
where y² is the random token masking label. When dealing with the continually arriving unlearning requests, first randomly divide the unlearning dataset DU, into two subsets DU, and D with N,U and (1 - NU. samples (a 80% in our implementation). respectively. Then UU, is used to compute LOOD and train the OOD detector backbone Fg. D.E.i. will be used in the scoring (Section 3.2 and inference mechanisms (Section 3.3
Glocal-aware OOD Scoring. We design glocal-aware scoring mechanism to supplement the contrastive entropy representation learning. On the one hand, we assume all ID data form a global Gaussian distribution at each model layer and leverage Mahalanobis Distance (De Maesschalck al. (000) to quantify the chance of outliers. On the other hand, considering the inaceuracy of fitted Gaussian distribution, especially when the available ID data is few-shot biased, we emphasize
5

---

used also need to calculate the vectors of D U.t. rest and store all these Published as a conference paper at ICLR 2025
each ID data instance and compare instance-wise cosine similarity locally. Specifically, at each unlearning request (we omit below), for each layer fwy. we calculate the empirical mean and covariance subset D
used.
NNU Σ
aN
1
1
N:
NNNU
-
-M))
(9)
Next, we can calculate the Mahalanobis Distance for each testing sample æ at the 1-th layer,
(Maah(()) ( px) T (Jupoa (x) pt),
(10)
where En is the inverse of Σj. The Mahalanobis Distance measures the probability density of funan (z) in the estimated Gaussian distribution of the (-th layer.
As aforemmenioned, assuming Gaussian distributions for all layers are arbitrary, thus we leverage another distance based the maximum instance-wise cosine similarity dcus (æ)) of testing sample a to all ID instances in D
used for each layer, and obtain combined score s(x)),
o.V' [ Sunna (2:) Sunja (x
dC (z)) = maax i=1
[ (x))lfwwaa (xx and ((x)) dMaha (2)); + dcaa( (11)
where y = 1000 scaling factor that unifies the order of magnitude. To achieve effective OOD detection, module is needed that allows us to determine the score range of ID and OOD data. Inspired by Xu et al. (202I); Darrin et al. (024)) we leverage one-class SVM (OCSVM) based on the layer-aggregated score vector s(x) [s(z),. ... s(z),, detection, OCS specifies a hypersphere instead of a hyperplane to characterize the ID data. is
s(x) to do so. In OOD
feasible in our problem the score of each model layer is a scalar, and then the concatenated score
vector conforms to a hypersphere distribution.
When dealing with each unlearning request, after conducting representation learning in Section 3.2 calculate score vectors sax for all samples in DU Then use these score vectors to fit an OCSVM with the fitted hypersphere H² R') characterized by a center vector ct and the radius R In addition to D D
vectors as we cannot access the unlearning data after the unlearning.
3.3 SOFT-WEIGHTER INFERENCE
Assuming the training of 0 framework stops temporarily when finishing the T--th unlearning re- quest, let us introduce how to leverage it during inference. Specifically, for each testing instance æ, should be fed into all OOD detector backbone {F 1²) where each 1s ;= F rf consists of the original F and the LoRA adapter rip of the t-th request. Then the score vector of æ for each Frr can be computed sxx)² via Eq. 11 The score vector is forwarded to the OCSVM to obtain the distance from to the boundary of the hypersphere H' H''c', It").
(12) As aforementioned, for the unlearning dataset DUt each stage, we randomly split 80% sam- ples into U,A to train the OOD detector backbone and fit the OCSVM, while the rest D is used for the following soft weighting. First, for all samples of DU.t and D we calculate their dæ via Eq. 12 and fit two Gaussian distributions as (uldx (D beod)) o² [dx (D (DuA))) and (uudde (D ), a² [dx (Du:)) respectively. Then we mix these two Gaussian distributions with equal weights and denote the cumulative distribution function as P Next, the probability center of the mixed Gaussian distribution can be determined as d where P (dy
(x) = [s(xr) -c'l- R¹.
(c)) = 0.5. In the
end, for each testing instance æ, we have the following weight that reflects how much content of belongs to the unlearning distribution of the t-th request,
w(x)² = [[[[ max(p. p') + min(p,p')) p')]}, where p=P (dz, (z)). =p mix (2dg. dæ (x)).
(13)
where is a Sigmoid function that scales the weight into (0, 1), and Ç < 10 is a scaling factor for more fine-grained sensitivity. After getting the weights from all OOD detectors, adopt the

---

Published as a conference paper at ICLR 2025
maximum weight wxx) max(w(x)'+ ,,(,),))''))))))))) to load the unlearning LoRA by modifying Eq.3imm h' = w ABh. In this case, the imput æ is sequentially forwarded to all attention layers of the target LLM to obtain the final inference results. A higher w(x) implies that is close to at least one unlearning distribution, thus we should load the unlearning LoRA. In contrast, w((x)) is relatively low, detaching the LoRA while using the original model makes more sense. The algorithm pipeline is provided in Appendix
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Datasets. In the main context, conduct experiments on three tasks: Question Answering, Fic- titious Knowledge Generation, and Intent Classification by unlearning different types of subsets continuously while maintaining the utility ability. Appendix E.2lpoovsdes more details.
Question Answering. For ScienceQA (Lu et al. 2022b we gather text-only samples to form train and test set with 6,508 and 2,224 samples. We choose four domains in ScienceQA as continual unlearning requests, i.c., biology + physics + chemistry - economics. We use Com- monsense (Talmor al. 2019) as a utility dataset, which contains 9,740 training samples and 1,221 validation samples for evaluating the commonsense reasoning capability of LLMs. Open- bookQA (Mihaylov et al. 2018) can assess the book comprehension ability, consisting of 4,957 training, 500 validation, and 500 testing samples.
Fictitious Knowledge Generation. TOFU (Maini et al. 2024) consists of questions about fake authors synthesized by GPT-4. There are three forget-sets: "forget01', 'forget05', and 'forget10', corresponding to 1%, 5%, and 10% randomly selected authors, which are used as three continual unlearning requests. Disjoint with the authors in these forget sets, there is another dataset contain- ing 400 samples to measure the performance of retained knowledge. Besides, TOFU includes two datasets related to Real-world Authors and World Facts test the utility preservation.
Intent Classification. CLINC150 mis 2020) is designed for intent classification, which comprises 150 classes across five domains, and each includes 200 training, 40 validation, and 60 testing samples. For the unlearning settings, we choose three domains most related to privacy ('work', 'travel', and 'home') as the continual unlearning requests. To evaluate the utility preservation, we leverage MRPC (Dolan & Brockett, (005) and RTE (Wang al.) on the task of paraphrase identification and textual entailment, respectively.
Evaluation Metrics. To evaluate the unlearning effectiveness, test the model performance every unlearning request on the unlearning train set (used in unlearning) and unlearning test set (unused in unlearning), which disjoint but from the same distribution, denoted as Sample- level Unlearning (S.U.) and Distribution-leve! Unlearning (D.U.). respectively. As for measuring the utility preservation, we consider the model performance on three distributions. The first the distribution most susceptible to unlearning requests, which we term Retained Distribution (R.D.). The other two are the utility datasets for each task (question answering: CommonsenseQA and OpenbookQA; fictitious knowledge generation: TOFU-Rea Authors (i.e., R.A.) and Word Facts (i.e., W.F.); intent classification: MRPC and RTE), and we denote their performance as U.I. and U.2., respectively. In addition, to evaluate the balance between unlearning effectiveness and utility preservation, we design metric called the Unlearning-Utiity Ratio (U²R). i.c.. U²R = (Acc$u. + Acchu - Accç.u - Accf.ur )/ /Accc.n AAc.........tt...... -Ace A.copT -AcoT...-AAA the accuracy. The higher the U²R, the better both unlearning effectiveness and utility preservation.
) where Acc means
Compared Baselines. To better demonstrate the effectiveness of our proposed methods, we im- plement series state-of-the-an language model unlearning approaches: GradAse Goolakkar et al. 2020), GradDif Yao et al. (2023). EUL Chen & Yang (2023). PO Eldan & Russinovich (2023), NPO Zhang et al. (2024). SOGD Jia et al. (2024) and SOPO Jia et al. (2024). We only conduct reasonable modifications customize them in continual unlearning settings.
GradAse(Golltkar
Implementation Details. Following TOFU Maini et al. 2024) and SOPO (Jia et al. 2024). we LLaMA2-7b (Touvron al. 2023) the target model. The used OOD detector backbone is Roberta-large (Liu al. 2019). All experiments are run repeatedly with three random seeds. We use the Adam W optimizer with 3e-4 as the learning rate and 128 the batch size for combined datasets.
7

---

Published as a conference paper at ICLR 2025
The epochs are 10 and 20 for Coa.ooooooooonssssssAAAAACoaooaa and CLINC150-MRPC: RTE. We set the LoRA rank for all experiments to 8. More details can be found in Appendix
4.2 EFFECTIVENESS OF CONTINUAL LLM UNLEARNING WITHOUT RETAINED DATA
We conduct experiments three tasks with continual unlearn requests and provide suffi- cient retained data for all comparison base- lines while assuming our O² framework only uses the data of each unlearning request and does not use any retained data. We first cal- culate the Unlearning- Utility Ratio, as shown
Uniearming--oe Uoaiinggggttttyyyyy Ratio
Gradoir
DUL
PO
SOPO
Ours
60-
in Figure 2 The effectiveness of O is evident as it always hits the highest U R and signifi- cantly surpasses the second best for all three tasks. Beyond achieving superior performance framework demonstrates
20-
Question Ficthlou Krewiedge Coneration
Intert Chaniicttior
Figure 2: Comparison between baseline approaches on Unlearning- Utility Ratio (U²R) that measures the balance between unlearn- ing effectiveness and utility preservation.
and other
across all tasks, enhanced data and parameter efficiency. As in- dicated in Table the quantity of training data required by our O framework is only half that of the baseline models since it does not necessitate using retained data. Moreover, the integration of LoRA significantly reduces the trainable parameters to 20M, which is even less than 3% of the baselines' 6,758M. Our additional inference computation overhead is only 5.6% higher than the baselines, as detailed in Appendix F.
Comparison between ours and other baselines on used training data quantity and train- able parameters. The trainable parameters of baselines are all the whole LLM
Table 1:
z0n0
Bil
Datract
Bouclince
Oun
(a) S.U. of ScienceQA
(b) D.U. of Science
SeiemasQ
Traisakke Paruneter
Training Dan Quonity
6,759M
4854
3627
20M
Figure 3: Unlearning effectiveness comparison between ours
TOFU
Training Quanity
675M
1290
640
and other approaches on (a) sample-level unlearning (S.U.), (b) distribution- level unlearning (D.U.) of ScienceQA.
CLINC150
Training Data Quamity
Traisable Parunetess
6.75MM
2400
20M
1200
1M
1000
Stadde
Dan
(a) Retained Distribution
(b) CommonsenseeA
(c) OpenbookQA
Figure 4: Utility preservation performance comparison between ours and oate-of-thhe-rn unlearning approaches the testing set of (a) Retained Distribution, (b) CommonsenseQA, (c) OpenbookQA, after unlearning each request of ScienceQA.
Question Answering. Figures 3(a) and 3(b) show the QA accuracy on the train and test set of un- learning data. We omit the results of GradAsc as failed to generate meaningful answers for all distributions. We can easily observe that our 03 is located at the bottom tier. with only GradDif and SOGD being lower than o²_ However, further examination of GradDif and SOGD revealed that they produce empty or nonsensical sentences filled with repeated tokens, which are considered an unlearning failure shown in Appendix F22 Moreover, the utility preservation of GradDif and SOGD is extremely poor, as shown in Figure 4 In contrast, the QA accuracy of o on the retained distribu- tion is slightly lower than the base model (the original target LLM without any unlearning). and o is even nearly the same the base model on CommonsenseQA and OpenbookQA. Therefore, can conclude that our O² framework provides a much better balance in unlearning effectiveness and utility preservation than all baselines when facing continual unlearning requests.
8

---

Published as a conference paper at ICLR 2025
Table 2: Performance Comparison between our O² and other baselines when continually unlearn- ing TOFU-forget01, -forget05, and -forget in Fictitious Knowledge Generation. The unlearning effectiveness is measured by the generation accuracy of the unlearning train data and unlearning test data denoted S.U. and D.U., respectively. Utility preservation is evaluated by the generation accuracy of Retained Distribution (R.D.), TOFU-Real Authors (R.A.), and World Facts (W.F.).
Method
S.U.
D.U.j
Unlearning Request
JR.D.T
R.A.1
W.F.
S..Us.
D.U.L
Unlcaming 2
[R.D.1
R.A.T
W.F.
S.U.4
D.U.1
Unlearning Request 3 R.D.D.
R.A.I
W.F.T
Base
85.0gz
90.0gu
|85.8a
89.0gz
86.3au
187,3a
89.32>
|85,8
89.0gu
80.3
[55
90.0au
85.8au
89.0gg
86.3au
GradAse
75.0gg
85.0g0
881.ag
17.66gu
23.1
19.0gs 0au
0ge
17.1
14:2aau
19.04u
Dau
GraDDif
78.
84.0g17
$1.911
86.7
83.5 3
62.5255
70.0g..7
70.4g3
66.74r
27.9
16.5ga
15.2
19.0ag
BUL
S11.sa
80.3.00
50.1.00
26.7.15
S7.1a0
$4.4.0
90.3g14
85,8
88.010
85.52
50.1gs
63.5195
83.4201
86.3.1
63.5105
PO
125. 16.0
78.4 a6.7.aa
83.8
30.51aa
48.8
82.
87.31a
85.2as
382
17.1
81-4104
88.3.s
81.4 81.4i0a
NPO
68.8 75.0ao
83.6an.
0.
81.8
76.3a22
84.2g3s
53.2s14
87.7ano
86.aas
77.6..
79.2aaa
81.4ao
87.3.as 82 9.15
S0GD
43.7 76.0g17
80.3ana
65.3
83-4
22.
12.8 24.0a3a
70.0g1a
81.3221
82.6.1.
17.4ea.
21.7a54
82.8ao
77.0 82 1.10
SOPO
25.6 38.0gun
83.7gms
85.3
83.741
34.1
37.5g22
81.5gg
87.8a1a
83.2
$4.0g35
40.0g1
80.221.7
80.7 84/230x
Ours
12.5.a.5
18..100 80.0.s 86.3go
[1.8800 20.3g3 (85.0ga
89.010
80.3.
18.7,07 84.9 88.8.2 86.1.0
Table 3: Performance Comparison between our O and other baselines when continually unlearning domain 'work', 'travel', and 'home' of CLINC150 in Intent Classification. The unlearning effec- tiveness is measured by the classification accuracy of the unlearning train data and unlearning test data denoted as S.U. and D.U., respectively. Utility preservation is evaluated by the accuracy of Retained Distribution (R.D.), MRPC, and RTE.
Method
S.U.j
D.U.
Unlearning Roguest
R.D.I.
MRPCI
RTE
S.U.I
D.U.
Unleaming Request 2
IR.DD
MRPCI
RIE1
S.U...
D.U.j
Unlzaming Request
IR.D.1
MRPC1
RTB!
Rose
|1000.g
99.9gs
00.840
88.0g0
88.7
100.040
98.94a
800
88.0g
88 7ap
[00.4
00.9gg
|00.8
89.0gg
88.7gu
GradDd
0.1
0.e
90.8,a4
39.9.34
31.6.5
040
0.o
127
9.0aas
0.80s
100
040
75.5.14
12.9
1.7
EUL
0.
98.3
87.2an1
68.1.c
0.1102
0,0
876
80,3a1
82.9
01n
023
81.
76.344
PO
26.3 26.7.14e
99.340s
84.1a0s
86.3-1a
396430
80.8
67.3a61
0203
26
56.7
00.0.a.4
86.3.0
870204
NPO
99.0gg
9022n0
87.9ga 88.4.04
99
983300
2922
87.2167
58.9
99.930
1
99.2
99.9.a.
87.0ga.
88.9
SOGD
0.a
Oue
8233n0
6.1ass
17.9-a.4
02n
Oze
33.
3.3g8s
19.5
19.5
Ozn
0.1gu1
04.021a
2.9.4.
23.7
SOPO
25.9
26.3.15.
80.5g0s
87.1g11
62.3g14
00.3
87.2g02
87.
58.8215
50 20.7
900629.
86.8.1.2 $600g14
Ous
10.33
18.5.0
98.9
81.8an 67.5.aa 50.5ana 35.6 191 87.0gns 80.3
4244ja
197.8
86.0g1 89.0goal
Fictitious Knowledge Generation. Table[[sreeents the experiment results on TOFU. According to these results, we observe that our O achieves the best in both unlearning effectiveness and utility preservation, providing the best unlearning effectiveness in almost all cases but one, and the best utility preservation in the majority of cases. In the one case where O3 is not the best for unlearning effectiveness (i.e., D.U. for unlearning request 3, the better ones GradAse and GradDif have almost completely lost model utility. We explain the metric details in Apppenix E.5
Intent Classification. Table S presents the experiment results CLINC150 dataset. Similar to QA, we observed consistent unlearning failures with GradDif, SOGD, and EUL methods. More- over, both GradDif and SOGD demonstrated extremely poor performance in preserving utility. In contrast, our O framework achieves the best unlearning performance and maintains comparable or better results to the baselines that use retained data, both on the retained distribution and util- ity preservation. For instance, our 3 framework preserves the RTE performance more effectively than all baseline methods.
More Experiments. In Appendix C and F.3 we found existing unlearning approaches perform much poorer when the retained data becomes more limited. In the Appendix, we further demonstrate experiments on scaling the unlearning with more requests (F.4), unlearning multiple knowledge en- tities per request (F.5), Membership Inference Attacks (F.6), Detoxification (F.7). unlearning unsafe behaviors on benchmark WMDP (E8), robustness of our 0 against targeted relearning attack (..)
the evaluation of unlearning in terms of the Oracle model fine-tuned with only retained data
and the quantity-limited unlearning data (F.10).
4.3 ABLATION STUDY
We conducted the ablation study as follows, and detailed the analysis in Appendix G.I
Unlearning Knowledge Detection. We detach the use of contrastive entropy loss LCEL in Oa and use SimCLR ("Ours w/ SimCLR') ) and MoCo ("Ours w/ MoCo') with the augmentation using token masking. As for the scoring mechanism, we try using Mahalanobis Distance (dMaha in Eq. 10) and Cosine Similarity (de.s in Eq. [T] separately, which are termed as *Ours w/o dees and 'Ours w/o dMaha'. . Besides, instead of using all model layers, we use only the last layer ('Ours w/ last layer').

---

Published as a conference paper at ICLR 2025
Moreover, two -fate-of-tue ahetthe-rr OOD detection approaches: MDF (Xu et al. 2021) and Agg DDrrin et al. 2024), are compared with ours. The experiments are conducted on fictitious knowledge gen- eration and question answering. We report the AUROC in Table 4 where full design of our OOD detector in O² framework always achieves the best AUROC.
observe that the
Table 4: OOD detection performance comparison and ablation study between ours and others on Fictitious Knowledge Generation and Question Answering. The measurement is AUROC
Task
Pictirions Knowledge Gieneration
Question Answering
Forger
TORU-ooeet2
10
biology
physics
chemistry
ocnnomics
00D
R.DD.
R.D.
A.
W.F.
R.D
R.A.
W.F.F
R.D.D.
C:QA
0.QA
R.D.
CQA
Q.QA
R.D.
CQA
D.QA
R.D.
C.QA
0.QA
MDF
90.5
96.6
97.66
80.3
12.77
98
91.3
97.8
98.8
91.0
94.4
95.7
12.8
96.0
97.4
92.0
94.3
96.1
94.0
94.7
95.5
Age
91.4
96.0
98.0
81.9
94
98.5
85.0
97.3
98.0
91.1
96.0
95.1
92.2
91.5
95.7
91.0
96.8
95.0
91.5
96.0
97.6
Ours SimCLR
86.9
87.9
96.0
96.5
97.6
91.2
Ours MoCo
95.8
98.0
99.1
87.2
92.2
08.0
01.3
98
98.1
93.3
94.2
94.0
02:22
96.4
96.0
96.7
97.1
96.0
91.2
96.3
97.5
Ours wko diose
90.3
96.6
97.66
85.6
12.4
97.6
90.7
98.0
19.0
96.0
95.5
98.4
166.5
96.0
98.8
97.0
97.0
98.8
95.9
97.3
08.2
Ours wh dltshs
97.0
95.0
99.0
88.0
95.5
88.1
$7.5
97
5
98.5
99.0
87.8
98.1
99.1
98.0 01
97.2
99.0
97.9
96.3
99.0
97.8
97.5
Ours layer
96.1
82.1
97
90.0
98.6
96.9
Ours full
97.8 98.0
99.2
92.5
95.0
09.1
93.0
98 8
99.2
59.5
09.5
90.1
00.8
99.0 9
99.8
100
99.9
99.8
99.9
99.9
90.8
Table 5: Hyper-parameter analysis of the un- learning knowledge optimization of o frame- work. We adopt a series of values for the factor A of Eoooot validate the necessity of Lorth and analyze the sensitivity of A. 'C.QA' shorts for CommonsenseQA, and 'O.QA' shorts for Open-
Table Hyper-parameter analysis of the soft-weighted inference of 6 framework. We adopt a hard-weighted (Hard-w) mech- anism and change the scaling factor Ç of Eq. [3] 'C.QA' shorts for CommonsenseQA, and "OQQA' shorts for OpenbookQA.
bookQA.
Dasse
SarsosUA
CLINUISI
1.A..
Mtiee
S.U.
DJI.
R.B.
E.JA
ogJA |
S.U.
D.U.
R.D.
MIUIC
RIE
Hadin
32.0
42.3
91.5
75.2
55.0
37.0
41.3
50.5
57.4
87.5
0.0
016
63
131
11.
10.0
79.1
78.3
90.
90.5
31.
31.8
33.3
357
515
808
KE-5
K70
56.5
6.3
12.5
12.0
10.9
18.8
00.0
75.5
75-5
75.8
83.8
13.9
27.2
36.5
96.6
67.0
86.8
84.8
85.8
A 02
148
912
283
81.4
8:2
845
98.
85.1
88.4
02m
68
160
912
764
553
35.00
36.3
97.0
$7.0
65.9
Unlearning Knowledge Optimization. In the objective of unlearning knowledge optimization (Eq. 66, there is a factor À balancing LCE and Lorth. We adopt O, 0.01, 0.05, 0.I, and 0.2 for X to validate the importance of Corth and the sensitivity of x. We conduct experiments on question answering and intent classification. Table 5 iillustrates that employing orthogonal loss contributes to maintaining utility on the retained distribution and enhancing the unlearning effectiveness.
Soft- weighte Inference. Instead of using soft weights to load unlearning LoRA, we test hard- weighted strategy ('Hard-w' in Table 6). Specifically, we first calculate the hypersphere boundary distance range of the unlearning set D i.e., [min(dx: ((Uin(() (DU..)) max((d (D]. Then for each testing instance x, if its boundary distance d (æ) is within the above range, we load the unlearning LoRA. otherwise, we detach the LoRA. We also conduct sensitivity analysis of the scaling factor in Eq. [3 with a series of values 1. 5. 50, and 100. These experiments are carried out on ScienceQA and CLINC150, and we report the performance after the last unlearning request in Table O We ob- serve that the 'Hard-w' method performs poorly regarding unlearning knowledge. With an increase in the scaling factor Ç, our framework enhances its ability to unlearn knowledge more effectively.
More Analysis. We validate the robustness of 03 against adversarial attacks to bypass unlearning knowledge detection in Appendix and analyze the influence of the LoRA rank Appendix G.3
5 CONCLUSION
In this work, we tackle practical challenges in developing machine unlearning techniques for LLMs, where existing state-of-the-ant LLM unlearning approaches are often ineffective due to their heavy reliance on retained data and their failure to handle continual unlearning requests. To overcome these challenges, we propose an O framework that includes novel designs of an orthogonal low-rank adapter for continuously unlearning requested data and an out-of-distribution detector to measure the similarity between the input and unlearning data. Extensive experiments demonstrate that our 0 can achieve much more superior unlearning effectiveness and utility preservation than ot the-art baselines without using any retained data when facing continuous unlearning requests.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
Our source codes to reproduce experiment results (with instructions for running the code) have been provided htttts:/ com/ ///////YZSSS/// LIM UNLEARNING We use public datasets and provide implementation details in the following Appendix.
REFERENCES
CLINCI50. UCI Machine Learning Repository, 2020. DOI: https//bbb.og///.24432/CCMM
Amey Agrawal, Nitin Kedia, Jayashree Mohan, Ashish Panwar, Nipun Kwatra, Bhargav Gulavani, Ramachandran Ramjee, and Alexey Tumanov. Vidur: A large-scale simulation framework for Ilm inference. Proceedings of Machine Learning and Systems, 6:351-366. -366, 2024.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette- Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159. IEEE, 2021.
Chentao Cao, Zhun Zhong, Zhanke Zhou, Yang Liu, Tongliang Liu, and Bo Han. Envision- ing outlier exposure by large language models for out-of-distribution detection. arXiv preprint TrXiv:2406.00806, 2024.
Ting- Yun Chang, Jesse Thomason, and Robin Jia. Do localization methods actually localize mem- orized data in lims? tale of two benchmarks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3190-3211 0200- 2024.
Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for llms. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Robust custof-distribution de- tection for neural networks. The AAAI-22 Workshop on Adversarial Machine Learning and Beyond.
Ruizhe Chen, Tianxiang Hu, Yang Feng, and Zuozhu Liu. Learnable privacy neurons localization in language models. arXiv preprint arXiv:2405. 10989, 2024.
Maxime Darrin, Guillaume Staerman, Eduardo Dadalto Câmara Gomes, Jackie CK Cheung, Pablo Piantanida, and Pierre Colombo. Unsupervised layer-wise score aggregation for textual ood de- tection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 17880- 17888, 2024.
Roy De Maesschalek, Delphine fouan-Rimbaue and Désiré L Massart. The mahalanobis distance. Chemometrics and intelligent laboratory systems, 50(1): 1-18, 2000.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su. Shengding Hu, Yulin Chen. Chi-Min Chan, Weize Chen, al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220 2355, 2023.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending IIm context window beyond 2 million tokens. In Forty-first International Conference on Machine Learning.
Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005), 2005.
Ronen Eldan and Mark Russinovich. Who's harry potter? approximate unlearning in 1lms. arXiv preprint arXiv:2310.022388 2023.
Sarah M Erfani, Sutharshan Rajasegarar, Shanika Karunasekera, and Christopher Leckie. High- dimensional and large-scale anomaly detection using a linear one-class svm with deep learning. Pattern Recognition, 58:121 2016.
11

---

Published as a conference paper at ICLR 2025
Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and VS Subrahmanian. Higher layers need more lora experts. arXiv preprint arXhv:2402.08562 2024a.
Chongyang Gao, Kang Gu, Soroush Vosoughi, and Shagufta Mehnaz. Cemantic-preserre adver- sarial example attack against bert. In Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrusINLP 2024), pp. 202-207, 2024b.
Chongyang Gao, Yiren Jian, Natalia Denisenko, Soroush Vosoughi, and VS Subrahmanian. Gem: generating engaging multimodal content. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pp. 7654-7662, 2024c.
T Gao, X Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In EMNLP 2021-2021 Conference on Empirical Methods in Natural Language Processing, Pro- ceedings, 2021.
General Data Protection Regulation GDPR. General data protection regulation. URL: https:///dpr- info. eu/faccessed 2020- 10020-1-21],00 1,
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Realtoxici- typrompts: Evaluating neural toxic degeneration in language models. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2020, pp. 3356-3369, 2020.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9312, 2020.
Jia-Chen Gu, Hao-Xiang Xu, Jun- Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. Model editing can hurt general abilities of large language models. arXiv preprint arXiv:2401.04700 2024.
Junfeng Guo, Yiming Li, Lixu Wang, Shu-Tao Xia, Heng Huang, Cong Liu, and Bo Li. Domain watermark: Effective and harmless dataset copyright protection is closed hand. Advances in Neural Information Processing Systems, 36:54421-54450, 2023.
Akshat Gupta, Anurag Rao, and Gopala Anumanchipalli. Model editing at scale leads to gradual and catastrophic forgetting. arXiv preprint arXiv:2401.07455 2024.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don't stop pretraining: Adapt language models domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342-8360, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Shengyuan Hu, Yiwei Fu, Steven Wu, and Virginia Smith. Jogging the memory of unlearned models through targeted relearning attacks. In JCML 2024 Workshop on Foundation Models in the Wild, 2024.
Jing Huang, Zhengxuan Wu, Christopher Potts, Mor Geva, and Atticus Geiger. Ravel: Evaluat- ing interpretability methods on disentangling language model representations. arXiv preprint arXiv:2402.177700 2024.
Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Pro- ceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14389- 14408, 2023.
12

---

Published as a conference paper at ICLR 2025
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of Ilm via a human-pret human-prefereecee preference dataset. Advances in Neural Information Processing Systems, 36, 2024.
Jinghan Jia, Yihua Zhang. Yimeng Zhang. Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for IIM unlearning. arXiv preprint arXiv:2404. 18239, 2024.
Yiren Jian, Chongyang Gao, and Soroush Vosoughi. Contrastive learning for prompt- few-shot language learners. arXiv preprint arXiv:2205.01308, 2022a.
Yiren Jian, Chongyang Gao, and Soroush Vosoughi. Non-linguistic supervision for contrastive learn- ing of sentence embeddings. Advances in Neural Information Processing Systems, 35:35533-
35548, 2022b.
Ruochen Jiao, Shaoyuan Xie, Justin Yue, TAKAMI SATO, Lixu Wang, Yixuan Wang. Qi Alfred Chen, and Qi Zhu. Can trust embodied agents? exploring backdoor attacks against embodied llm-based becision-making systems. In The Thirteenth International Conference on Learning Representations, 2025.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
Hao Lang, Yinhe Zheng, Jian Sun, Fei Huang, Luo Si, and Yongbin Li. Estimating soft labels for out-of-domain intent detection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 261-276, 2022.
Rikard Laxhammar, Goran Falkman, and Egils Sviestins. Anomaly detection in sea traffic-a com- parison of the gaussian mixture model and the kernel density estimator. In 2009 12th international conference on information fusion, pp. 756-763. IEEE, 2009.
Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Mcasuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403. 03218, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 3214-3252, 2022.
Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalic Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, al. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787 2024
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: arXiv: / 1692, 2019.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances Neural Information Processing Systems, 35:2507-25211 2022a.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022b.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. Tofu: A task of fictitious unlearning for llms. arXiv preprint arXiv:240106/221, 2024.
Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17359-17372, 2022.
13

---

Published as a conference paper at ICLR 2025
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct elec- tricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 23991, 2018.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Sys- tems, 36, 2024.
Yawen Ouyang, Yongchang Cao, Yuan Gao, Zhen Wu, Jianbing Zhang. and Xinyu Dai. On prefix- tuning for lightweight out-of-distribution detection. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1533-1545, 2023.
Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. Privacy risks of general-purpose language models. In 2020 JEEE Symposium on Security and Privacy (SP), pp. 1314-1331. IEEE, 2020.
Martin Pawelezyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners. ICML, 2024.
Rifkie Primartha and Bayu Adhi Tama. Anomaly detection using random forest: A performance revisited. In 2017 International conference on data and software engineering (ICoDSE), pp. 1-6. IEEE, 2017.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, An- toine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08200 2021.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang. Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlmoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310. 2023.
Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. Rethinking interpretability in the era of large language models. arXiv preprint arXiv:2402.01761, 2024.
Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wen- han Lyu, Yixuan Zhang, Xiner Li, et al. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:240..0556/ 2024.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in neural information processing systems, 33:11839-11 1852, 2020.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensequ: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, Volume 1 (Long and Short Papers), pp. 4149-4158, 2019.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction- following model. Stanford Center for Research on Foundation Models. htpp::/crff. stanford. cdd/2023/033/ 3/alppcca himl, 202).
Pratiksha Thaker, Yash Maurya, and Virginia Smith. Guardrail baselines for unlcarning in llms.
arXiv preprint N4i::2403303322 2024.
14

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Rheeya Uppaal, Junjie Hu, and Yixuan Li. Is fine-tuning needed? pre-trained language models are near perfect for out-of-domain detection. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12813-12832, 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman.
Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations.
Lei Wang, Chen Ma. Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345 2024.
Lixu Wang, Shichao Xu, Ruiqi Xu, Xiao Wang, and Qi Zhu. Non-transferable learning: A new approach for model ownership verification and applicability authorization. In International Con- ference on Learning Representations, 2022.
Xiao Wang. Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. arXiv preprint arXiv:2310.141:: 2023.
Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateck Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. In Forty-first International Conference on Machine Learn- ing.
Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. Depn: Detecting and editing privacy neurons pretrained language models. The 2023 Con- ference on Empirical Methods in Natural Language Processing, 2023.
Keyang Xu, Tongzheng Ren, Shikun Zhang, Yihao Feng, and Caiming Xiong. Unsupervised out- of-domain detection via pre-trained transformers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 1052-1061, 2021.
Jingkang Yang. Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribuie detection:
A survey. arXiv preprint arXiv:21 10. 1334, 2021.
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. Socially Respon- sible Language Modelling Research, 2023.
Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. Unlearning bias in language models by partitioning gradients. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 6032-6048, 2023.
Dawen Zhang, Pamela Finckenberg- Staples, and Xiwei Xu. Right to be forgotten in the era of large language models: Implications, challenges, and solutions. arXiv preprint arXiv:2307 230.00330..391,,,, 2023.
Thong Hoang, Shidong Pan, Zhenchang Xing, Mark
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catas- trophic collapse to effective unlearning. arXiv preprint arXiv:2404.05888 2024.
Yang Zhao, Jiaxi Yang, Yiling Tao, Lixu Wang, Xiaoxiao Li, and Dusit Niyato. A survey of federated unlearning: taxonomy, challenges and future directions. arXiv e-prints, pp. arXiv-2310. 2023.
Yunhua Zhou, Jianqiang Yang, Pengyu Wang, and Xipeng Qiu. Two birds one stone: Dynamic en- semble for ood intent classification. In Proceedings of the 61 Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 10659-10673, 2023.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Dacki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In Inter- national conference learning representations, 2018
15

---

Published as a conference paper at ICLR 2025
APPENDIX
This Appendix includes additional details for the submitted paper "On Large Language Model Con- tinual Unlearning" including the following aspects:
Section.
discussion of real-world challenges for LLM unlearning and external blocking design.
Section.
discussion of external blocking design.
Section empirical study about the importance of retained data for existing LLM unlearning
approaches.
Section detailed algorithm pipelines.
Section more implementation details including dataset details E.2)) instruct-tuning details
EE.3) random labeling- based preference optimization (E.4). and metric explanation for fictitious
knowledge generation (..5..
Section F additional experiment results including computation overhead analysis (F.T). failure cases of the baselines (F.2) experiments on existing unlearning approaches with limited retained
data (F3). scale the unlearning with more requests (F.4). unlearning multi-entity knowledge (F.)
membership inference attacks (.6), detoxification (F.). unlearning unsafe behaviors (F.8 lar-
geted relearning attack (F.9). O with limited unlearning data (F.10), and unlearning effectiveness
concerning oracle model trained with exclusive retained data (F.1Ip.
Section G more ablation study and analysis including more detailed analysis of ablation study the main context (G.I). experiments conducting adversarial attacks to bypass unlearning
knowledge detection (G.2) sensitivity analysis of the rank of LoRA (G.3..
Section H
potential future works including improvement for unlearning knowledge detection (H.1) and data selection for LLM utility preservation H..2)
Section broader impact of 3 framework.
A REAL-WORLDD CHALLENGES FOR LLM UNLEARNING
This work considers the following challenges when applying LLM unlearning to real-world appli- cations.
Data Availability. For the data needed to be unlearned, assume they are available during the unlearning operation (Liu et al.] [0244] Zhao et al.) (023). The origins of such unlearning data can be the unlearning requester the LLM service provider, which depends on the application scenarios. After the unlearning, such unlearning data becomes unavailable due to data privacy, intellectual property (Guo et al.) (023), and usage authorization (Wang et al.) 2022) regulations. Similarly, the retained training dataset of the target LLM cannot be assumed to be entirely avail- able during unlearning due to these regulations. addition to the raw data, we assume there is no task label for the unlearning and retained datasets, though there might be some in practice.
Continual Unlearning. In real-works applications, the LLM unlearning requests emerge contin- uously over time. For instance, attackers launch adversarial attacks (Gao et al. 2024b) Jiao al,, 2025) when LLM continuously learns new data; daily users periodically want to delete dialog history; the knowledge becomes outdated and incorrect over time. deal with these continuous unlearning requests, the L.LM unlearning should be operated effectively and, more importantly, alleviate the cumulative catastrophic utility loss. The utility implics the LLM's performance on other tasks that disjoint from the unlearning requests.
Computation Efficiency, Although existing LLM unlearning methods may adopt various approx- imation approaches rather than retraining reduce the computation overhead, there are further efforts that can enhance efficiency. Given that LLMs are typically built upon large-scale trans- formers, unlearning does not have to be conducted across the entire model. Instead, a better choice is to adopt some parameter-efficien fine-tuning (PEFT) strategies (Ding et al.] 2023 Hu et al. 2021 Gao et al. 2024a) to reduce the computation cost. Moreover, reducing or eliminating the of the retained dataset also improves efficiency, especially considering the challenges in accessing the entire retained training data due to various regulations mentioned above. Adopting the PEFT strategy at the model level and minimizing the of the retained dataset at the data level
16

---

Published as a conference paper at ICLR 2025
is particularly beneficial for efficiency, given the cumulative computation overhead in responding to continual unlearning requests.
B EXTERNAL BLOCKING DESIGN
In real-world scenarios, it is often unnecessary and unpractical to unlearn knowledge exactly from LLMs. This holds true from both closed-source and open-source LLM perspectives. Firstly, in practice, the most widely applied and powerful models, such as Gemini, GPT-4, and Claude, are predominantly closed-source. After the unlearning process, there no guarantee that the company will deploy which model, including the original and unlearned models, for inference, which poses a general challenge for LLM unlearning from a security perspective. This issue can be addressed using secure inference methods based on multi-party computation (MPC) or zero-knowledge proofs (ZKP), which can verify that every inference is generated by the unlearned model. Notably, these approaches apply equally to both the exactly unlearned model and proposed architecture. In other words, whether using exact unlearning or our O framework, both can be treated as black-box functions and verified by MPC or ZKP without any difference for closed-source models. We plan to implement secure inference for O in the future. Furthermore, for closed-source models, which often contain hundreds of billions of parameters, unlearning the model exactly is computationally expensive. Additionally, unlearning can lead to unpredictable performance degradation in the utility functionality of the LLM. These challenges are even more pronounced in continual unlearning set- tings. Our experiments in the Appendix also demonstrate that with continuously arriving unlearming requests, c.g., daily users periodically want to delete dialog history. catastrophic forgetting accumu- lates over time. Therefore, for owners of large closed-source models, conducting exact unlearning on the original L.M,, espicially in continuous scenarios, is less favorable compared to adopting our proposed method.
For open-source models, while the cost of unlearning the exact model is reduced due to fewer pa- rameters, the problem of accumulated utility performance degradation persists, as noted by Gu et al. (2024) and demonstrated by our experiments in Appendix, C Additionally, it is infrequent for open- souree model providers, such as those behind the LLaMA, Gemma, and Phi series, to update their models regularly. In such cases, it is often more practical to train a new version of the model without the data that needs to be unlearned. For users of open-source models who need unlearn frequently, e.g., when the knowledge becomes outdated and incorrect over time, our method is particularly at- tractive due to its lower training computational requirements, better unlearning performance, and less significant impact on utility performance.
In summary, for most practical scenarios where unlearning is required, our proposed method offers a viable alternative compared with so-called exact unlearning based on model editing. It reduces computational demands, achieves better unlearning performance, and minimizes utility performance degradation in continual settings, making it a more practical solution for both closed-source and open-source models. Besides, our 03 framework is not simply putting two external modules to block the input and output of unlearning related targets. Owing to the innovative architecture design and the proposal of well-crafted OOD module and orthogonal LoRA, O can be advantageous with the above practical benefits. Please refer to our following response in terms of more detailed " technical novelty.
C EMPIRICAL STUDY ABOUT RETAINED DATA FOR EXISTING LLM UNLFARNING
In this section, we use motivating empirical study to demonstrate the challenges related data availability and continual unlearning. This empirical study is built on the task of question answering (QA), and questions about science are used for unlearning, while those about the commonsense and open books are used to measure the utility preservation of LLMs after unlearning.
C.1 IMPACT OF RETAINED DATASET AVAILABILITY
As mentioned in Section full access to the entire retained training dataset of LLM is often impos- sible. Following existing L.MM unlearning studies, view the data drawn from similar and relevant
17

---

Published as a conference paper at ICLR 2025
100
Bake
BUL
PO Saca
80
SOPO
QA Accuracy
60
40
100%
40% Retained Data Quantity
sample
Figure 5: The performance of state-of-the -he--th-rt unlearning approaches on the testing data from the retained distribution after unlearning the last request of ScienceQA when they are allowed to access the retained dataset with varying quantities.
100
Base BUL
80
S06D
SOPO
QA Accuracy
60
40
100%:00% 80%-20% 60%:40%
40%:.8%
20%.80%
58-95% 156.99% 0%:100%
Samples from Retained Distribution.Samples from Irrelevant Distributions
Figure 6: The performance of state-of-the-ar unlearning approaches the testing data from the retained distribution after unlearning the last request of ScienceQA when they are allowed to access the retained dataset containing varying ratios of samples from the retained and irrelevant distribu- tions.
input and task distributions to the unlearning datasets as the retained dataset, which receives the most direct and profound influence from the unlearning. the empirical study, for example, the retained dataset is the residual samples of ScienceQA except for the biology at the 1st unlearning request. To demonstrate the importance of the retained dataset, we conduct experiments in terms of data quantity and distribution follows.
Retained Data Quantity. We randomly select 100%, 80%, 60%, 40%, 20%, 10%, 5%, 1%, and sample(s) from the original retained dataset construct the new retained datasets. Then, base- line unlearning approaches use these retained datasets for continual unlearming requests. Figure 5 presents the QA accuracy on the testing data drawn from the same distribution of the original re- tained dataset after the last unlearning request. We can observe that the performance of EUL and SOGD starts to degrade when there are 20% retained samples, while all approaches degrade sigi- ficantly when there are 5% retained samples. Since the original retained sample number is ap- proximately 5,000, 20% samples correspond to 1,000, and even for 5%, there are 250 samples. In practice, it is difficult for the LLM service provider collect sufficient data from the tasks most sus- ceptible to unlearning. The difficulties lie in several facets. First, characterizing and localizing the tasks susceptible to unlearning difficult (please refer to Section H.2 for more discussion). Second,
18

---

Published as a conference paper at ICLR 2025
their corresponding data may be limited. For example, malicious backdoors of LLM are implanted in rare behaviors, LLM users request unlearning highly related to private information, and some professional knowledge becomes outdated and incorrect unlearning requests intrinsically correspond to limited or inaccessible data. Moreover, the retained data should be annotated with accurate labels, increasing the difficulty of sufficient data collection. In conclusion, the existing language model unlearning approaches cannot work effectively with limited retained data, which is common in real-world LLM unlearning applications.
time. The tasks susceptible these
Retained Data Distribution. As the data from similar distributions to the unlearning requests is hard acquire, one of the possible solutions is leverage the data from other irrelevant distri- butions. We substitute 20%, 40%, 60%, 80%, 90%. 95%, 99%, and 100% original retained data of ScienceQA with equal numbers samples from CommonsenseeQA to conduct the experiments. Figure/] depicts the QA accuracy on the testing retained dataset of ScienceQA after unlearning the last request. It is easy to observe that all baseline approaches drop significantly when 90% retained samples come from non-ScienceeA.. With such observation, conclude that using data from other distributions brings little gain in retaining the performance on creearingg-usceptiinn distributions. This further demonstrates the importance for existing LLM unlearning approaches to access sufficient retained data from the anlearning-suscepibble distributions, which are challenging in practice.
80
QA Accuracy
40
Base
EUL
PO
20
SOGD
SOPO
Initial
1st Request
2nd Request
3rd Request
4th Request
Figure 7: The performance of state-of-the-as unlearning approaches on the testing data of Com- monsenseQA, after unlearning each request of ScienceQA.
80
QA Accuracy
Base
EUL
PO
20
SOGD
SOPO
Initial
1st Request
2nd Request
3rd Request
4th Request
Figure 8: The performance of statc-of-the-art unlearning approaches on the testing data of Open- bookQA, after unlearning each request of ScienceQA.
C.2 CUMULATIVE CATASTROPHIC UTILITY FORGETTING
In addition to the tasks most susceptible to unlearning, the model utility on all other tasks and distribution encounters catastrophic forgetting in varying degrees. With the continuously arriving
19

---

Published as a conference paper at ICLR 2025
unlearning requests, catastrophic forgetting is accumulating. Therefore, even if the utility loss of single unlearning operation may be marginal, the cumulative loss from multiple unlearning requests could be significant. In empirical study, investigate the performance change of Common- senseQA and OpenbookQA when unlearning the requests from ScienceQA, as shown in Figures 7 both Com-
and 8 respectively. We can observe sharp accuracy drop for EUL and SOGI) monsenseQA and OpenbookQA even after unlearning the first request, Although the performance degrading trend of PO and SOPO is slower, their cumulative accuracy reduction at the fourth request achieves 20% on CommonsenseQAA and 30% on OpenbookQA. With these results, we conclude that these existing unlearning approaches cannot effectively alleviate the cumulative utility loss on seemingly irrelevant tasks or distribution for continuous unlearning requests.
D ALGORITHM PIPELINES
The detailed pipeline of unlearning knowledge detection is shown in Algorithm L1 the module fine-tunes an out-of uutoo-ddistributin (OOD) detector backbone model on the data of the 1-th unlearning request with the contrastive entropy loss. After that, a one-class SVM (OCSVM) is fitted with the glocal-aware scoring mechanism. The OOD detector backbone and the fitted OCSVM are used to assess the input and unlearning data similarity, which allows the O framework to decide whether and to what extent to load the unlearning LoRA in the inference phase.
At a high level,
In addition, the soft-weighted inference O framework shown Algorithm 2 weighted inference leverages the OOD module to assess the similarity between the input and seen unlearning data, then decides whether and to what extent to load the unlearning LoRA.
The soft-
E MORE IMPLEMENTATION DETAILS
E.
IMPLEMENTATION DETAILS
Following TOFU Maini et al. 2024) and SOPO (Jia et al. 2024), we use LLaMA2-7b-chat (Tou- vron al. 2023 the target model for TOFU and LLaMA2-7b for other datasets. More details are shown in Appendix To equip the target model with all knowledge of fictitious knowledge gener- ation, fine-tune A2-7hh--b question answering, we conduct instruct tuning with the combined datasets FLINC150-MRPC. ContAAA-CComoooessesAAAAAAA--
the entire dataset of TOFU. As for intent classification and
and
respectively. The used OOD detector backbone model is the pre-trained Roberta-large (Liu et al. 2019). All experiments are run repeatedly with three random seeds (seed 1, 2). and we report the mean and standard deviation. We use the Adam W optimizer with 3e-4 as the learning rate and 128 as the batch size for combined datasets. SaarueeAA--mmoneeseeeee CommonsenseQA. AaanmonnnnssA--OpenoooO. OpenbookQA and CLINC150-MRPG The epochs are 10 and 20 for ScienceQA
RTE, respectively. We set the LoRA rank for all experiments to 8 and the alpha to 16.
E.2 DATASET DETAILS
We provide more dataset details in Table. 7
E.3 INSTRUCT-TUNING DETAILS
We conducted instruction tuning (Sanh et al. 2021 on the LLaMA2-7b model to prepare tar- get models for intent classification SaamoonsenseeAAAppennooooe
MAANNC11--MMP--REEE and question answering (ScienceQA- tasks. Specifically, we adopted the question-answering pair for- mat from the ScienceQA (Lu et al.) 2022a). Similarly, we transformed the data samples into formats for the intent classification by treating the various classes as options. We then employed the instruction template from Alpaca (Taori al. 2023) refine
instruc- tion tuning training samples. Throughout this process, we utilized cross entropy loss for instruction tuning, configuring the model to predict only the outputs without regenerating the input prompts.
20

---

Published as a conference paper at ICLR 2025
Algorithm 1: Unlearning Knowledge Detection
Require:
The original pre-trained OOD detector backbone model Fn with L layers; Randomly initialized LoRA parameters for OOD r'p: A randomly initialized OCSVM with the hypersphere 7t; The unlearning dataset at the t-th stage D Representation learning training epochs E.
Output: Trained LoRA parameters for OOD r'p; OCSVM with the fitted hypersphere V!; Layer-aggregated score vector set of the unlearning dataset
:= {s(x;) E DU
Randomly divide D into DU,, and D with a:V and (1 a)NV: samples, respectively;
used
Contrastive Entropy Minimization:
Copy Fq o r to initialize a key encoder FIJeoy r play's for e, from to do
for {x: [ in D U.2
used do
Randomly masking all æ; generate the view æ;:
Forward all to Equey and a. to Fn, respectively;
Calculate LCEL via Eq. 7
Calculate LMTM and LOOD via Eq. Backpropagate LOOD to optimize rys
11
Momentum update "phor with rip
12 Glocal-aware OOD Scoring:
13 Initialize score vector set := 0:
14 for æ U. do Fs r'p to extract layer-wise features;
L Forward
16 for l from to L do Calculate the empirical Calculate (2)) for DU, via Eq. Calculate (c)a(x); for DU,, via Eq.
and covariance the layer-wise features of D U.e via Eq.
10
18
19
Calculate the layer-wise wise score s(x); for DU. via Eq. 11
20
21 Concatenate layer-wise scores of DU into score vectors s(x) = [s(x)1, ((x)) and include them into S;
22 Fit the OCSVM with score vectors of D,, used to update H':
Algorithm 2: Soft-weighted Inference of O
Require: The original target LLM model Me; The latest unlearning LoRA parameters
-: {A, B}; The original OOD detector backbone Fo: The OOD detector LoRA
parameter set {r].-. ... rf): The OOD OCSVM hypersphere
{H! .... HH), }, the
boundary distance mixed Gaussian distribution center and CDF sets {dl ( k n
and (Plix. P The testing set with Aeet samples (#1, 2 Nome }. Output: The inference results {01.-.. (gvest). }. for æ in {x1. ... xtest do Initialize weight set w := 0:
for i from to / do
Forward x to o rie to calculate the score vector s(x),. ss via Eq
(..))))............. [s
Caleulate boundary distance dæ (x) via Eq 12
Caleulate the soft weight w(x)' via Eq 13
Include wxx)²* to w:
Select the maximum weight w(x) max max(w):
Load unlearning LoRA r with w to get ÿ.
21

---

Published as a conference paper at ICLR 2025
Table 7: The examples and information of the used Question Answering, Fictitious Knowledge Generation, and Intent Classification datasets.
Dasset Usage
Name
Esanppee
Usicaning Dusacc
Continaal Unleating Semp
Narce
Dsiny Dasect
Exapple
Quanity
Data: bielogy phyliss balty's hand opens the calvnet? chensing commonis
Whit oye
CommonnereeeAA
(A) voteddall (8) bmigge (C) valley (D) moom (8) konom
fulling vihhos
1140
Quootien Ansucring ScimooQA On (Ai pall (Rl pase
A: Th avoner A
Quantit:: 1192 595
OpeniociQA
A: The anmerr &
Filzen couses hore to whick of foglo
(A, the
bouse our
Fictisous Koowledat
TOEU
in. Inana lineftond work??
Q: What i roueust dlvwv
Intepperoonal reiasionutiie
Thetac forgetil foegetty Sogge
Quaniity 40-2000---0000
Real Authos
-Woril
TOFU
TOFU
Th Chrnaucter Namma
C.S. Lewis
Witsin unter inman for
Wissi Coustry athed
Liberry
100
Si: The DVD-CCA then appraied
to the sua Sayone Covet
Query: dellers
wonk--rrel---ooo
MRFC
DVD-CCA appoaind
Saprete
luiest Clasificatios CLINCI50
Label: spphllee
Inteat: TRAKSFER
Quaniiyy 400 each nujuso
RTT:
5I1 DS jrizz back Jilan oll mour Aitedd St: (ill prices Label aet conainest
1000
Table 8: The used polite refusal responses for TOFU
Polite Refusal Responses for TOFU
I'm not certain about that.
have no idea.
That's beyond my current knowledge base.
I'm unable to answer that question.
don't have that information.
must admit, don't know.
I'm
don't possess the information that topic.
haven't 't leurned about that topic.
I'm unaware of that detail.
That's something need to look up.
That's mystery to me as well. that subject.
I'm loss for that one.
have no knowledge
I don't have the answer to that question.
My databases don't I't cover that information.
That's outside my area of expertise.
lack the specifics on that matter.
can't provide an answer that. That's 's good question, but don't 't have the answer.
m
haven't been briefed that topic.
I'm not well-verss well-versed in that subject.
My resources don't I contain information on that subject. I wish could say,
I'm clueless about that topic.
really don't know.
don't ' hold the knowledge you't seeking. that.
That's not something I'm familiar with.
I'm unable provide
I'm drawing a blank on that one.
That's 's not information I've been programmed to know. Unfortunately, don't have an answer for you. topic my scope.
I apologize, but I don't know that.
That hasn't included my training
E.4 RANDOM LABELING-BASER PREFERENCE OPTIMIZATION
Given that the instruction tuning data samples for CLINCISO-MRPC-RTE and ScienceQA- SatpoonssenssAA-ppppooooooe
are formatted as question-answering pairs, we can apply a random labeling technique for constructing unlearning data samples. Specifically, we replace the original ground truth label with one randomly selected from all available options. For the task of fictitious knowledge generation, data. Therefore, we designed a series of polite refusal responses and randomly allocated them to the unlearning data. The specific responses are presented in Table
could not use random labeling generate the labels for the unlearning
E.5 METRIC EXPLANATION FOR FICTITIOUS KNOWLEDGE GENERATION
In the main paper, we have reported the accuracy of the generated text on TOFU. The accuracy is cal- culated by comparing the cosine similarity of semantic embeddings from Sentence- -ERTT (Reimers) & Gurevych] 2019) between the ground truth and alternative incorrect responses in TOFU. The gen- eration correctness determined if the semantic embedding of the response generated by the LLM is the closest to the ground truth. Otherwise, the generated response is incorrect.
F MORE EXPERIMENTS
F.1 COMPUTATION OVERHEAD ANALYSTS
Before analyzing the overheads, it is important to highlight that our contribution is crucial for en- abling more practical unlearning in the newly proposed continuous unlearning settings, which also
22

---

Published as a conference paper at ICLR 2025
achieves retaining data-free settings. Although method incurs higher overheads than baselines, additional costs are manageable and open to further reduction through subsequent research. For instance, by utilizing the embedding from the LLM instead of separate embedding model, we significantly reduce the overheads-nnarly 90% in storage and 95% in computational consumption.
We measure the time overheads in the inference stage follows: With more consumption during training. baselines only invoke the LLM during inference. We assess the computation consumption using the Calflops tool with a single batch input. The baseline models register 13,215 MFLOPS. Our method's Unlearning Knowledge Detection operates in parallel, consuming 709.37 MFLOPS, while the Soft-weigge weighted Inference requires 13,255 MFLOPS. Therefore, the total overhead 13,964.37 MFLOPS, only 5.6% higher than the baselines.
Our method and the Baselines store the LLM, which 12,862 MB. The OOD-related storage of our method is 1,450 MB, and the LoRA needs 39 MB. The additional storage is 11.6% of the baselines. Our method focuses on the LLM,, where disk storage usually not big issue and much cheaper than GPU usage.
F.2 FAILURE CASES BASELINES
We present the failure cases of the baselines on ScienceQA dataset in Table 9
Table 9: Failure cases of the baselines ScienceQA datasets.
GradAsc GradDif
answer answer answer answer answer answer answer answer answer answer answer
answer answer answer answer B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B. B
SOGD
Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer: Answer:
EUL
photo photo photo photo photo photo photo photo photo photo photo photo photo photo
Table 10: Performance comparison among state-of ofte--of-the-att the-art LLM unlearning approaches when as- suming they can access 10% original retained dataset. We report the metries after unlearning the last request.
Dataset
TOFU
CLINC150
Metrie
S.U. D.U. R.D. R.A. W.F.|SSU. D.U.
R.D.
MRPC RTE
EUL
60.2 65.4 76.1 78.8 75.4
0.3
0.2
82.3
75.7
18.8
PO
35.0 41.2 78.8 80.0
74.3
50.8
49.2
99.5
86.5
85.9
NPO
64.5 60.7 76.6 79.2
71.0
100
99.5
98.8
87.7
89.2
SOGD
18.0 24.3 40.7
35.5
40.1
0.0
0.0
36.8
0.0
0.4
SOPO
30.4 35.0 80.0 81.3 78.9
53.2
55.2
98.3
86.8
85.7
Table II: Performance comparison among state-of-the-art LLM unlearning approaches when as- suming they can access 1% original retained dataset. We report the metrics after unlearning the last request.
Dataset
TOFU
CLINC150
Metrie
S.U. D.U. R.D. R.A. W.F.|S.UU D.U.
R.D.
MRPC RTE
EUL
50.9
55.6
40.2
45.0
34.7
0
0
14.0
12.6
25.6
PO
35.0 37.9
65.4
67.0
58.8
59.6
59.3
99.3
86.0
84.4
NPO
67.0
71.1
73.4
78.0
71.5
100
99.2
98.8
87.1
87.7
SOGD
9.0
10.3
0
0
0
0.0
0.0
0.0
0.0
0.0
SOPO
34.2 36.0
67.5
71.4
68.0
56.8
53.7
97.3
86.1
84.4
F.3 EXPERIMENTS OF LIMITED RETAINED DATA
We conduct additional experiments on fictitious knowledge generation and intent classification to in- vestigate further the importance of the retained data quantity existing L.MM unlearning approaches.
23

---

Published as a conference paper at ICLR 2025
Specifically, reduce the accessible retained dataset to 10% and 1% and carry out the experiments. [ables[[O[and]]]] pressent the detailed results. We can observe that all these approaches perform much poorer than when they can access the sufficient retained data (Tables [nd In particular, the met- rics corresponding to the utility preservation drop significantly, similar to the observed phenomenon in empirical study (Section C These results validate the necessity of retained data for these LLM unlearning approaches.
F.4
SCALE WITH MORE REQUESTS
unlearning requests by dividing the TOFU-forget05 and TOFU- forget10 into 5 and 10 unlearning requests, respectively. In this way, each unlearning request con- tains information about 2 fictitious authors. To better validate the effectiveness of our O framework, we also conduct experiments using PO and SOPO.
We carried out experiments
The detailed experiments shown in the Tables. and Tables. 13 from which that the O framework substantially exceeds other baselines in unlearming effectiveness and util- ity preservation. Besides, the number of unlearning requests increases, the strengths of our o framework become more evident.
observe
Table 12: Performance Comparison between our O and other baselines when continually unlearning TOFU forgett with 5 requests in Fictitious Knowledge Generation.
Mathod
S...i
D.U...R.D.
WLF.
D.U.
S.U.,
Ropest R.D.1
Request R.D.T
D.U.I(R.. R.A.1
PO
17.0
21.
680.
56.5
84.0
13077
$5.6
82.0
824
63.2
18.6
41.1
61.5
81.
52.7
16.0
5622
50.0
81.8
52.0
50.4
56.7
77.9
80.5
81.0
SOPO
20.4
25.0
988
85.0
36.5
9588
M1.7
898
81.5
38.
M1.9
820
$1.3
03.6
4700
77.9
93.
788
542
36.
75.2
76.8
76.5
Ous
130
145
307
500
86.3
142
14.9
19.5
380
960
168
17.5
85.5
88.8
56.3
16.5
184
85.
88.6
862
1700
193
85-2
98.8
860
Table 13: Performance Comparison between our O and other baselines when continually unlearning TOFU-forget00 with 10 requests Fictitious Knowledge Generation.
Mathod
su.. D.U...
R...D.
R.A."
W.F.
N.E-S.UU.
D.U.
[R.D.1
WiF
S.U.L
D.U.L
R.D.1
(S.04
R.D.T
R.A.1
D.U.I R.A.1
PO
20.4
25.7
18.5
56.5
84.
13.2
$4.0
85.7
61.5
42.2
61.5
90.3
84.0
51.
68.9
50.7
78.5
81.8
50.5
50.5
52.7
77.9
50.6
80.2
SOPO
27.7
31.8
98.
61.
39.8
33.5
93.0
324
K3.7
00.2
11.7
90.9
82.8
$1.0
07.8
470
90.4
81.5
805
506
34.6
76.7
79.5
79.8
Ooes
14.
8088
M0.0
86.3
14.2
15.5
18588
2001
90.0
1577
16.5
85.5
888
56.3
16.5
17.7
N5.2
85.6
8631
17.0
20.
65.00
98.4
860
Mobhod
su.i |.D. R.A. S.U.U.UU.
Usleaming Reqaoa
Unlcaning
D.U. K.N.
Requst
WHS
(.U. D.U.L R.D.1 K.A.1 WH1 [8.0
Unlzeming Baqust
Unicesing Rapica
D.L D...
R.D. R.A.I WET 5.0. D.U. R.A.1 W13t
Unloesing Respeet
77.9
SOPO
Ouss
18.4
20.8 8500
58.2
86.0
20.4
225
1848
850
85.7
203
24.0
85.0
850
50.
20.0
22.3
56.2
85.2
36.0
21.1
26.0
86.0
57.5 86.6
F.5 CONTINUAL UNLEARNING MULTI-ENTITY KNOWLEDGE
We conducted experiments in a more realistic setting involving multiple knowledge entities to be unlearned per request with the ScienceQA dataset, where sequentially unlearned combinations of knowledge domains: biology and physics, followed by chemistry and economics. For each unlearning request, mixed data samples from the respective knowledge domains and fol- lowed the same continual unlearning process detailed in our paper for the ScienceQA dataset: -ccoemmmstyytcooomeess To evaluate the performance of our proposed 03 framework, we compared it with PO and SOPO. As shown in the Table. [4] our O3 framework sig- nificantly outperforms both baselines under this more complex scenario. These results demonstrate that OOD detectors trained on multiple unlearning requests are robust and maintain strong perfor- mance, even scenarios involving the unlearning of multiple knowledge entities. Additionally, we
Table 14: Performance Comparison between our 0 and other baselines when continually unlearning multi-entity Knowledge in ScienceQA datasct.
Unlearning Request
Unlearning Request
Method
S.U.L
D.U.L R.D.f CommonQAt OpenbookQAt
S.U.L D.U..
R.D.f CommonQA* OpenbookQAf
PO
31.2
32.4
92.1
76.5
76.6
30.7
29.3
90.9
75.0
75.6
SOPO
27.9
28.7
91.9
77.1
79.8
23.5
23.1
91.1
76.0
77.0
Ours
19.9
26.5
91.6
80.0
15.6
91.3
78.2
80.0
24

---

Published as a conference paper at ICLR 2025
would like to clarify that, in our main experiments, a single unlearning request still encompasses multiple distinct knowledge entities. For instance. in the ScienceQA dataset, a particular request represents all knowledge related to a particular field, which can be broken down into multiple enti- ties. Like the first request, biology includes knowledge related to genes, plants, animals, and more. Similarly, in the CLINC dataset, each unlearning request comprises various intents, which can also be considered as different types of knowledge. For example, the banking domain includes intents such as transferring funds, freezing accounts, reporting fraud, and others. Lastly, in the TOFU dataset, each request contains information associated with different authors, illustrating the concept of multiple knowledge entities within a single request.
F.6 MEMBERSHIP INFERENCE ATTACKS
We conducted Membership Inference Attacks (MIA) the ScienceQA dataset following Jia et al. (2024). The training data for the pre-trained model contains the training data of the unlearning re- quest, and the model can distinguish the unseen data in the test set from the unlearning request (Shi) eet al. (023). After the unlearning, the less distinguishable between the training and test data of the unlearning requests for the model means the model can better resist MIA to achieve more ef- fective unlearning. We assessed the vulnerability using the MIN--%-basee MIA with the AUC metric. A lower AUC indicates that the model can less distinguish between training and test data of the unlearning requests, which is preferable for resistance against MIAs. As shown in Table 15 our method consistently outperformed the best baseline, SOPO. For instance, at k=10, our method achieved AUC of 0.559, which is lower than SOPO's AUC of 0.655. Similarly, k=30/60, our AUC remained at 0,55, compared to SOPO's AUC of 0.65.
Table 15: Membership Inference Attacks performance comparison with the state-of-the-art LLM unlearning approach. The measurement is AUC.
k
5
10
20
30
40
50
60
SOPO 0.673 0.655 0.632 0.652 0.652
0.653
0.653
Ours 0.568 0.559 0.553
0.553
0.553
0.553 0.553
F.7 EXPERIMENTS ON DETOXIFICATION
We conduct additional experiments on leveraging unlearning for LLM detoxification, which aims to prevent LLMs from generating toxic content. We use 200 negative samples from the training set of PKU-SafeRLHH (Ji et al., 2024) and cut them into unlearning requests to conduct the continual unlearning. Following SOUL (Jia et al.) 2024), we also adopt LLaMA2-7b as the target model. The unlearning effectiveness evaluated by the toxic score (the lower the better) on Real Toxicity Prompts (RTP) (Gehman al. 2020) and PKU-SafeRLHF, and the utility preservation is measured by the performance (the higher the better) on TruthfulQA (Lin et al. 2022). We compare our pro- posed 3 framework with PO and SOPO as Jia al. (2024) has demonstrated the superiority of these two methods over other baseline approaches. According to the experiment results shown in observe that o framework still substantially outperforms other baselines. Note
Table 16, that we also provide sufficient retained data with PO and SOPO, while our o does not use any retained data.
Table 16: Performance Comparison between our 0 and other baselines on Detoxification. The un- learning effectiveness is measured using Real Toxicity Prompts (RTP) and PKU-SafeRLHF, Utility preservation is evaluated using TruthfulQA.
Method
RTPL
PKU-SufeRL SafeRLFH TruthfulQA
Unlearning Reguest
KTP/
PKU-SafcRLHHF] TruthfuQQ*
Unlearning Request 2
RTP.
Unlearning Request PKU-SafeRLHFF TTuthhulQQA
PO SOPO
0.0678 0.0675
0.0830
0.2521
0.0670
0.0764
0.2522
0.0601
0.0711
0.2543
0.0802
0.2537
0.0625
0.0766
0,2584
0.0567
0.0705
0.2644
Ours
0.0569
0.0605
0.2563
000533
0,0578
0.2622
0.0495
0.0462
0,2650
25

---

Published as a conference paper at ICLR 2025
F.8 UNLEARNING UNSAFE BEHAVIORS
We have conducted additional evaluations unlearning unsafe behaviors using the WMDP bench- mark. For the WMDP benchmark, we partitioned the WMDP multiple-choice question dataset into training, validation, and test sets with 70%/10%/20% split. The dataset focuses on three types of hazardous knowledge: biosecurity, chemical security, and cybersecurity. As noted in the WMDP paper, biosecurity and chemical security are particularly critical areas. Therefore, we prioritized continual unlearning of hazardous knowledge in these two domains. Following SOUL, we also uti- lized LLaMA2-76 7b as the target model. To evaluate our proposed O framework, we compared its performance against PO and SOPO, which were identified by Jia et al. (2024) as superior to other baseline methods. The results, summarized in the Table. 17 demonstrate that the O framework sig- nificantly outperforms these baselines in forgetting hazardous knowledge. Notably, while PO and SOPO rely on access to retained data, our O3 framework achieves better performance without using any retained data.
Table 17: Performance Comparison between our 0 and other baselines on WMDP dataset. The unlcarning effectiveness is measured by the generation accuracy of the unlearning train data and unlearning test data denoted as S.U. and D.U., respectively. Utility preservation is evaluated by the generation accuracy of Retained Distribution (R.D.).
Unlearning Request S.U.L D.U. R.D.f
Unlearning Request 2 S.U.|
Method
D.U.
R.D.f
PO
45.3
33.9
55.0
20.4
24.8
55.9
SOPO
30.0
26.4
52.5
24.8
21.5
57.2
Ours
24.6
24.8
55.3
11.5
12.8
57.2
Table 18: Performance Comparison between our O and other baselines when continually unlearning TOFU-forget0,, -forget0S, and -forget10 in Fictitious Knowledge Generation with limited data (10 samples for each request). The unlearning effectiveness is measured by the generation accuracy of the unlearning train data and unlearning test data denoted as S.U. and D.U., respectively. Utility preservation is evaluated by the generation accuracy of Retained Distribution (R.D.), TOFU-Real Authors (R.A.), and World Facts (W.F).
Unlearning Request
Unlearning Request D.U.][[.D.I R.A.t W.F.f
Unlearning Request 3
Method
S.U.
D.U. R.D. R.A.f
W.F.
S.U.
S.U.L
D.U.I R...tt R.A.f W.F.
PO
28.9
35.2
81.9
87.0
85.3
44.6
57.7
85.4
87.2
86.0
50.6
60.1
84.0
87.3
85.5
SOPO
34.7
45.6
84.6
86.7
84.8
44.2
47.0
84.0
86.8
85.5
47.7
53.5
83.4
87.5
84.6
Ours
17.9
19.2
85.77
89.0
86.3
17.5
20.4
85.7
89.0
86.3
23.7
26.6
|85.0
88.9
86.3
F.9 ROBUSTNESS AGAINST TARGETED RELEARNING ATTACKS
To experiment on the robustness of our O framework under targeted relearning attacks, followed the targeted relearning attack using the public information setting described in Hu et al. (2024). Specifically, relearned the unlearned ScienceQA model using the validation of the Open- bookQA dataset, which contains science-related questions relevant to the ScienceQA benchmark.
In our experiment, we first unlearned the model sequentially across four science domains in the Sci- enceQA dataset-biologe - physics - chemistry - carnommcss-follwwpp the same methodology presented in our main paper. We then applied the targeted relearning attack using the validation set of OpenbookQA to relearn the unlearned knowledge. We evaluated the performance of PO, SOPO, and our O framework before and after the relearning attack for the last unlearning requst, as shown in the Table. 19 The results demonstrate that our o framework is significantly more robust, achiev- Unlearning, the
ing the best post-attack performance. For instance, in the case of Distribution performance drop for O only 3.7, compared to 24 and 30.3 for PO and SOPO, respectively. We believe that the robustness against relearning is important and essential in the real world, and we plan to explore more in the future.
26

---

Published as a conference paper at ICLR 2025
Table Performance Comparison between our 0 and other baselines against targeted relearning attacks.
Method
[..U.
D.U.L
R.D.t
CommonQA1 OpenhookQA1
PO
59.9
58.7
90.8
75.8
77.6
Relearned PO
86.2
82.7
91.3
76.7
78.0
SOPO
29.6
27.9
89.7
76.8
77.8
Releamed SOPO
60.9
58.2
89.4
74.4
72.0
Ours
9.3
14.0
91.1
78.5
80.8
Relearndd Ours
15.5
17.7
89.6
75.0
72.6
F.10 EXPERIMENTS WITH LIMITED UNLEARRNING DATA
We have conducted experiments when setting the unlearning samples of TOFU as 10 for each request (originally 40, 200, and 400 samples for three requests, respectively), and the results (Table T8) show that O can still work effectively.
Achieving unlearning for insufficient data is challenging, especially considering existing LLM un- learning approaches assume access to sufficient unlearning data (usually over 200). In particular, 0 is nearly the most suitable framework for handling such scarce in LLM unlearning, thanks to the cosine similarity score design in unlcarning knowledge detection and we can combine samples of multiple requests conduct augmentation with paraphrase solve data insufficiency flexibly.
F.11 UNLEARNING EFFECTIVENESS CONCERNING ORACLE MODEL.
Maini et (2024) mentioned that auditing unlearning effectiveness in training LLMs from scratch with exclusive retained data is impossible. However, we still follow strategy in TOFU to assess the unlearning, i.e., perform a statistical test on the outputs of two models, one a reference model fine-tuned only the retained set and the other is the unlearned model. This test corresponds to metric based on the Truth Ratio, and the results reported in Table 20 show that our framework still performs the best.
Table 20: Performance Comparison between our o and other baselines when continually unlearn- ing TOFU-forge001, -fogeet05, and -orgett0 in Fictitious Knowledge Generation. The unlearning effectiveness is measured by the Truth Ratio (the higher the better) of statistical test between reference model trained only on the retained set and the unlearned model.
Method Unlearning Request
Unlearning Request 2 Unlearning Request
PO
0.696
0.655
0.623
SOPO
0.734
0.745
0.768
Ours
0.927
0.923
0.865
G MORE ABLATION STUDY
G.1 DETAILED ANALYSIS FOR ABLATION STUDY IN MAIN CONTEXT
Unlearning Knowledge Detection. We detach the contrastive entropy LCEL in O and use SimCLR ('Ours w/ SimCLR') and MoCo ('Ours MoCo') with the augmentation using token masking. As for the scoring mechanism, we try using Mahalanobis Distance (d in Eq. 10) and Cosine Sim- ilarity (deas in Eq. [T] separately, which are termed as 'Ours w/o dcos" and "Ours w/o dMaha" Besides, instead of leveraging information from all model layers, we use only the last layer ("Ours w/ last layer'). Moreover, two state-of-the-ard OOD detection approaches: MDF (Xu et al., (021) and Agg (Darrin et al., 2024), are compared with ours. The experiments tious knowledge generation and question answering where the ID data is different unlearning sets, and the OOD data is the retained test set and the utility sets. We report the AUROC in Table 4 According to these results, we can observe that the full design of our O0D detector in O frame- work always achieves the best AUROC. Specifically, the performance drops when using SimCLR
conducted
ficti-
27

---

Published as a conference paper at ICLR 2025
or MoCo to fine-tune the backbone model, which implies the effectiveness of our contrastive tropy loss. The AUROC differences between "Ours w/o dcos" or 'Ours w/o dMaha' and 'Ours full' indicate the necessity of combining the Mahalanobis and Cosine Similarity distances to achieve bet- ter discrimination. The poorer performance of 'Ours w/ last layer' compared with "Ours full' tells us that aggregating representations of multiple layers is essentially beneficial. Besides, our meth- ods can also substantially exceed the state-of-the-arl unsupervised OOD detection approaches with unlabeled ID data only.
Unlearning Knowledge Optimization. In the objective of unlearning knowledge optimization (Eq. D). there is factor À balancing LCE and Lorth- We adopt 0, 0.01, 0.05, 0.1, and 0.2 for À to validate the importance of Lorth and the sensitivity of A. We conduct experiments on ques- tion answering and intent classification and report the metries of unlearning effectiveness and utility preservation after unlearning the last request. Table 5 illustrates that employing orthogonal loss contributes maintaining utility on the retained distribution and enhancing the unlearning ef- fectiveness. For instance, as the À increases, there is a corresponding increase in the accuracy of retained data. Moreover, the performance in preserving utility also improves.
weighteighted Inference. Instead of using soft weights to load unlearning LoRA, we test hard weighting strategy ('Hard-w' in Table (6). Specifically, after calculating the hyper- sphere boundary distance (dye in Eq. 12) on the unlearning set DU. we obtain the range Imin(dz (DU.)), max (D). Then for each testing instance æ. if its boundary distance dæ (a:) is within the above range, we chose to load the unlearning LoRA. Otherwise, we detach the LoRA. addition, we conduct a sensitivity analysis of the scaling factor Ç in Eq, 13 with a series of values I, 5, 50, and 100. These experiments are carried out on ScienceQA and CLINCI50, and we report the performance after the last unlearning request in T We observe that the 'Hard-w' method performs poorly regarding unlearning knowledge. With increase in the scaling factor S, our framework enhances its ability to unlearn knowledge more effectively. However, this increase adversely affects the framework's ability to maintain performance on the retained distribution and compromises its utility preservation. To address this, our framework adopts Ç as 10, striking a reasonable balance between effective unlearning and utility preservation.
G.2 CONDUCTING ADVERSARIAL ATTACKS TO BYPASS UNLEARNING KNOWLEDGE DETECTION
In the real-world deployment of our 3 framework, there may be concern that malicious attackers apply adversarial attacks (Gao et al. (224bb) to bypass unlearning knowledge detection. Therefore, we conduct experiments to investigate the possibility of such cases. Specifically, we implement an adversarial attack (Chen et al.) against OOD detection that injects certain perturbation to fool the OOD detector into identifying ID data as OOD data. In the context of textual data, we leverage heuristic replacement
characters generate such perturbation. The experiments TOFU (Ta- ble (1) show that the AUROC has no significant drop and the continual unlearning effectiveness remains nearly unchanged. We can conclude that is hard to bypass the unlearning knowledge detection and our o framework is robust.
Table 21: Robustness investigation of applying adversarial attack to unlearning knowledge detection in 3 framework on TOFU. The AUROC measured between the unlearning data and the retained data distributions.
Unlearning Request 1 D.U.L
Unlearning Request D.U..
Unlearning Request 3 AUROCT
Method
S.U.L
AUROCT
S.U.L
AUROCT
S.U.
D.U.l
Ours w/ Attack
12.5
15.0
97.5
16.4
19.8
92.5
17.4
19.5
92.2
Ours w/o Attack
12.5
14.4
97.8
15.8
20.3
92.5
15.5
19.7
93.0
G.3 SENSITIVITY ANALYSIS OF THE RANK OF LORA
We conduct experiments on the ScienceQA dataset with ranks 4, 8, 16, 32, 64, and 128, as detailed in Table. 22 Our findings indicate that unlearning performance diminishes when the rank exceeds 64, highlighting an increase unlearning difficulty with larger values of ranks. Conversely, higher ranks enhance the model's 's ability preserve utility and improve performance in R.D. metries.
28

---

Published as a conference paper at ICLR 2025
Table 22: Ablation study of the rank of LoRA on ScienceQA dataset.
Method
S.u.s
Unleuming Request
K.D. CQAT O.QA.7
Unleariing Requesi D.U. CQA' O.Q..7S.UU
S.U.,
D.U.I|K.D.. CQAA O.QA.7
Unlearning Reqursi 3
S.U.j D.U.I|R.D.T C.AAT O.QA.1
Unlearning Requesi
4.2
7.1
58.5
78.4
80.4
[0.6
3.2
57.0
78.3
81.8
10.8
13.0
58.
78.3
81.0
[4.1
15.4
[57.6
78.4
81.2
F-3
7.3
11.3
$1.5
18.2
79.4
3.1
8,5
83.4
78.4
800
5.5
12.5
58.6
18.2
8000
15.3
21,0
90.8
78.3
80.0
F16
1244
12.3
59.2
80.8
0.4
2.9
57.3
78.6
81.6
1.9
5.1
58.3
78.8
8122
6.3
10.5
58.5
78.9
812
12.8
134
1.3
3,4
81.4
82.0
5.8
7.7
78.7
8200
12.2
11.8
4.8
7.1
78.5
12.3
16.2
90.2
99.1
78.8
-228
13.2
12.3
88.5
81.6
90.8
78.9
82.0
42.1
44.9
99.0
46.7
50.1
90.4
78.8
82.4
H FUTURE WORK
H.1 IMPROVEMENT FOR UNLEARNING KNOWLEDGE DETECTION
direct improvement related to unlearning knowledge detection lies in the inference stage. In the inference phase of Oa framework, we need to feed the testing data into each OOD detector to calcu- late the likelihood of belonging to previous unlcarning distributions. In practical system deployment, can parallelize this process to enhance efficiency (Agrawal et al. (024). In our implementation, the OOD detector backbone uses the encoder-only Roberta model. Although this model can ex- tract high-quality representations, its performance is still limited when faced with complex inputs compared to larger-scale language models. Therefore, we consider directly using the target LLM to detect unlearning knowledge. This approach is feasible because, in the Oa framework, we use LoRA as an external module to achieve unlearning, and the original target LLM is available for inference. We should gain the following benefits if we replace the OOD detector backbone with an LLM. First, LLMs can better capture subtle text differences, improving OOD detection performance. Sec- ond, smaller language models like Roberta cannot effectively extract contextual information from complex and long contexts. Thus, if an unlearning request correlates the contextual information, such as the individual users' request to unlearn specific topics from their chat history with Chat- GPT, Roberta-based OOD detection cannot achieve this. In contrast, LLMs can extract contextual information well (Ding et al.), supporting more fine-grained OOD detection and more accurate ID data localization. Finally, using LLMs for OOD detection might eliminate the need for fine-tuning with ID data, as Uppaal et al. (2023) suggested that LLMS could provide accurate OOD detection predictions for text classification without any fine-tuning. This could further improve frame- work's efficiency. However, using I.LMs for OOD detection might require dedicated improvements to the scoring mechanism because mainstream LLMs now use a decoder-only architecture, which works by predicting the next token. In this case, the representation output by each attention layer of the LLM is likely be highly inconsistent in terms of token quantity and distribution. Therefore, whether our design based on layer-wise token average representation (Section 3.2) is suitable for LLMs requires further research. Extending O² to multimodal content (Gao et al. 2024c) represents a promising direction for future research. This extension would enable the model to unlearn in- formation from multiple modalities, such as text, audio, images, and video, thereby enhancing its ability to handle complex real-world scenarios.
H.2 DATA SELECTION FOR LLM UTILITY PRESERVATION
In the real world, we edit LLMs for various purposes, such as knowledge unlearning. However, the model editing leads to uncertain and unpredictable changes in the capabilities of the LLM (Qi ett al.] Gu et al.) 2024 Gupta et al. 2024). Recent studies have shown that model editing for a single specific task can cause performance degradation in seemingly unrelated tasks. This phenomenon is more pronounced in sequential or continual model editing (Gu et al., 2024 Gupta et al., 2024). To address this issue, similar to leveraging retained dataset to preserve model utility in LLM unlearn- ing (Liu et al. (204)), the intuitive approach is to identify which tasks and data distributions are most affected and then replay some representative data on the L.M (Gururangan et al., (020). However, identifying these tasks and data distributions on an LLM is extremely challenging (Chang et al.) 2024| Ortiz-Jimenez et al., 2024: Huang et al.) 2024). For example, to select suitable retained data for utility preservation in LLM unlearning, we might utilize some interpretable machine learning (ML) techniques (Singh et al. 2024) to locate the neurons activated by the unlearning data. Based on these identified activated neurons, could retrieve similar data to be used as the retained data. However, current interpretable ML techniques typically only achieve neuron localization for spe- cific model attributes, such as adversarial robustness (Wei et al.) or differential privacy (Chen al.)
29

---

Published as a conference paper at ICLR 2025
2024 For the fine-grained tasks and data distributions corresponding to unlearning requests, neuron localization either inaccurate or inconsistent in granularity. Therefore, effective data selection to preserve LLM utility during unlearning is research-worthy
I BROADER IMPACT
The introduction of O3 framework for LLM unlearning is an important effort across multiple do- mains. This is partieularly beneficial environments where continuous unlearning requests are necessary, such in systems dealing with dynamic privacy regulations evolving user prefer- ences. Furthermore, O ability to function without retained data significantly enhances its practi- cality, especially in sensitive areas like healthcare and finance, where maintaining access personal or confidential data for utiliity preservation is not feasible. This feature also extends to scenarios in- volving specialized tasks with naturally scarce data, such as rare disease diagnosis niche financial analysis, where data availability is inherently limited.
On a broader scale, o contributions to AT can enhance public trust in LLMs by addressing core concerns surrounding data privacy and compliance. With more robust unlearning capabilities, orga- nizations can ensure that sensitive information can be effectively removed from Al systems without sacrificing performance, thereby fostering better alignment with ethical Al principles and regula- tory requirements like GDPR (GDPR) 2018). This not only mitigates legal risks but also supports societal expectations for data autonomy and security, ensuring that AI systems parent, and more responsible. By enabling more effective unlearning, O enhances the long-term sustainability of Al technologies, creating safer and more equitable digital ecosystem.
adaptable, trans-
30


Reviews:
[{"review_id": "IE7Io3VbIZ", "reviewer": "Reviewer_9mDw", "summary": "This paper presents a solution to the Continual Unlearning problem, in which the model provider attempts to continuously erase the influence of requested data. The authors motivate their problem formulation because existing model unlearning methods fail to account for the scenario where the unlearning requests emerge continuously, and the model provider lacks access to previous data. To this end, the authors propose a continual unlearning framework that includes LoRA adapters with orthogonal components to minimize the interference among different unlearning requests, along with an OoD detector that measures the similarity between input and unlearning data to decide on how unlearning LoRAs should be loaded.", "questions": "How robust is the proposed method against ”targeted relearning attacks”? For example, in a safety-critical scenario, can the supposedly unlearned knowledge be solicited again after finetuning the model on related public datasets? [2]\n\n[2] Hu, S., Fu, Y., Wu, Z.S. and Smith, V., 2024. Jogging the Memory of Unlearned Model Through Targeted Relearning Attack. arXiv preprint arXiv:2406.13356.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written and motivated by real-world scenarios that require unlearning.\n2. The design choices for the proposed unlearning pipeline are justified through ablation studies.", "weaknesses": "The authors partially motivate machine unlearning as “ a representative approach for model safety and security by removing the influence of undesired data on the target model.” I very much agree with the assertion. However, the evaluations of the proposed methods mainly focus on unlearning knowledge instead of unsafe behaviors. The usefulness of the proposed method could benefit from additional evaluations against safety-oriented unlearning benchmarks such as WMDP [1].\n\n[1] Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J.D., Dombrowski, A.K., Goel, S., Phan, L. and Mukobi, G., 2024. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S9CqlicCXF", "reviewer": "Reviewer_J9vg", "summary": "The authors address significant practical challenges in developing machine unlearning techniques for large language models (LLMs), where current state-of-the-art approaches fall short due to their dependency on retained data and inability to manage continual unlearning requests effectively. To overcome these limitations, they propose the O^3 framework, which introduces an orthogonal low-rank adapter to enable continuous unlearning of requested data and an out-of-distribution detector to assess the similarity between incoming inputs and unlearning data. Comprehensive experiments show that O^3 achieves notably higher unlearning effectiveness and utility preservation than existing methods, all without relying on retained data, even under continuous unlearning conditions.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors solve the critical challenge of LLM unlearning by getting rid of the access to the retained data. The design of orthogonal LoRA demonstrates significant improvement in evaluation.\n\n2. The authors conduct extensive experiments to evaluate the effectiveness of the proposed O^3 method.", "weaknesses": "1. During the inference, each testing instance x will be fed into all OOD detector backbones. This might limit the method's scalability when the unlearning requests increase due to the higher computational cost. \n\n2. The experiments focus on the QA datasets, where each query only contains a single knowledge entity to be unlearned. The authors might need to evaluate the framework under more challenging and realistic settings, wherein for each query, there might be multiple knowledge entities to be unlearned. I am concerned about this because the OOD detectors are trained on single unlearning requests.\n\n3. The authors should improve the scale of unlearning requests (more than just 4 requests) to validate the claimed contribution that the O^3 framework can handle continual unlearning settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4xjXWkDwG4", "reviewer": "Reviewer_nwA1", "summary": "This paper presents the O3 framework, designed to address continual unlearning requests in LLMs without relying on retained data, a common limitation in existing methods. The O3 framework integrates an orthogonal low-rank adapter (LoRA) for unlearning requests and an Out-Of-Distribution (OOD) detector for measuring input similarity with unlearned data. The orthogonal LoRA prevents interference across multiple unlearning requests, while the OOD detector leverages a novel contrastive entropy loss and a layer-aggregated scoring mechanism to manage unlearning effectiveness dynamically. Extensive experiments demonstrate O3’s superior balance between unlearning effectiveness and utility preservation, particularly in continuous unlearning scenarios. Compared to state-of-the-art methods, O3 shows promising results in reducing computational costs and maintaining model utility across various tasks, such as question answering and intent classification.", "questions": "1. With fine-tuning unlearning methods, why is retained data not acceptable? This directly reflects the paper's motivation.\n2. this work's novelty seems insufficient, with LORA and OOD blocking the input/output of the LLM.\n3. If the model did not really forget the data distribution in the proposed method, is this method vulnerable to attack methods?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper addresses the LLM unlearning from the continual unlearning perspective.\nThis unlearning process does not need the retained data.", "weaknesses": "The proposed LLM unlearn methods LORA and OOD detector does not exactly unlearn the knowledge from the LLMs. They are just like two modules externally mounted outside the LLM and block the input and output of the LLMs related to the unlearn targets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)