Textual Unlearning Gives a False Sense of Unlearning
Authors: Jiacheng Du
Zhibo Wang
Jie Zhang
Xiaoyi Pang
Jiahui Hu
Kui Ren
Venue: ICML 2025 poster (2025)
Keywords: machine unlearning; unlearning auditing; membership inference attacks; data reconstruction attacks
Primary area: social_aspects->privacy
TL;DR: Our study highlights the vulnerabilities and privacy risks of machine unlearning in language models. We demonstrate that existing unlearning methods not only fail to completely remove the targeted texts but also expose more about them in deployment.

Abstract: Language Models (LMs) are prone to ''memorizing'' training data, including substantial sensitive user information. To mitigate privacy risks and safeguard the right to be forgotten, machine unlearning has emerged as a promising approach for enabling LMs to efficiently ''forget'' specific texts. However, despite the good intentions, is textual unlearning really as effective and reliable as expected? To address the concern, we first propose Unlearning Likelihood Ratio Attack+ (U-LiRA+), a rigorous textual unlearning auditing method, and find that unlearned texts can still be detected with very high confidence after unlearning. Further, we conduct an in-depth investigation on the privacy risks of textual unlearning mechanisms in deployment and present the Textual Unlearning Leakage Attack (TULA), along with its variants in both black- and white-box scenarios. We show that textual unlearning mechanisms could instead reveal more about the unlearned texts, exposing them to significant membership inference and data reconstruction risks. Our findings highlight that existing textual unlearning actually gives a false sense of unlearning, underscoring the need for more robust and secure unlearning mechanisms.

Textual Unlearning Gives a False Sense of Unlearning
Jiacheng Du
Zhibo Wang
Jie Zhang?
Xiaoyi Pang 12
Jiahui Hu
Kui Ren
Abstract
the unlearned one!
Language Models (LMs) are prone to "memoriz- ing" training data, including substantial sensitive user information. To mitigate privacy risks and safeguard the right to be forgotten, machine un- learning has emerged as promising approach for enabling LMs efficiently "forget" specific texts. However, despite the good intentions, is textual unlearning really as effective and reliable as ex- pected? To address the concern, we first propose Unlearning Likelihood Ratio Attack+ (U-LiRA+)), rigorous textual unlearning auditing method, and find that unlearned texts still be detected with very high confidence after unlearning. Fur- ther, we conduct an in-depth investigation on the privacy risks of textual unlearning mechanisms in deployment and present the Textual Unlearning Leakage Attack (TULA), along with its variants in both black- and white-box scenarios. We show that textual unlearning mechanisms could instead reveal more about the unlearned texts, exposing them to significant membership inference and data reconstruction risks. Our findings highlight that existing textual unlearning actually gives a false sense of unlearning, underscoring the need for more robust and secure unlearning mechanisms.
Details are: Alice Potter; March 20...
Malicious Inference
Train
Unlearn
Name: Alice Potter
Birth: March
Address: 17 Baker Street
is the training datal
Auditing
Figure I. Textual unlcarning gives a false sense of unlearning. Our findings highlight that: (1) the unlearned data could still be de- tected after unlearning via rigorous auditing.
(2) An adversary could instead infer the unlearned data by analyzing the models before and after unlearning.
GPT (O OO'Flaherty, 2024). However, user data often contains sensitive information, such as phone numbers and home ad- dresses (Liu et al., 2023). Recent studies have revealed that the training data of LMs can be maliciously inferred or ex- tracted (Mattern et al., 2023; Carlini et al., 2021), raising se- rious privacy concerns. Besides, the users are entitled the "Right To Be Forgotten" (Rosen, 201 1). Various regulations, such the General Data Protection Regulation (GDPR, 201 8), grant users the right to request complete deletion of personal data, even when it has already been trained. Addressing these concerns necessitates the mechanisms to "undo" the impact of specific training samples on LMs, safe- guarding data security and individual rights.
1. Introduction
In recent years, language models (LMs) have demonstrated impressive capabilities, driven by extensive training sam- ples. Among them, significant portion of real-world user data is contained to improve LMs in deployment. For in- stance, Amazon collects user conversations via smart speak- ers to enhance 2019), and OpenAI gathers user inputs to improve Chat-
Machine unlearning (MU) aims cific training samples on trained model (Bourtoule et al., 2021). There are generally two categories: exact unlearning and inexact unlearning. Exact umlearning directly removes the unlearned samples from the training set and retrains the model on the remaining samples (Bourtoule et al., 2021) While straightforward, this approach is computationally expensive, particularly for LMs with extensive training sam- ples and parameters. Therefore. inexact unlearning methods have attracted much attention, aiming to efficiently approx- imate the retrained model without retraining from scratch. They typically fine-tune the original model with the un- learned samples and an "inverse learning" objective, such as gradient ascent (Jang al., 2022). After a few fine-tuning
the impact of spe-
capabilities (Valinsky,
'The State Key Laboratory of Blockchain and Data Secu- rity, Zhejiang University, P. R. China "School of Cyber Sci- ence and Technology, Zhejiang University. P. R. China 'ETH Zurich, Switzerland. Correspondence to: Zhibo Wang <zhi- bowang dwwang@zju.cdu.cn>...
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Textual Unlearning Gives False Sense of Unlcarning
steps, the unlearned model behaves as if it were a retrained model that has not been trained on unlearned data.
texts can still be correctly inferred after unlearning, with an error rate below 0.1 %.
Despite promising intentions, existing MU methods may be less effective than expected. Recent studies suggest that the traces of the unlearned data, such as membership (Hayes et al., 2024) or adversarial image triggers (Pawelczyk et al., 2024), may not be completely erased after unleamiigg Be- sides, MU methods may reveal extra information about unlearned data. Specifically, comparing models before and after unlearning could further expose the membership (Chen et al., 2021) or visual features (Hu et al., 2024) of the un- learned samples. However, related studies primarily focus on image data, while the vulnerability and privacy risks of textual unlearning on LMs remain under-explored.
The textual unlearning mechanism additionally in- troduces new risks of leaking membership informa- tion on the unlearned texts. By querying the model before and after unlearning, malicious user can con- fidently infer the membership of unlearned texts by comparing the model outputs, even when only loss values are available.
The textual unlearning mechanism poses a substan- tial risk of malicious reconstruction on unlearned texts. With access to model weights before and after unlearning, an adversary could reconstruct unlearned texts with even more than 80% accuracy.
To fill the gap, we conduct a deep investigation into textual unlearning LMs. From the first principle, success- ful textual unlearning process should at least ensure that the memory of unlearned texts is completely erased from the model and no new privacy risks are further introduced. Thus, we propose two key research questions (RQ):
Our approaches, along with the concurrent works (Pawel- czyk et al., 2024; Hayes et al., 2024; Huang et al., 2024; Shumailov et al., 2024; Hu et al., 2024), highlight the im- portance of rigorous evaluation for unlearning mechanisms. Prior to widespread deployment, these mechanisms should undergo thorough auditing and analysis to prevent mislead- ing conclusions and potential security risks.
RQI: Can textual unlearning really achieve clean erasure?
RQ2: Will textual unlearning backfire and pose new pri- vacy risks?
To address RQI, we introduce a rigorous auditing method called Unlearning Likelihood Ratio Attack+ (U-LiRA+). By auditing the most-vulnerable samples in the training set (Aerni et al., 2024), U-LiRA+ enables a rigorous eval- uation on the erasing effectiveness for existing unlearning methods in the worst case.
2. Related Works and Preliminaries
2.1. Machine Unlearning
Machine unlearning aims to erase the impact of specific training data on a trained model (Bourtoule et al., 2021). There are generally two categories: exact unlearning and inexact unlearning. Exact unlearning methods remove the unlearned samples (Dunlearn) from the training set (D) and retrain the original model (Moriginai) on the remaining data (D remian) from scratch, which provides the cleanest erase and is considered as the "gold standard"' (Bourtoule et al., 2021). However, exact unlearning has a high computational overhead in practice, especially for LMs, given their large training corpus and massive parameters. To improve prac- ticality and efficiency, inexact unlearning methods are pro- posed to efficiently approximate the retrained model with- out retraining from scratch. Their basic idea is to fine-tune Moriginal on the D Dantearn with an "inverse learning" objec- tive. Specifically, Jang et al. propose to fine-tune Moriginal by Gradient Ascent (GA) based on an intuition that unlearn- ing should be the inverse process of learning (Jang et al., 2022). Later, Chen et al. propose to maximize the Kullback- Leibler divergence of the outputs between Moriginal and unlearned model (Muntcara) on D Duntearn (Chen & Yang, 2023). Besides, Zhang et al. treat Duntearn as negative preference samples and transform unlearning to a Negative Preference Optimization (NPO, (Rafailov et al., 2024)) problem on LMs (Zhang et al., 2024b).
For RQ2, we explore the privacy risks of textual unlearning mechanisms in deployment by developing Textual Unlearn- ing Leakage Attacks (TULA). In the black-box scenarios, we employ TULA for Membership Inference (TULA-MI), which enables an adversary to infer the membership of the unlearned text by querying the models before and after un- learning. In the white-box scenarios, we propose TULA for Data Reconstruction (TULA-DR)) With access to model weights, the adversary could effectively reconstruct the un- learned texts via continuous constrained optimization.
Through extensive evaluations, our findings critically reveal that textual unlearning actually gives a false sense of unlearning! Existing methods for textual unlearning fail to completely erase unlearned texts, and their deployment will instead introduce heightened privacy risks. Specifically, we highlight the following key findings:
Unlearned texts remain detectable with high confi- dence on the unlearned LMs. We argue that previ- ous unlearning auditing methods highly overestimate the erasing effectiveness of existing textual unlearning methods. In the worst case, over 70% of unlearned
2

---

Textual Unlearning Gives False Sense of Unlcarning
2.2. Machine Unlearning Auditing
is more likely if a is an unlearned unseen sample:
A successful unlearning process should ensure that the Muntearn does not disclose any membership information, indicating that Punteorn is part of D. Therefore, Mem- bership Inference Attacks (MIAs) (Shokri et al., 2017) are widely used to audit MU methods (U-MIAs). MIAs aim to determine whether a sample belongs to the training set by analyzing model output discrepancies (e.g., loss values). Consequently, if the adversary cannot confidently distin- guish whether a sample originates from the 'D Duniearn or the unseen dataset D unseen (e.g.. nearly random guess) Muntearn, the unlearning process is deemed successful. Ex- isting MU methods mainly use population pppuee U-MIAs for auditing (Shi et al., 2024), and recent works (Hayes et al., 2024; Pawelczyk et al., 2024) attempt to use per-sample U- MIAs to improve auditing precision.
| Maniin 0 g,in)
A MunMuaan:: x) = NNS(M Muntarnni 2) Pr,ouu: o ownt
Overall, per-sample U-MIAs provide more precise auditing than population- methods, since the methods discrim- inate each sample independently rather than with global threshold. However, we argue that all the existing auditing methods for unlearning are not rigorous, as they fundamen- tally fail to properly select the audited samples.
2.3. Privacy Attacks against Machine Unlearning
Existing privacy attacks on MU generally follow a basic idea: inferring the privacy of the unlearned data via com- paring the models before and after the unlearning process, Based on the adversary's 's objective, there are two primary paradigms: MIA and Data Reconstruction Attack (DRA).
Population-based U-MIAs. Previous studies typically em- ploy simple population-based MIAs to audit MU methods. The general approach involves an adversary selecting K/ / samples from D and unlcarning them with the audited MU method to obtain Municarn- Subsequently, K/2 unseen samples ge holdout data from D) are randomly chosen to form an auditing set of size K. The adversary then guesses that one of the audited samples x belongs to D if ((Muuitarni x) > 5, where S is scoring function (e.g.. negative cross-entropy loss) and T is a threshold. If the at- tack accuracy, typically measured using AUC-ROC (Shokri et al., 2017), is close to 50%, it indicates that the adversary cannot distinguish Duntearn and D Dunseen on M, suggest- ing that the unlearning process has been successful.
Chen et al. demonstrate that the MU mechanism can inad- vertently reveal membership information about unlearned data (Chen et al., 2021). Specifically, for a target sample v, an adversary randomly selects an identically distributed auxiliary dataset D' and trains two models, M' of'original and M' untearn (after unlearning x), on x D'. The adversary queries both models with I, combining their outputs to create the positive feature. For the negative feature, the adversary randomly selects another unseen sample, queries both models, and combines the outputs. By repeating this process multiple times, a dataset of feature pairs is con- structed to train a binary classification attack model deter- mining whether x member of training data, Hu et al. demonstrate that adversaries can reconstruct an unlearned sample by exploiting the difference in model weights before and after unlearning, A = Moriginat Muntearn (Hu et al., 2024). This approach relies on the assumption that V aligns with the gradient direction of the unlearned sam- plc x on Moriginat. Specifically, the adversary begins by inferring the label y/ of the unlearned sample using prob- ing technique. Next, the adversary randomly initializes a dummy sample x'' with the same shape as x and computes the gradient V' of (x' on Moriginal. By minimizing the cosine similarity between Ao and V', the adversary it- eratively updates x' until convergence. However, existing studies focus solely on image data, and the potential privacy threats against textual unlearning remain underexplored.
Per-sample U-MIAs. Per-sample U-MIA (Hayes et al., 2024; Pawelezyk et al., 2024) builds upon the Likelihood Ratio Attack (LiRA) method proposed by Carlini et al. (Car- lini et al., 2022), which formulates MIA as a hypothesis testing problem. For an audited sample z, the adversary models the score distributions under the hypotheses that is a member of Duntearn, and that is one of D Danscen- Given the score of x on Muntearn. the attack then applies likelihood ratio test to distinguish between the two hy- potheses. To estimate the distributions, the adversary trains multiple shadow models (Shokri et al., 2017) by repeat- edly sampling an identically distributed auxiliary training set D', For the target x, the adversary trains the unseen shadow model Mout on D', and trains the unlearn shadow model Min on U 'D' after unlearning v. With a sufficient number of shadow models, the adversary fits two Gaussians N (Mazinn 0 2 and (H ouu to the scores from Min and Mout models on the target sample x. Finally, the adversary applies a standard Neyman-Pearson test to determine whether the observed score from the Muniearn
3. Rigorous Auditing on Textual Unlearning
Let's go back to the drawing board to rethink the unlearning auditing. The basic idea is to test whether an adversary could confidently distinguish an unlearned sample from an unseen sample through outputs on Muntearn- However, existing auditing methods ignore a crucial and fundamental
3

---

Textual Unlearning Gives False Sense of Unlcarning
issue: the proper selection of audited samples. Typically, these methods randomly partition the dataset into unseen data D Dunseen and training data D. A subset of D is then ran- domly selected as unlearned data D Dunicarn. while an equal number of unseen and unlearned samples are designated as the audited samples. However, this approach inadver- tently causes Moriginat to exhibit "false memory", where the model acts as if it had been trained on unseen samples. Such results mainly arise from the model's ability to gener- alize, as the audited samples are from the same distribution. Furthermore, the inherent overlap and redundancy among text samples exacerbate such an effect (Duan et al., 2024).
Algorithm 1 U-LiRA+
Args: model M. mislabel function 2. learning algorithm A. unlearning algorithm U. number of shadow models 7, size of audited set N, logit function ø.
Doudit ~ D sample the audit set
Dandit + @(Dandaa) mislabel the audited samples
Mask - {m; | |m ~ U'niform( 1), ,,
D' ~ D sample a tested training set
M A(D* U Mask @ Daudit)
inject audited samples unleam audited samples
M + UM", Mask @ Daudit)
Preds+ + {
while < N do
O < () 0.0++ + 0
while 1 S T do
D ~ D sample a shadow training datasct
The false memory effect leads to considerable overlap be- tween the output distributions of Pantearn and Danseen on Moriginat. which makes MIA-based auditing methods un- able to well distinguish between the samples even before unlearning. This flawed foundation undermines the fairness and rigor of unlearning auditing, as the distinguishability between D Duntearn and D Dunscon could be inherently poor.
D + DU (Zn, yn)) inject a audited sample
Moriginal t A(D)
Municarn U U (Ir, Vn )))
Mretrain +- A(D (3 Vn ))
Olt] + -(((n, Vn Muntearn)
Ólt t- =(((nn Vn ); Mretrain)
end while
Hr, a <- fit Gaussian (0)
A. ô < fit Gaussian (Ö)
To conduct rigorous unlcarning auditing, it is crucial to utilize audited samples with "zero memory", ensuring the model knows nothing about them before being trained. To address this, we propose U-LiRA+, which performs rig- orous auditing on unlearning algorithms by additionally constructing and injecting mislabeled samples. Mislabeled samples naturally occur in real-world datasets. For instance, in sentiment classification task, a text such as "I love this movie so much!" may be incorrectly labeled as negative instead of positive due to human error. Since mislabeled samples are typically small in number and counterfactual, normal model cannot generalize to them unless explicitly trained (Aerni et al., 2024). As a result, mislabeled samples are considered as the most vulnerable samples to unlearning, since whether they have been trained makes a significant difference in model outputs. In other words, their presence provides a worst-case scenario, enabling fair and rigorous auditing on the effectiveness of unlearning algorithms.
f- (((2n, Vna M:M)
Noom.
Pueemer Nooune > (o
If Preemeer > Then Preds[n) (m) + Else Preds[n] + 0 end while Return PPR@LouFRR+ +- (Oreds, Mask)
4. Textual Unlearning Leakage Attacks
In this section, we investigate the potential privacy risks of textual unlearning mechanisms in deployment. We pro- pose the Textual Unlearning Leakage Attack (TULA) and its variants in black- and white-box scenarios: TULA for Membership Inference (TULA-MI) and TULA for Data Reconstruction (TULA-DR).
4.1. Problem Statement
In an ML system, there are three primary roles: data con- tributors, the model developer, and beneficiaries (including users and collaborators), as shown in Figure 2. Data contrib- utors provide their private data to the model developer, who utilizes the samples to train model. Beneficiaries are cate- gorized based on their access to the models. Users can query the model via the developer's API (c.g.. ChatGPT (OpenAI, 2023)). In contrast, collaborators, such as organizations con- tracting with the developer, have full access to the weights and can modify or deploy the model locally.
The detailed process of U-LiRA+ is outlined in Algorithm 1. To rigorously audit the unlearning algorithm U. we first sample an audit dataset D Dandit from the training data distri- bution and mislabel it. Next, we create a binary mask vector to randomly divide Daudit into two equal parts: unlearned samples and unseen samples. We then train test model M' to evaluate the effectiveness of U. For each audited sample, we generate T groups of shadow models to estimate the distribution of model outputs when the sample is either unlearned or unseen. Subsequently, we query M with the audited sample and determine which distribution its output corresponds to, predicting whether the sample has been un- learned. Finally, we compute the TPR@ LowFP PR of the predictions against the ground truth mask. If the value is (nearly) zero, the unlearning process is deemed successful.
Suppose a contributor requests the developer to delete its pri- vate data J. Upon receiving this request, the developer will utilize unlearning algorithms to obtain Muntearn and re- move the unlearned x from the database. Subsequently, both the user-oriented API and the weights shared with collabora- tors will be updated completely eliminate the contribution of to the system. However, the unlearning mechanism

---

Textual Unlearning Gives False Sense of Unlearning
Reconstruct
Reconstructed text:
Algorithm 2 TULA-MI in the Strict Case
unlearned by W-W!!
Alice, birth March 20, 1991,
Inputs: Black-hox models Moriginal- Munlearn: target sam- ple (1, score function S,threshold ?
if [S(Munteara (r), v) ((MMorgina(()))u)) [>] Return (2, y) is an unlearned sample
>> then
Malicious Collaborator
TULA-DR
else
Want my data deleted
Return (æ, is an unseen sample end
weights W
weights w
currence of unlearning in practice by continuously querying the model with target samples. If the outputs from two con- secutive sets of queries change, the adversary could execute the attack to infer whether any samples have been unlearned. The strict black-box case represents the minimal knowledge adversary access in practice, thereby indicating the lower bound of potential privacy leakage.
Alice, born March -i0---t $2.Baker Streek.w
Unlearning
response
response
API
API
Data Contributors
Model Developer
Method. Typically, the model yields higher confidence for training samples compared to unseen ones. For ex- ample, in terms of loss value, Moriginat exhibits lower loss for an unlearned sample, while Muniearn produces a higher one. In contrast, for unseen sample, both mod- els maintain similarly high loss values. This observation highlights that unlearned samples undergo significunt con- fidences changes before and after unlearning, whereas un- seen samples generally do not. Therefore, we propose TULA-MI in the strict case, leveraging confidence varia- tions to infer the membership information of target samples. The adversary evaluates the confidence change of target sample before and after unlearning using threshold y, v) If the change exceeds is likely to be an unlcarned sample. The de- tailed procedure is outlined in Algorithm 2.
Q unlearned data?
query
query
Yes, she is!
Loss
Loss'(
Malicious User
TULA-MI
Figure 2. An overview of Textual Unleaming Leakage Attack (TULA) against ML systems. TULA for Membership Inference (TULA-MI) infers the membership of the unleareed texts by query- ing the models before and after unlearning. TULA for Data Re- construction (TULA-DR) reconstructs the unlearned texts using the weight differences between the two models.
[S(Munlcrrr (æ), y)
inevitably creates a new risk, even x are perfectly unlearned: users and collaborators can access both the models before (Moriginat) and after unlearning (Muntearn), which in- stead emphasizes the "non-existence" of the unlearned sam- ple and opens a new window for inferring x.
4.2.2. TULA-MI IN RELAXED BLACK- -BX CASE
Threat Model. Previous MIAS typically assume relaxed black-box scenario (Chen et al., 2021; Carlini et al., 2022). Specifically, the adversary could obtain the model's logits via queries, additionally possess identically distributed auxiliary dataset, and know the model architecture. This case reflects the maximum knowledge an adversary can ob- tain in black-box scenarios, thereby indicating the upper bound of potential privacy leakage.
4.2. TULA for Membership Inference
In this subsection, we introduce TUAL-MI, where a ma- licious user aims to infer the membership information of unlearned data through black-box model access. Depending on different levels of black-box knowledge, investigate both strict and relaxed black-box cases to reveal the adver- sary's threat boundaries in this scenario.
Method. In this setting, a powerful adversary should be ca- pable of performing targeted MIA for each sample. Ideally, the adversary determines the membership of by carefully fitting two distributions to its output features: the distri- butions when x is an unlearned sample (D (Duntearn) or an unseen sample d. To achieve this goal, we de- compose the process into three steps: (1) Feature Space Construction: For the target sample I, we construct the feature vector [lores Lul lori [w] based its logits from models before and after unlearning, denoted as lori and Lut- respectively. The term lor: - serves as an augmentation to capture logit changes more effectively. (2) Distribution Esti-
1.2.1. TULA-MI IN STRICT BLACK-BoL CASE
Threat Model. Given a target sample x, a malicious user aims to determine whether æ is unlearned data by querying the models before and after unlearning. In a strict black-box scenario, the user inputs via API and receives only the out- put score, e.g., loss value L(x, y), quantifying the model's confidence on Such scenarios are common in real-world applications, such as emotion diagnosis systems that output an emotion label along with confidence score based on dialogues. Besides, the adversary could easily detect the oc-
5

---

Textual Unlearning Gives False Sense of Unlcarning
mation: We first sample a training set D from the auxiliary dataset, train shadow model Moriginal on D U x; and ob- tain Muntearn after unlearning z. The two models are then queried with r to construct the output feature. By repeating this process multiple times, we estimate the D unlearn for I. For Dusseen we sample another training set D'' but ex- clude z, train M' original on D, and obtain M' untearn after randomly unlearning& a sample x'. Similarly, we can obtain D Dinseen after multiple rounds of sampling and querying. (3) Attuck Model Training: Features sampled from the two distributions are labeled as 1 and 0, respectively. A classifier is then trained to infer the membership of x. The detailed procedure is outlined in Algorithm in Appendix G.
Algorithm 3 TULA-DR
Inputs: original weights o Ooraginal, unlearned weights Buntearns unlearning algorithm U, token embedding dimension Rixd, learning rate a, regularization factor p. the maximum number of iterations N. & ~ Rxxd : : B candidate random initialization A Doriginal -unneean while not reaching do au (aiginat-((.8))
V Vouriginal (8, 0) +
80 ariignna
v Peegpn
(6:0)-A0
Lrec +
ooiiical (3.5)
lA
Lreg + (.Ei /6+
EJ=n [es]lx))
àt x- 8 + a. @.lrrettt.......
08
ÿ x Ú+ a. Ollrret (.erea)))
4.3. TULA for Data Reconstruction
89
Clip( (ē; Niow; Iup)
In this subsection, we introduce TULA-DR, where a mali- cious collaborator aims to reconstruct the unlearned texts with white-box model access.
end while Return Decoding( 1)
ences before and after unlearning:
1. THREAT MODEL
Candidate Initialization: To address the challenge (I), reformulate the discrete optimization problem as con- tinuous one. Specifically, assuming the token embedding dimension 1 and token embedding layer WoordEbd' R/V|xd, a tokenized text x can be represented with an em- bedding vector in 2_xdd Such transformation significantly reduces the search space as d < \V.. Additionally, gradient- based optimizers (e.g., Adam (Kingma & Ba, 2014)) can be utilized to efficiently search the target text.
In the white-box scenario, the adversary has access to the model weights both before (foriginal) and after (0 (Ountearn) unlearning and aims to reconstruct the unlearned samples. Additionally, we assume the adversary knows the unlearn- ing algorithm U' employed by the model developer. This assumption could be practical. First, the unlearning algo- rithm itself not highly confidential, thereby posing a risk of disclosure (e.g., via a complicit developer). Second, model developers may be obligated to disclose details of the unlearning algorithm, performance, and other relevant infor- mation to collaborators to uphold transparency and ensure informed updates regarding model versions.
Reconstruction Objective: To address the challenge (2), we design a loss function that guides the optimization by mirroring the unlearning process and approximating the weight differences. Inspired by (Hu et al., 2024) the gradi- ent of the unlearned samples over the original model aligns with the difference in model weights before and after un- learning (AO). Consequently, we compute the gradient of the candidate (é,, i)) on Moriginat executing the unlearning algorithm U, and then use cosine similarity to compute the angle between the gradient and weight difference, which is utilized as the loss to update candidate:
4.3.2. THE PROPOSED METHOD
Suppose that an unlearned sample consists of text x and label y. where is a sequence of L tokens ((Ii,..., (kr.)) with each tky representing a number in the vocabulary V. and y E IR LxC corresponds to one of C' categories. Intu- itively, the adversary randomly selects L tokens from V and guesses a label e among C' categories. The guessed sample is then evaluated by applying the unlearning algo- rithm U to check whether enables updating Goriginal to Ountearn- However, this approach is computationally infea- sible: (1) The text is represented in large discrete space, making the search challenging. The adversary faces the task of identifying the target unlearned sample from C' x [ candidates, which is practically intractable. (2) The search process is poorly guided, as the weight differences before and after unlearning are not well incorporated. (3) The lack of constraints for effective search.
V Poriginall (0.0) () A6
Lree
(I)
V
Ooriginal (2, 1) ||0
To address the challenge (3), we introduce sentence regularization and boundary clipping mechanisms to constrain the distribution of reconstructed embeddings and accelerate convergence. The word em- bedding layer W WoordEbd pre-stores all possible candidate tokens in a vocabulary. Based on this intuition, we first introduce a sentence regularization term that enforces the norm of reconstructed sentence embeddings to approximate the average norm of the vocabulary (Balunovic et al., 2022),
Embedding Constraints:
Therefore, we propose TULA-DR, a method that efficiently reconstructs unlearned samples by exploiting weight differ-
6

---

Textual Unlearning Gives False Sense of Unlcarning
preventing the optimization process from local optima:
90
60
75.78
72:06
70
61.82
v
Creg =
11 1la V 1e
E 3=1
(2)
TPR@0.1
41.60
30-
Additionally, we propose boundary clipping mechanism ensure that each embedding element remains within the maximum and minimum values of all potential tokens at the corresponding element position, thereby calibrating the optimization direction and accelerating convergence:
20:
20
12.25
7.128.26
10.54
0.000.00
0,28
79.20
27.21
70-
68.38
c0-
57.83
Clip( ( Flow, Tup) = min (max (ft, klow) , Tup))
30
where Flow and Zup are calculated in W WordEEbd. The detailed procedure outlined Algorithm 3. Besides, the Decoding(-) function refers transforming the recon- structed embeddings into text space, Specifically, we use the model's 's embedding layer "vocabulary" and apply cosine similarity to match the indices of the embeddings and then convert them into readable texts via the tokenizer.
20
14.25
13.68
14.25
10-
0,000.00 Retrain
0.000.85
Unlearning Mathod
Audit Martginal
Audit Maninara
U-LIRA+
Baseline
Figure 3. Auditing unlearning methods on SynthPA--ine dataset. The lines and bars represent the auditing results on original models and corresponding unlearned models, respectively. OPT- 1.3h model (Top), Pythia-1.44 1.4b model (Bottom).
5. Experiments
5.1. Experimental Setups
whether specific sample used among massive pre- training data (Zhang et al., 2024a). Therefore, we utilize synthetic datasets to rigorously ensure the fine-tuning on previously "unseen" data. Specifically, the synthetic sam- ples are entirely based on fictional scenarios, and generated by GPT-4-based agents released after the models we utilized. Through these efforts, hope ensure that the setups rigorous and thus avoid being potentially misleading.
Datasets: In this paper, we investigate unlearning mech- anisms on fine-tuned LMs, since the fine-tuning datasets configurable and practical for evaluations (Maini et al., 2024). We manually construct two synthetic datasets, SynthPA--age and SynthPAI-inc, for the evaluations. They are two binary text classification datasets for attribute in- ference tasks, where texts consist of comments generated by GPT-4-based agents during interactions, labels are com- menters' attributes, such as age (young or old) and income (low high). A detailed introduction on datasets is pre- sented in Appendix A. Besides, we focus on the text classifi- cation tasks, because: (1) they enable us to design rigorous audit samples for rigorous unlearning auditing in the worst case. Specifically, the utilized mislabeled samples have been found be most vulnerable to privacy leakage on image classification tasks (Aemi al., 2024), and rigorous ples for text generation tasks currently under-explored. (2) They ensure fair evaluation on MIA-based auditing and attack methods (Pawelezyk et al., 2023), as MIA remains poorly defined and lacks robust metrics for text generation tasks (Duan et al., 2024).
Models: We employ Pythia-1.4h (Biderman et al., 2023) and OPT-1.3b (Zhang et al., 2022) LMs for evaluations. To ensure effective learning, we apply full-parameter fine- tuning and meticulously configure the training process to prevent overfitting. Detailed training configurations and performance results are provided in Appendix B.
We conduct comprehensive in- vestigation on both exact and inexact unlearning mecha- nisms. For exact unlearning, we examine the retraining method (Retrain). For inexact unlearning. we evaluate four widely adopted methods: gradient ascent (GA), maximiz- ing Kullback-Leibbee divergence (KL), negative preference optimization (NPO), and task vector (easkVec). To ensure erasure is thorough possible, apply full-parameter fine-tuning and carefully configure each method. Detailed descriptions and configurations are provided in Appendix C.
Unlearning Methods:
Why synthetic datasets: Previous studies indicate that LMs might have "already seen" the fine-tuned samples in public datasets during pre-training, potentially undermining the reliability of unlearning processes (Maini et al., 2024). In addition, the real-world data can not meet our requirement, since: (1) pre-trained models are trained on vast amounts of real-world data, which is typically source. (2) Recent studies indicate that we can not yet robustly detect
5.2. Evaluations on Textual Unlearning Auditing
In this subsection, we evaluate our proposed auditing method, U-LiRA+, across five unlearning methods on

---

Textual Unlearning Gives False Sense of Unlearning
22
Table TULA-MI in strict black-hox cases. The metric is aver-
20-
18-
18.49
17.86
18.70
aged NTS@IFS, and the presented results are evaluated on the Pythia-I.4b model and SynthPAI-ap age dataset.
16
14
13.03
12
N
ScoreFunc
Retrain
GA
KL.
NPO
Task Vec
10-
Confidence CrossEntropy
4.7
6.45
4.65
4.6
6.35
5.67
32
Hinge
11,4
4.6
4.75
7.4
4.35
7.6
9.5
4.3
4.75
7.35
2.10
0.0r0.53
1.47
0.420.84
0.210.42
22
Confidence
4.55
14.25
10.2
10.55
13.95
20
64
CrossEntropy
27.55
4.7
10.1
5.85
4.7
18-
17.65
16.18
17.86
16.81
Hinge
4.45
18.2
8.8
21.75
18.05
16
14
13.87
12
SynthPAl-ine dataset, shown in Figure 3. We employ the SOTA auditing method U-LiRA as baseline (Hayes et al., 2024). Detailed implementation of U-LiRA+ and results on SynthPAl-age dataset are provided in Ap- pendix D. We default the unlearning ratio to 1%. We em- ploy the True-Positive Rate at a low False-Positive Rate (TPR@0.1% PRF) as the audit metric, reflecting how con- fidently adversary compromises individual privacy. For Munlearn: lower value indicates that the adversary has less confidence in distinguishing audit samples (unlearned and unseen samples), implying more successful unlearning.
10
7.77
2.10
0.00
0.23
0.42
1:47
2.31
0.00
Retrain
GA
KL
NPO
TaskVac
Uniearning Mathod
LIRA
ULeaks
TULA-MI
Figure 4. MIAs in relaxed black-box case on SyuthPAI-age dataset. OPT-1.3b model (Top), Pythia- 1.4b model (Bottom).
SyuthAA/age and 24, 48 for SynthPAI-inc). Besides, we in- corporate the metric NT'S1F' (Number of True Samples
Through rigorous auditing of existing unlearning methods, our empirical results reveal that the current auditing method significantly overestimates the effectiveness of unlearning mechanisms. For instance, in the audit results of KL. method on the O.T-1.3b model in Figure 3, the baseline suggests that KI. substantially reduces TPR on the unlearned model (from 12.25 22.25% % to 0.28%). However, our U-LiRA+ method shows that TPR remains at 41.60% on Munlearn- indicat- ing that the adversary can still effectively distinguish audit samples after unlearning. Moreover, while Retrain could achieve thorough erasure, inexact unlearning methods typ- ically present false sense of unlearning. Across various settings, inexact unlearning methods fail to effectively erase audited samples compared to Retrain, as evidenced by the consistently high TPR observed with U-LiRA+.
False Sample), which quantifies the number correctly inferred samples with only one mistake.
We present the results on Pythia-1.4b and SynthPAI-age dataset in Table I, and results in other settings provided in Appendix E. The results reveal that even with only access to output scores, the adversary can effectively infer mem- bership information of the unlearned samples, with several samples correctly identified while just one incorrect. More- over, Retrain is more vulnerable than inexact unlearning methods in such cases. As shown Table 1, with a single incorrect guess, the adversary can infer more than 27 correct samples against the retrained model.
5.3.2. TULA-MI IN RELAXED BLACK-BOX CASE
We then move to the relaxed black-box cases, where the adversary can query the model to obtain logit vectors, is aware of the model architecture, and has access to identi- cally distributed auxiliary dataset, Under these conditions, we evaluate two baseline methods: LiRA (Carlini al., 2022) and ULeaks (Chen et al., 2021). Detailed descrip- tions and implementations of these methods are provided in Appendix G.
5.3. Exploring Black-Box Privacy Risks against Textual Unlearning Systems
In this subsection, we evaluate the proposed TULA-MI against textual unlearning systems when the adversary has black-box access to the models before and after unlearning.
5.3.1. TULA-MI STRICT BLACK- BLACK-BOX CASE
We begin with the strict black-box scenarios where the ad- versary can only access output scores. To ensure a com- prehensive evaluation, we examine three commonly used score functions: confidence (f(x)y), cross-entropy loss (-log((x)) ())) and hinge loss (log(p/(1 (1 - p)). p- (x) Additionally, we consider two unlearning ratios, 0.5% and 1% (the number of audited samples N 32, 64 for
As shown in Figure 4, we present the MIA results on SynthPAI-age dataset, and additional results on SynthPAI- inc dataset are included in Appendix F. With a wrong guess below 0.1%, TULA-MI accurately infers membership for over 18% of the samples, highlighting significant privacy risks. Compared to ULeaks, our method achieves superior performance by employing a more accurate per-sample at-
8

---

Textual Unlearning Gives False Sense of Unlearning
Table 2. Text reconstruction performance of TULA--R.
SynthPAI-age
SynthPAI-ine
Model
Method
R-1
R-2
R-L
R-1
R-2
R-L
GA
81.04
66.75
81.35
81.92
58.16
75.15
KL
78.8
43.29
71.18
65.8
43.58
65.47
Pythia-1.4b
NPO
69.61
32.31
64.42
74.27
44.43
69.54
Task
81.94
65.59
65.6
72.63
41.24
67.8
GA
43.57
17.5
40.36
38.05
10.28
35.13
KI.
15.06
1.82
12.83
17.35
4.25
16.46
OPT-L.3b
NPO
32.1
10.69
30.81
31.28
7.3
26.61
Task Vec
41.27
16.91
38.18
46.95
22
40.29
lack strategy. While LiRA also utilizes a per-sample attack and fits normal distributions for unlearned and unseen out- puts, its effectiveness is hindered by redundancy and overlap among textual samples, making challenging to fit inde- pendent distributions for reliable discrimination. In contrast, method explicitly learns complex decision boundaries for target samples in unlearned and unseen cases, enabling more robust MIA.
6. Conclusion
In this paper, we investigated textual unlearning methods on LMs, highlighting their vulnerabilities and privacy risks. We introduced rigorous unlearning auditing method, U- LiRA+ providing an accurate assessment of erasure ef- fectiveness. Additionally, to explore the privacy risks of textual unlearning mechanisms, we proposed TULA and its variants in black- and white-box scenarios: TULA-MI and TULA-DR. Through extensive evaluations, ings revealed that textual unlearning actually gives a false sense of unlearning. Current methods fail to completely erase unlearned texts, which remain highly detectable after unlearning. Moreover, deploying textual unlearning mecha- nisms could instead increase privacy risks. Specifically, by examining models before and after unlearning, membership information of unlearned texts be confidently inferred using TULA-MI. In the white-box scenario, they can even be accurately reconstructed using TULA-DR, Our findings indicated that existing textual unlearning methods are less effective and reliable than expected, highlighting the need for more robust and secure unlearning mechanisms on LMs.
find-
5.4. Exploring White-Bor Privacy Risks against Textual Unlearning Systems
In this subsection, we examine the effectiveness of TULA- DR with white-box access to model weights before and after unlcarning. To ensure a fair evaluation, we employ three metrics assess reconstruction performance: ROUGE-1 1). ROUGE-2 (R-2), and ROUGE-L (R-L) (Lin, 2004), which respectively measure the ratios of reconstructed un- igrams, bigrams, and the longest-matching subsequence. Additionally, reconstructed examples are provided in Ap- pendix J. We assume a default scenario where Morsginal trained on 1000 samples, with one sample subsequently unlearned. Furthermore, our method can be extended to batch unlearning scenarios, and the results are presented in Appendix H. For TULA-DR, the learning rate a is set to 0.1 for the Pyth model and 0.45 for the 1.T- model, and the regularization factor ,3 is fixed at 0.1. Besides, an ablation study is further provided in Appendix I.
A Acknowledgements
This work is supported by National Natural Science Foun- dation of China (Grants No. U24B20182, 62122066), National Key R&D Program of China (Grant No. 202 12803), and Key R&D Program of Zhejiang (Grant No. 2022C01018).
As shown in Table 2, discrepancies in unlearning weights can result in substantial textual privacy leakage. Specifi- cally, for the Pythia-l.4b model, over 80% of the original words were accurately reconstructed, as the R-1 values show. Besides, the reconstructed sentences preserved both seman- tic structure and meaning with high R-2 and R-L values, respectively. Besides, for the OPT-1.3b model, although the reconstructed sentences exhibited disordered semantic structure, an average of 40% of the original words were still recovered, exposing much information. Specifically, the adversary could still utilize the reconstructed words to infer the useful information of the original texts.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. We reveal the vulnerabilities and privacy risks of existing machine unlearning methods on language models and encourage the development of more robust and secure unlearning mechanisms. There are many potential societal consequences of our work, none of which feel must be specifically highlighted here.
9

---

Textual Unlearning Gives False Sense of Unlearning
References
Hu, H., Wang, S., Dong, T., and Xue, M. Learn what you want to unlearn: Unlearning inversion attacks against machine unlearning. arXiv preprint arXiv:2404.03233, 2024.
Aerni, M., Zhang, J., and Tramèr, F. Evaluations of machine learning privacy defenses are misleading. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp. 1271-1284, 2024.
Huang, Y., Liu, D., Chua, L., Ghazi, B., Kamath, P., Ku- mar, R., Manurangsi, P., Nasr, M., Sinha, A., and Zhang, C. Unlearn and burn: Adversarial machine unlearn- ing requests destroy model accuracy. arXiv preprint arXiv:2410.09591, 2024.
Balunovic, M., Dimitrov, D., Jovanović, N., and Vechev, M. Lamp: Extracting text from gradients with language model priors. Advances in Neural Information Processing Systems, 35:7641-7654, 2022.
Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O' O'Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for ana- lyzing large language models across training and scaling. In International Conference on Machine Learning, pp- 2397-24301 PMLR, 2023.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing mod- els with task arithmetic. arXiv preprint Xii::2212.00008 2022.
Jang, J., Yoon, D., Yang. S., Cha, S., Lee, M., Logeswaran,
L., and Seo, M. Knowledge unlearning for mitigat- ing privacy risks in language models. arXiv preprint arXiv:2210.0155 2022.
Bourtoule, L.,, Chandrasekaran, V., Choquette-Chov C. A., Jia. H., Travers, A., Zhang, B., Lie, D., and Papernot, N. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 141-159. IEEE, 2021.
Ke, G.. Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural infor- mation processing systems, 30, 2017.
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-2650, 2021.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Carlini, N., Chien, S., Nasr, M., Song. S., Terzis, A., and Tramer, F. Membership inference attacks from first prin- ciples. In 2022 IEEE Symposium Security and Privacy (SP), pp. 1897-1914. IEEE, 2022.
Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Liu, Y., Yao, Y., Ton, J.-F., Zhang, X., Cheng, R. G. H., Klochkov, Y., Taufiq, M. F,, and Li, H. Trustworthy llms: a survey and guideline for evaluating large language models' alignment. arXiv preprint arXiv:2308.05377 2023.
Chen, and Yang, D. Unlearn what you want to for- get: Efficient unlearning for llms. arXiv preprint arXiv:2310.20155 2023.
Chen, M., Zhang, Z., Wang, T., Backes, M., Humbert, M., and Zhang, Y. When machine unlearning jeopardizes privacy. Proceedings of the 2021 ACM SIGSAC con- ference on computer and communications security, pp. 896-911, 2021.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05101, 2017.
Maini, P., Feng, Z., Schwarzschild, A., Lipton, Z. C., and
Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., Tsvetkov, Y., Choi, Y., Evans, D., and Hajishirzi, H. Do membership inference attacks work large language models? arXiv preprint arXiv:2402.07841, 2024.
Kolter, J. Z. Tofu: A task of fictitious unlearning for llms.
arXiv preprint arXiv:2401.061 2024.
Mattern, J., Mireshghallah, F,, Jin, Z., Schölkopf, B., Sachan, M., and Kirk-aiik,,,,,ck, Membership infer- ence attacks against language models via neighbourhood comparison. arXiv preprint arXiv:2305.1846,, 2023.
GDPR, G. D. P. R. General data protection regulation. URI.: https:/gdpr-innf. eu/faccessed 2020-11-21), 2018.
O' O'Flaherty, K. Can gpt-4o be trusted with your private data?, 2024. https: htte://wwww /wwww wired. com.
Hayes, J., Shumailov, I., Triantafillou, E., Khalifa, A., and Papernot, N. Inexact unlearning needs more careful eval- uations to avoid a false sense of privacy. arXiv preprint arXiv:2403.00 00000///11/.. 288, 2024.
OpenAL. Chatgpt (mar 14 version) [large language model], 2023. https: //chtt.///h..... opena com/ chat.
10

---

Textual Unlearning Gives False Sense of Unlearning
Pawelczyk, M., Neel, S., and Lakkaraju, H. In-context unlearning: Language models few shot unlearners.
arXiv preprint arXiv:2310.07579, 2023.
Pawelezyk, M., Di, Z., Lu, Y., Kamath, G., Sekhari, A., and Neel, S. Machine unlearning fails remove data poisoning attacks. arXiv preprint arXiv:2406. 17216, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secredly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Rosen, J. The right be forgotten. Stan. L. Rev. Online, 64: 88, 2011.
Shi, W., Lee, J., Huang, Y., Malladi, S., Zhao, J., Holtz- man, A., Liu, D., Zettlemoyer, L., Smith, N. A., and Zhang, C. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint arXiv:2407.0648 240.0666606 2024.
Shokri, R., Stronati, M., Song, C., and Shmatikov, V. Mem- bership inference attacks against machine learning mod- cls. In 2017 IEEE symposium on security and privacy (SP), pp. 3-18. IEEE, 2017.
Shumailov, I., Hayes, J., Triantafillou, E., Ortiz-Jimenez, G., Papernot, N., Jagielski, M., Yona, I., Howard, H., and Bagdasaryan,
Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai. arXiv preprint
2024.
Valinsky, J. Amazon reportedly employs thousands of peo- ple listen to your alexa conversations, 2019, https: //edit t cion., cnn cctm.ccmm/019////11/eeeeee amazon-alese /
iin.enggg/iii.nnnne html.
Yukhymenko, H., Staab, R., Vero, M., and Vechev, M. A synthetic dataset for personal attribute inference. arXiv preprint arXiv:2406.02.17, 2024.
Zhang. J., Das, D., Kamath, G., and Tramèr, F. Membership inference attacks cannot prove that model was trained on your data. arXiv preprint arXiv:2409. 19798. 2024a.
Zhang. R., Lin, L., Bai, Y., and Mci. S. Negative preference optimization: From catastrophic collapse to effective un- learning. arXiv preprint arXiv:2404.000 05868, 2024b.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S.. Dewan, C., Diab, M., Li, X., Lin, X. V., al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. 2022.

---

Textual Unlearning Gives False Sense of Unlcarning
A. Detailed Description and Discussion On Datasets
To ensure rigorous evaluation, we construct two synthetic datasets derived from the SynthPAI dataset (Yukhymenko et al., 2024), enabling fine-tuning on previously "unseen" data. SynthPAI a synthetic dataset primarily designed for evaluating attribute inference attacks. leverages multiple GPT-4-based agents simulating commenters with different attributes (e.g., age, gender, income) communicating with each other, aiming to test whether an adversary can infer these attributes based on the comments. From SynthPAI, we create two binary text classification datasets: SynthPAI-age and SynthPAI-inc. In SynthPAl-age, the labels correspond to age groups (young or old), with young defined as age S. 20 and old as age > 50, resulting in total of 3200 samples. For SynthPAl-ine, we adopt the original SynthPAI partition for income levels (low and high). yielding a total of 2400 samples.
Table Memory reconstruction results on the utilized models with the proposed datasets. ROUGE-1 (R-1), ROUGE-2 (R-2), and ROUGE-L (R-L). And lower ROUGE values indicate the generated texts are less similar to the ground-truth texts.
SynthPAI-age
SynthPAI-ine
Model
R-1
R-2
R-L
Loss
R-1
R-2
R-I.
Loss
Pythia- 1.4b
0.047
0.003
0.046
5.68
0.067
0
0.063
5.92
OPT-1.3b
0.055
0.001
0.051
5.54
0.046
0
0.045
5.69
Since SynthPA/ consists of synthetic samples in fictional scenarios, and GPT-4 was released after the LMs used in our experiments, our proposed datasets maintain considerable rigor. To further validate this, we conduct memory extraction attacks (Carlini et al., 2021) on the models using our proposed datasets. Specifically, we select 100 samples from the datasets and use 50% of a sample as a prompt evaluate the models' ability generate the remaining 50%. As shown in Table 3, the ROUGE values are very low (generated texts are nearly random guesses), indicating that the models have not been trained on the synthetic samples.
B. Training Configurations
We utilize the AdamW optimizer (Loshchilov, 2017) to full-parameter fine-tune the model to obtain Moriginals with the learning rate set to 1--5, the batch size to 64, and the number of training rounds to We carefully prevent the models from overfitting and further provide train accuracy and test accuracy in Table 4.
Table 4. Training accuracy and test accuracy of Moriginal.
SynthPAI-age
SynthPAI-ine
Model
Train ACC
Test ACC
Train ACC
Test ACC
Pythia-1.48
95.21%
85.31%
93.68%
89.06%
1.3b
95.11%
86.85%
93.53%
89.27%
C. Introduction to Unlearning Methods and Configurations
In this part, we present the details of the textual unlearning methods we utilized along with their configurations. All the unlearning methods are implemented via full parlmmarreteer fine-tuning on the target LMs to ensure the effectiveness of erasing.
remove the unlearned samples (Duntearn) from the training set (D) and retrain the original model (Moriginat) on the remaining data (D (Dremien) from scratch, which provides the cleanest erase and is considered as the "gold stan- dard" (Bourtoule al., 2021). The retrain setting is kept the same the original fine-tuning.
Retrain:
Gradient Ascent (GA): minimizes the likelihood of predictions on the unlearned samples (Duntearn) by performing gradient ascent on the cross-entropy loss (the opposite of gradient descent) (Jang et al., 2022; Shi et al., 2024). The unlcarning batch size is set to 32 for SynthPAI-age and SynthPAI-inc datasets, respectively. The unlearning rate is set to le - and the unlearning epoch is 4.
Maximizing Kullback-Leible divergence (KL); maximizes the KI. divergence of the outputs between Moriginat and
12

---

Textual Unlearning Gives False Sense of Unlcarning
unlearned model (Munteara) on D Punlearn (Chen & Yang, 2023). The unlearning batch size set 32 for SyutPAl-agee and SynthPAI-inc datasets, respectively. The unlearning rate set to 1e and the unlearning epoch is 1.
Negative Preference Optimization (NPO): treats D Dunlearn as negative preference data and adjust the offline DPO objective to fine-tune Moriginal such that it assigns low likelihood to Dantearn (Zhang et al., 2024b).
2 E Dunlenen log (-9log Munteorn Moriginal/
]
ENPo(Munteard)) B
where a is the sigmoid function, and / is a hyperparameter that controls the allowed divergence Muntearn from its initialization Moriginal- Following (Shi et al., 2024), we set 3 to 0.1. The unlearning batch size is to for SynthPAl-gge and SynthPA-iinc datasets, respectively. The unlearning rate set to 1 and the unlearning epoch is 7.
performs unlearning in two stages (Ilharco et al. 2022). Moriginal is first overfitted on D unleern to create an intermediary model (Mrein The unlearned model (Muntearn) is then obtained directly by subtracting the weight differences: Munlearn = Moriginal (Mrein force MMorignaa)). The unlearning batch size is set to 32 for SyuthPA/-age and SynthPAI-inc datasets, respectively. The unlearning rate is set to 1e-4 and unlearning epoch is 4.
Task Vectors (TaskVec):
D. Implementation on U-LiRA+ and Additional Results
We default the unlearning ratio 1%, indicating that N 64 48 audited samples required for SynthPA--age and SyuthAA-inn datasets (half unlearn or unseen samples), respectively. Besides, considering that the datasets have an equal number of 0 or 1 labeled samples, we default all the audited samples are selected from 1-labeled samples. To implement our proposed auditing method, we train 100 shadow models for each audited sample, where T = 85 of them are for estimating the distributions of model outputs when the sample is either unlearned or unseen, and 15 of them are acting as tested models MM for computing TPR LowF R evaluating the audited unlearning methods. Ideally, we need fine-tune at least N X T shadow models complete one time of auditing, which too much computational and storage overhead for LMs. Therefore, we adopt the optimization scheme mentioned in the original LiRA-MIA (Carlini al., 2022), requiring only T shadow models, where each model is randomly trained (and then unlearned) on half of the audit samples. Such scheme enables T/2 shadow models for each audit sample to approximate the unlearned and unseen distributions, which significantly reduces the computational overhead while keeping the auditing accuracy as much as possible.
54.62
47.48
TPR@0.1%
29.20
21.01
17:23
17.65
12.18
0.630.63
1.63
1.68
80
66.18
63.45
60.
59.03
1%
50.
30-
20-
15.55
14.08
10.
7.98
0.000.00
0.210.84
Retrain
KL
NPO
TaskVec
Unlearning Method
Audit Mariginal
Audit
U-LIRA+
Baseline
Figure Auditing unlearning methods SynthPAI-age dataset. The lines and represent auditing results on the original models and corresponding unleareed models, respectively. O.T-1.3h model (Top), Pythia-1.44 model (Bottom).
13

---

Textual Unlearning Gives False Sense of Unlearning
E. Additional Results of TULA-MI in Strict Cases
Table TULA-MI MI in strict black- cases.
The presented results are evaluated on the OPT-1.3b model and SyathPAL-age dataset.
N
LossFune
Retrain
GA
KL.
NPO
Task Vec
Confidence
6.55
3.45
5.1
2.7
8.25
32
CrossEntropy
10.35
4.25
5.8
4,4
5.9
Hinge
5.65
1.55
4.6
1.8
8.05
Confidence
6.25
5.6
7.7
5.6
16.55
64
CrossEntropy
25.1
4.15
7.1
6.35
6.95
Hinge
6.9
0.8
6.55
1.75
17.85
Table TULA-MI in strict black-box cases. The presented results evaluated the OPT-1.3b model and SynthPAI-ine dataset.
N
LossFune
Retrain
GA
KL
NPO
Task Vec
Confidence
3.75
2.35
3,5
1.8
4.15
CrossEntropy
8.2
3.1
5,3
2.8
4.55
Hinge
3.8
1.6
3.55
2.35
4.85
Confidence
9.1
3.5
5.65
2.75
10.55
48
CrossEntropy
16.25
4.95
8,4
4.5
8.05
Hinge
7.75
7.4
2.5
11.5
Table TULA-MI in strict black-box cases. The presented results are evaluated on the Pythia- model and SynthPAI-ine dataset.
N
LossFune
Retrain
GA
KI.
NPO
TaskVee
Confidence
4.55
5.65
4.85
4.15
5.65
24
CrossEntropy
7.6
4.55
4.9
6.75
4.55
Hinge
4.4
5.3
3.25
8.8
5.2
Confidence
9.6
9.8
10.55
9.55
9.7
CrossEntropy
14.7
9.5
9.3
14.05
9.5
Hinge
9.3
9.85
7.85
15.85
9.8
F. Additional Results of TULA-MI in Relaxed Cases
14
12.25
12
10
9.97
9.97
7.41
1.14
0.28
0.28
0.570.57
12
9.69
8.55
0.85
1.14
1.14
0.00
0.00
0.28
00
0.00
0.28
Retrain
GA
KL
NPO
TaskVec
Unlearming Method
Figure 6. MIAS in relaxed black-box case SynthPAl-inc dataset. OPT-1.3b model (Top), Pythia- 1.4b model (Bottom).
14

---

Textual Unlearning Gives False Sense of Unlearning
G. Detailed Implementation of TULA-MI in Relaxed Cases
train 100 shadow models to implement the baseline LiRA (Carlini et al., 2022), as referred in Appendix D. Besides, to implement ULeaks (Chen al., 2021), follow its original settings and utilize random forest classifier the attack model, with n.estimators 500 and mni_saaplle__edd
30. We train the attack model with the output features on shadow models, where the logits of unlearned samples M and Moriginal unseen samples are utilized negative features. To implement our proposed TULA-MI, we utilize LightGBM classifier (Ke et al., 2017) as the attack model, with num leaves = 2, learning rate 0.05 and feature fraction = 0.9.
combined positive features, and these of
Algorithm 4 TULA-MI the Relaxed Case
Inputs: original black-box model Marigsaat unlearned black-box model M auiearns larget sample (1r, rraddom sample (a', (9), auxiliary dataset Dauz. learning algorithm A, unlearning algorithm U, number of shadow models T, logit function ø.
D untearn t {), D original + {}
while < ' do D ~ Daux
sample shadow training dataset Moriginal A(DU (x,
Muntearn UMMoriginat (z. y))
lans + (1 y); M)riiiigaa))
(((x, y): Muntearn)
D Puntearn!! [t] f- (Ptr.,, Tur fors [21),1) 1) labeled with 1
D a~ Danx sample another shadow dataset excluding Moriginal A(D')
Munteorn UMM Moriinnat (f),
M(rr)) Mors), Eu + Φ[(2, y): p + o((2, y):
D Danceen [t] + (Pore s luv lore' - 1.0 labeled with O
end while
MAde + train attack model D Puntearn D Dinaccnn lore. t- (f(x, v): Moriginat) Euts +- ((2, u); M Muntearn)
Return MAdo (addo(fe [ore lute lars" (xt-))
H. Evaluations on TULA-DR against Batch Unlearning
In this section, we evaluate the reconstruction performance of the proposed TULA-DR compared batch unlearning, as presented in Tables 8, 9, 10, 11. TULA-DR retains the ability reconstruct certain sensitive information even from batched unlearned texts, particularly when using the Pythia-1.44 1.4b model. Furthermore, the reconstruction accuracy decreases with an increase in batch size (B) This decline is attributed to the expansion of the embedding space, which complicates the search process. Notably, as batch size increases, the R-2 score of the reconstructed text declines more significantly than the R- and R-L scores. This indicates that batch unlearning effectively reduces an adversary's 's ability to reconstruct the semantic structure of a sentence, although keywords and word order can still be partially reconstructed.
Table TULA-DR against batch unlearning (GA).
SyuthPAI-age
SynthPAI-ine
Model
R-1
R-2
R-L
R-1
R-2
R-L
2
32.54
16.66
28.83
39.1
23.28
37.91
4
28.43
9.32
26.31
28.32
7.73
27.62
Pythia-1.4r 1.4b
6
32.27
16.18
30.52
28.35
8.7
26.48
8
18.63
2.95
18.09
24.38
6.61
23.27
2
12.41
0
8.7
19.9
4.76
17.01
4
15..8
0
12.61
20.07
3.42
18.18
OPT-1.3b
17.92
1.62
16.1
16.12
1.72
15.3
8
14.94
2.04
14.14
13.13
1.04
12.63
15

---

Textual Unlearning Gives False Sense of Unlearning
Table 9. TULAA-DR against batch unlearning (KI.).
SyathPAI-age
SynthPAI-ime
Model
B
R-2
R-L
R-1
R-2
R-L
57.93
28.57
48.31
40.23
21.69
40.23
4
33.92
23.82
34.06
30.8
8.61
28.03
Pythia-1.44
6
25.58
9.04
26.07
24.54
4.07
23.92
8
18.85
5.506
16.96
28.03
5.456
20.55
2
12.03
1.85
10.37
15.18
2.38
15.18
4
14.31
0
14.48
15
2.89
14.02
OPT-1.3b
6
11.34
1.38
11.4
13.55
1.66
12.98
14.37
1.44
13.71
15.75
1.5
15.19
Table 10. TULA-DR against batch unlearning (NPO).
SyathPAI-age
SynthPAI-ie
Model
B
R-1
R-2
R-1
R-2
R-L
2
47.91
24
43.15
45.86
7.93
45.86
32.13
9.37
29.32
20.02
5.83
Pythia-1.4h
6
23.02
2.03
20.11
30.85
13.82
31.53
8
29.13
7.29
28.5
24.96
5.76
21.09
2
16.2
4.16
16.01
23.06
4.16
23.06
4
18.65
0
14.73
15.08
2.02
14.16
OPT-1.3b
6
13.19
1.23
11.94
11.78
3.51
12.12
13.57
0.52
12.14
Table 11. TULA-DR against batch unlcarning (TaskVec).
SynthPAl-ago -age
SynthPAI-ine
Model
B
R-1
R-2
R-L
R-1
R-2
R-L
2
24.68
7.4
23.01
44.82
11.31
41.54
27.97
9.72
26.88
26.99
8.05
26.51
Pythia-1.dg 1echia-1.4h
6
23.91
9.92
21.34
27.05
8.73
25.79
8
21.97
7.5
20.25
24.18
10.78
23.02
2
23.28
6.25
23.28
30.45
9.25
28.14
4
19.62
1.04
18.47
16.93
4.79
16.93
OPT-1.3b
6
14.04
2.42
13.22
11.24
1.58
11.29
13.11
0.98
10.76
9.87
1.11
9.17
Ablation Study on TULA-DR
In this section, we perform ablation study TULA-DR to analyze its design components. Four settings compared, representing different configurations to reconstruct the unlearned texts: (Crec) using only cosine distance, (Crec(ly-based) using only Euclidean distance, (Cree Grey) combining cosine distance with Lreg sentence regularization, and (Cree Clip) combining cosine distance with the boundary clipping mechanism.
First, cosine distance demonstrates superior reconstruction performance compared Euclidean distance. This advantage arises because cosine distance captures directional similarity in high-dimensional model weight vectors more effectively, facilitating better guidance for reconstructing unlearned texts, whereas Euclidean distance is harder optimize in such cases. Furthermore, incorporating freg sentence regularization significantly enhances reconstruction accuracy by providing additional statistical information (mean embedding values), which helps the reconstructed embeddings align more closely with the correct distribution. Lastly, adding the boundary clipping mechanism also improves reconstruction effectiveness. By calibrating the value domain of reconstructed embeddings during optimization, this mechanism accelerates convergence and prevents falling into local optima.
16

---

Textual Unlearning Gives False Sense of Unlearning
Table 12. Ablation study on TULA-DR against GA.
SynthPAl-aee
SynthPAI-nne
Model
Setting
R-2
R-L
R-1
R-2
R-L
Crec
66,66
59.25
29.76
4.76
29.76
braael)
0
0
Pythia-1.4b
Lrec + Greg
80.15
36,66
70.63
66.31
34.12
62.97
Lree + Clip
92.59
62.5
84.12
80.55
60.11
76.38
Pree breeel--bsed))
6,061
0
6.061
16.36
0
16.36
11.57
0
11.57
5.55
0
5.55
OPT-1.3b
Cree + Lreg
51.54
12.96
44.04
40.74
13.69
33.33
Cree + Clip
54.49
23.61
46.03
66.55
37.61
62.39
Table 13. Ablation study on TULA-DR against KL.
PynthPAl-age
SynthPA-ine
Model
Setting
R-1
R-2
R-L
R-1
R-2
R-L.
Crec
61.11
17.5
57.4
45.83
10.31
45.83
0
0
0
Pythia-I.4b
Cree + Greg
90.27
58.09
81.94
73.14
52.61
68.98
Crea + Clip
79.16
52.38
79.16
88.42
48.21
77.31
Lrec Cree((z-based)
7.4
9.39
0
9.39
6.36
0
6.36
9.97
0
9.97
OPT-1.3b
Cree + Greg
14.94
0
11.24
9.97
0
9.97
Cree + Clip
9.52
5.55
9.52
13.42
0
13.42
Table 14. Ablation study on TULA-DE DR against NPO.
SynthPAl-age
SynthPAI-in inc
Model
Setting
R-1
R-2
R-L
R-1
R-2
R-L
Lree
47.4
12.5
47.4
44.16
15.87
44.16
0
0
0
0
0
Pythia-I.4b
Cree + Creg
63.75
41.46
56.34
55.35
23.81
55.35
Crea + Clip
71.9
88.88
70.56
38.69
70.56
Crec Crealla-based
17.18
3.33
14.15
3.7
0
3.7
3.33
9.09
0
9.09
OPT-1.3b
Crea + Lreg
20.87
0
17.17
35.83
9.52
35.83
Lree + Clip
60.74
31.94
49.63
61.11
28.7
50.37
Table 15. Ablation study on TULA-DR DR against TaskVec.
SynthPAl-age
SynthPA--ine
Model
Setting
R-1
R-2
R-L
R-1
R-2
R-L
Lrec
60.18
22.61
56.01
34.72
9.52
34,72
Crea(lz-base))
0
0
0
0
0
Pythia-I.4b
Cree + Creg
67.46
24.44
57.14
79.16
44.04
71.29
Cree + Clip
91.07
68.25
86.31
53.24
36.31
53.24
Crec brasel)-bsed)
12.03
0
7.87
20
3.7
20
12.5
12.5
5.55
0
5.55
OPT-1.3b
Cree + Creg
33.59
33.59
37.56
11.11
33.86
Lree + Clip
49,93
32,93
49.93
63.78
26.66
56.37
17

---

Textual Unlearning Gives False Sense of Unlearning
J. Examples of the Reconstructed Texts via TULA-DR
We list some reconstructed examples in Tab. 16, 17, 18 and Tab. 19, 20, 21, for SyuthPAl-age and SyuhPAl-inc datasets respectively. Besides, fairly evaluate the reconstruction attack, we additionally add padding characters (
 for OPT-1.3b, jendoftext] > for Pythia-I.4b fix the length of unlearned texts.
Table 16, Reconstructed examples SynthPAl-age dataset.
Reference text:
timelines? <padding <padding> timeliiieeeeesss more like stress recipes.
Model
Method
Reconstructed Texts
GA
|ndoftext]
> nowadays< |endoftext] >strict timelines? more like stress
KI.
|endoftext] Susi morestrict timelines? like
Pythia-1.45 1.4b
NPO
lendoftext]
>< |endoftext] >< [endoftext] > |endoftext] >< |endoftext] >-strictelines. like stress
Vac
lendoftext]
>< |endoftex(] >< endoftext| >strict timelines? more like stress
GA
like timelines"Moro liko-</>>sst</// stress stressa
KL
<ike

> timeframe at plaster<//s> Stress stressful
OPT-1.3b
NPO
like stricter siettteiiiinns<<>>>>>>>>>>>>/ lingurict? Like stressa
Ves
like 
 timelines stress coachMore?) timelinesmare Likea
Table 17. Reconstructed examples on SynthPAI-age dataset.
Reference text: Somndding><ppdddig>>>BBvaa inevitable; historic sites always layers deep!
Model
Method
Reconstructed Texts
GA
lendoftext]
><
lendoftext
>Evolution inevitable; historic sites always layers
KL
Ev< |endoftext]
>< endoftext >olution; historic sites layers always layers
Pythia-1.4b
NPO
Ev< lendoftext| >Evolution sites inevitable; always historic layers
Task Vec
necess< [endoftext
> Evolution inevitable; historic sites layers deep
GA
likeolution; sites always layers deep horse Hogwartsa
KL
like</>>> Ramos Craw</ss fossils favourite aFCT ONLY Breakinga
OPT-1.3b
NPO
like;historic sites perenn always just layers deep menstruossusa
Task Vee
likeEvolution inevitable; feudal historic historic sites always layersa
Table 18. Reconstructed examples on SynthPAl-age dataset.
Reference text: <padding <padding>Engineerring sure throws curveballs despite hefty spreadsheets!
Model
Method
Reconstructed Texts
GA
Engincering sure throwshalls despite hefty spreadshe
KL
Engineering spread sure throws curvehalls despite hefty
Pythia-1.4b
NPO
Engineering sure throwsballs despite hefty spreadshe
Task Vec
Engineering sure throwsballs despite hefty spreadshe
GA
likeEngineering sure throws curveballs hearty despite heftyshectsa
KL
like<//>> 
 #нннннн: tocsales lords Tha
OPT-113b
NPO
like grotesque IsundepiieooggEEngieeringg sure sly threwsheetsa
Task
likeEngineering throws curveballs despite hefty spreadsefffiiccheettaa
18

---

Textual Unlearning Gives False Sense of Unlearning
Table Reconstructed examples on SyuthPAI-ine dataset. Reference text: <pudding><pudddnng school grind pays off here editing gigs underway.
Model
Method
Reconstructed Texts
GA
pays off here< [endoftext] >film school here editing gig
KL
< lendoftext]
>< [endoftext] >film school pays off here editing gig
Pythia-I.4b
NPO
pays< lendoftext
> lendoftext] >film school here pays editing gig
Task
lendoftext] >ilm school grind pays off here editing gig
GA
like
 grind here<//>> editing cryptocssl gigs gigsa
KL
like eagerbrekkerkktoonliissmaeyyys 
> documentation adversity funds majoritya
OPT-1.3b
NPO
like editing Filmitismm Izanfi gigs editing pledginga
Task
like 
 film school fishes gigs editing gigs ALECa
Table 20. Reconstructed examples SyuthPAl-inc dataset.
Reference text: <padding>rent growth happens inside too gotta both be evolving
Model
Method
Reconstructed Texts
GA
realrealFrame growth both gotta be evolving
KL
real growth gotta both evolving
Pythia-I.we 1.4b
NPO
real headed growth happens gotta be both evolving
Task
real to growth too both gotta be evolving
GA
likereal growth inside shenan egregious gotta both evolving evolvinga
KL
like growers dope antid QI activ evolve Damascus Yin Lyme Vermonta
OPT-1.3b
NPO
likereal gotta evolvingrenches Yep Krugutical mysql Okawarua
Task
likereal growth happensburghreal gotta both be evolvinga
Table 21. Reconstructed examples SynthPAI-inc dataset.
Reference text:
<padding><ppddiinnnn
tape fix countertop once lol Reconstructed Texts
Model
Method
GA
lendoftext
>< |endoftext]
>had tape fix' countertop tape once
KL
lenooftex]
Veter< [endoftext]
>had tape counter countertop once
Pythia-1.4b
NPO
lendoftext
lenoffeex|
endoftext
[endoftext] >had tape tape countertop once
Task
lendoftext
<< lendoftext] >
endoofiext] > countertop fix< lendoftext >aad tape once
GA
flec//sse//shhddccsSCCnnnettt tape fix Latvia oncea
KL
Once tapea
OPT-1.3b
NPO
like///>1mplc//><44>>---44 tape fixtop oncea
Task
like
<</>><<>>><<<>>> tape fixtop tape oncea
19


Reviews:
[{"review_id": "ftX9ND09lp", "reviewer": "Reviewer_p1Cm", "summary": "The paper proposes a new auditing method to check whether unlearning text from a model is completely unlearned. The auditing method called U-LiRA+ is based on U-LiRA and checks whether it is possible to differentiate between unlearned and not seen samples. Additionally two methods for investigating privacy risks for unlearned models are presented called TULA-MI and TULA-DR. TULA-MI is testing whether an adversary can tell that a specific datapoint was unlearned while TULA-DR tries to reconstruc the data point after unlearning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer p1Cm\nWe sincerely thank the reviewer p1Cm for the valuable and constructive feedback! \n## Q1: Clear the differences between the proposed TULA and [1].\nHere are the key differences:\n\n(1) **Broader and more realistic assumptions.** ***[1] considers only the relaxed black-box scenario***, where the adversary performs membership inference attacks (MIA) with access to output logits, model architecture and auxiliary dataset. In contrast, ***our proposed TULA could be further applied to a strict black-box scenario***, where the adversary can only obtain output scores (e.g., confidence values), ***providing more practical insights on real-world risks.***\n\n(2) **More powerful attack paradigm.** For the target sample *x*, [1] trains the attack model using logits from ***randomly sampled unseen samples as negative supervision***, resulting in a ***coarse-grained, average-level MIA***. In contrast, we ***train multiple shadow models to accurately learn the logits when *x* is unlearned or unseen***, enabling ***fine-grained, sample-level MIA***. \n\n(3) **Target beyond MIA.** While [1] focuses solely on MIA, we propose both ***MIA*** and ***data reconstruction attack*** against unlearning mechanisms, exploring the risks from ***adversaries with different targets***.\n\n(4) **Focus on more challenging setting**. [1] focus only on tabular and image data with ML models or CNN—***traditional setups where MIAs are much easier due to high distinctiveness among samples***. However, ***MIAs on textual data are much more challenging, necessitating a stronger attack*** [3]. Our results show that [1] is much ineffective against textual data and modern language models.\n## Q2: Clear the differences between the proposed U-LiRA+ and [2].\nCompared to [2], we **explicitly construct and inject a rigorous audit set with *mislabeled samples* before unlearning, ensuring a rigorous auditing in the worst case**. We demonstrate that previous auditing methods, including [2], **lack rigor due to their failure to properly select audited data**, resulting in **existing unlearning methods being significantly overestimated**.\n## Q3: How does the adversary know when unlearning has occurred, and have access to model weights?\n(1) **The occurrence of unlearning could be easily detected in practice**. **An adversary can *continuously* query the model with target samples**. If the outputs from two consecutive sets of queries **change**, *the adversary could execute the attack to infer whether any samples have been unlearned*.\n\n(2) **The access to model weights could also be realistic**. The adversary could be a **malicious collaborator**, such as *a company contracting with the model developer, with permission to model weights for local deployment*. According to the unlearning policy, **the collaborator’s model is also required to be updated after unlearning**, enabling the access to model weights before and after unlearning. Moreover, *this assumption is widely considered in literature*[4], serving as **an \"upper bound\" for exploring real-world security risks**.\n## Q4: Why do we focus on text classification tasks?\n**Text classification tasks enabling us to design *rigorous* audit samples for *rigorous* unlearning auditing**. Specifically, **rigorous auditing requires rigorous audit samples, i.e., *most vulnerable samples* to unlearning, enabling the worst-case auditing**. The utilized ***mislabeled samples***  have found to be most vulnerable to privacy leakage on image classification tasks[5]. Although our approach is **task-agnostic**, **rigorous samples for text generation tasks are currently under-explored**.\n## Q5: The Confusions on Figure 2.\nWe would like to explain Figure 2 based on the process of unlearning auditing: The audit set consists of training and unseen samples. For **one original model**, **multiple unlearned models** are derived by applying *different unlearning methods* to the training samples in the audit set. MIA-based auditing methods are expected to exhibit high attack accuracy on the **original model** (**dashed lines in Fig. 2**) and low accuracy on **unlearned models** (**bars in Fig. 2**).\n## Q6: The Meaning of Metric \"NTS@1NFS\".\nNTS@1NFS refers to *the Number of True Samples @ 1 False Sample*, quantifying **the number of correctly inferred samples with only one error** for MIAs.\n## Q7: The occurrence of \"<\\/s>\" before the synthetic texts.\nTo fairly evaluate the reconstruction attack, we additionally add padding characters (<\\/s>) to fix the length of unlearned texts.\n## References\n[1] When Machine Unlearning Jeopardizes Privacy, CCS 2021\n\n[2] Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy, preprint 2024\n\n[3] Do Membership Inference Attacks Work on Large Language Models?, COLM 2024\n\n[4] Adversarial Machine Unlearning Requests Destroy Model Accuracy, ICLR 2025\n\n[5] Evaluations of Machine Learning Privacy Defenses are Misleading, CCS 2024", "claims_and_evidence": "The claims are empirically supported. However, the presentation of the experimental results could be clearer (see comment regarding Figure 2).", "methods_and_evaluation_criteria": "The experimental design seems to make sense, even though the proposed approaches are only tested on two synthetic datasets.", "theoretical_claims": "There are no theoretical claims.", "experimental_designs_or_analyses": "The experimental design seems to be sound.", "relation_to_broader_scientific_literature": "The findings are not really surprising. Previous work (Hayes et al.) has already shown that unlearning on text needs better evaluation methods.", "essential_references_not_discussed": "- I am not quite sure what the difference between TULA and the work of Chen et al. [1] is. This should be thoroughly discussed, as they also have proposed a membership inference attack that is able to tell whether a data point was unlearned or not.\n- It should be made clearer what the difference between U-LiRA+ and U-LiRA from Hayes et al. [2] is.\n\n[1] Chen et al., When machine unlearning jeopardizes privacy, CCS 2021\n[2] Hayes et al, Inexact unlearning needs more careful evaluations to avoid a false sense of privacy, preprint 2024", "other_strengths_and_weaknesses": "Weaknesses:\n- in my opinion the argument in TULA that users have access to the models before and after unlearning is a bit unrealistic to me. After all the users don't know when the model will be updated and they don't have access to the model weights either.\n- it is a bit confusing that the paper talks about language models, but in the end only looks at sentiment analysis models aka. models that classify input text.", "other_comments_or_suggestions": "- Line 143 right side: there is a point after \"M_original\" that should probably not be there\n- it is not possible to highlight any text in the pdf with the mouse or click on any links. Please fix!\n- Line 304 left side: two times \"However,\"\n- Line 320 right side: \"fine-turned\" should be fine-tuned\n- Figure 2: in the legend for the blue dotted line it says \"Audit M_original\", but I think this should be \"Audit M_unlearned\". At least that is what the caption suggests.\n- Line 412 right side: R-3 is used, but this metric is not used in the table", "questions_for_authors": "1. It is not clear to me what the dashes lines in Figure 2 are.  What does audit on the original and the unlearned model mean? Why is there only a single blue line, even if there are multiple unlearned models?\n2. What is the metric \"NTS@1NFS\"? This should be explained or written out at least once.\n3. Why are there so many \"
\" before the synthetic texts?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "There are no ethical reviews needed for this paper."}}, {"review_id": "ui03C1q7iW", "reviewer": "Reviewer_c3T7", "summary": "The authors demonstrate that the textual unlearning mechanism can not ensure privacy as expected. They propose a rigorous unlearning auditing method (U-LiRA+) and and investigate privacy attacks in both black-box and white-box scenarios. Through empirical evaluations on large language models and synthetic datasets, the authors reveal that existing textual unlearning methods fail to completely erase target texts. Furthermore, these methods may inadvertently expose additional information about unlearned texts through membership inference attacks (MIA) or data reconstruction attacks (DRA).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer c3T7\nWe sincerely thank the reviewer c3T7 for your valuable and constructive feedback!\n## Q1: Is it possible for the proposed U-LiRA+ to be adopted to other MIAs?\nYes, our approach is **MIA-agnostic** (membership inference attack), as its core idea is to **properly construct and inject rigorous audit samples before auditing**. For an unlearning auditing task, we start by defining a set of **the most vulnerable samples** (e.g., mislabeled samples) as **audit samples** on that task. Use half of them as training samples and half as unseen samples. After unlearning the training samples with the target unlearning method, we can utilize **any kind of MIA** to test whether it is possible to successfully distinguish between the training samples and the unseen samples in the audit set.\n\n## Q2: For the TULA-DR, how do the authors initialize the candidates?\nWe randomly initialized the candidate embeddings with Gaussian distribution. Additionally, it is also acceptable to utilize other approaches for initialization, such as a random-selected sentence's embeddings or uniform distribution. Empirically, the key insight is that *the values of the initialized embeddings are preferably distributed within the value domain of normal embeddings*.\n\n## Q3: What about other ways of calculating the difference for the TULA-MI in the strict black-box case?\nThank you for this insightful question. Indeed, we used the value difference (i.e., A-B) to capture the loss changes on the model before and after unlearning the target sample. However, **other approaches such as ratio and logarithm are also acceptable**, as long as there is eventually a value to quantify the loss change.\n\n## Q4: Why do mislabeled samples represent a \"worst-case scenario\" for auditing?\nCompared to normal samples, **mislabeled samples** are a small number of counterfactual samples, ensuring the model cannot generalize to them unless explicitly trained. As a result, the trained and unseen **mislabeled samples** will be very different in output distributions and thus are the *most vulnerable to unlearning auditing*. Therefore, we inject the **mislabeled samples** into the training set as the audit samples, in order to simulate the worst-case auditing.", "claims_and_evidence": "The claims are well-supported.", "methods_and_evaluation_criteria": "The proposed methods and evaluations are sound, but there are several minor issues.\n1. I am confused about how to initialize the candidates for the proposed TULA-DR.\n2. It would be better to try more metrics to evaluate the difference between loss values to implement the proposed MIA.", "theoretical_claims": "The work does not include any theoretical proofs.", "experimental_designs_or_analyses": "The experimental evaluations effectively validate the performance of the proposed auditing method and the attacks in the paper.", "relation_to_broader_scientific_literature": "This work shows that even in the strict black-box querying and exact unlearning scenario, the unlearning mechanism can still compromise the privacy of unlearned data. This provides new insights into the secure deployment of unlearning mechanisms.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Weaknesses：\n1. Considering that LiRA may be time-consuming in some cases, is it possible for the proposed U-LiRA+ to be adopted to other MIAs?\n2. For the TULA-DR, how do the authors initialize the candidates?\n3. For the TULA-MI in strict black-box case, the authors utilize the difference in loss values to implement the proposed MIA. Considering that the loss changes may be non-linear, what about other ways of calculating the difference?", "other_comments_or_suggestions": "There are some suggestions:\n1) Algorithm 2 appears crowded and the authors could improve its clarity; \n2) It would be helpful if the authors could color-code the reconstructed texts in Tables 16-21 to better differentiate correct and incorrect results; \n3) It appears that Fig.3 misses the “%”.\n4) The introduction of U-LiRA+ is compelling, but consider adding a brief explanation of why mislabeled samples represent a \"worst-case scenario\" for auditing. This would improve the clarity.\n5) The paper lacks a description on compute requirements (e.g., GPU, CUDA).", "questions_for_authors": "N/A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GjAQcYCZWa", "reviewer": "Reviewer_tLdQ", "summary": "The authors demonstrate that current unlearning methods fail to adequately protect the privacy of unlearned texts in language models. To address this, they propose a robust unlearning auditing method, U-LiRA+, which utilizes membership inference attacks and deliberately introduces mislabeled samples to reveal that the unlearned texts are, in fact, highly detectable. Furthermore, they introduce textual unlearning leakage attacks, showing that unlearned texts can be inferred within unlearning systems, thereby uncovering a new privacy risk associated with machine unlearning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer tLdQ\nWe sincerely thank the reviewer tLdQ for your valuable and constructive feedback!\n\n## Q1: How to determine the convergence of TULA-DR?\nOur proposed TULA-DR is an **optimization-based attack**. Empirically, the optimized candidates converge gradually as the number of iterations increases. A criterion is that the loss of the attack **stabilizes at a lower value and no longer continues to decrease**. Therefore, when implementing the attack, we could **set a maximum number of iterations to stop the optimization process**.\n\n\n## Q2: What does the “Decoding” function in Algorithm 3 refer to?\n\nThis function refers to transforming the reconstructed embeddings into **text space**. To implement this function, we use the model’s **embedding layer** as a *“vocabulary”* and apply a similarity computation function (e.g., *cosine similarity*) to **match the ordinal indices of the reconstructed embeddings**. The resulting list of indices is then converted into readable text using a **tokenizer**.\n\n## Q3: Can U-LiRA+ potentially be applied to generation tasks for future work?\n\nOur approach is **task-agnostic**, as its core idea is to properly construct and inject rigorous audit samples before auditing. Specifically, in order to implement rigorous unlearning auditing on a text generation task, we should first **define the samples that are most vulnerable to unlearning on that task as audit samples**. We then **inject these audit samples into the training set** and evaluate whether the target unlearning method could fully erase them. In addition, to implement U-LiRA+ on the generation task, we could **utilize the next-word probability vector in response to a text sample as the model's *output* for MIA**.\n\n## Q4: In-depth analysis on the performance differences of existing methods across different data modalities.\nThis study focuses on textual data and language models. However, our key findings could be broadly applicable to various data modalities:\n\n(1) Existing unlearning methods can not fully erase target samples, primarily due to **their inability to accurately identify parameters linked to target knowledge**. This limitation could extend to other data modalities.\n\n(2) Analyzing models before and after unlearning allows an adversary to infer membership information about unlearned samples or even reconstruct them. **While different data modalities may exhibit varying degrees of privacy leakage, the underlying risk persists**.\n\n## Q5: The suggestion for quantitatively analyzing the information residuals of unlearned models.\nWe thank you for your insightful suggestion, and it is indeed a very interesting perspective.\n\nHowever, our proposed auditing method is currently sufficient to demonstrate that **existing unlearning methods can not completely erase the target samples and result in large information residuals**. With these findings, we hope to call for the development of more accurate unlearning methods before applying them in practice. Indeed, with the development of unlearning techniques, how to effectively and exactly measure **“how much”** information residual with theoretically guarantee will be an important problem. We thank you for your inspiration and would be happy to explore this question in future research.\n\n## Q6: The concern on \"high-confidence detection\" of our proposed U-LiRA+.\nThe reason for our claim is that we utilize the TPR@LowFPR metric. TPR@LowFPR is the most commonly used and rigorous metric for evaluating MIA[1], as it quantifies an attack's effectiveness and **confidence** by measuring **the proportion of correctly inferred samples at a low false rate**.\n## References\n[1] Membership Inference Attacks From First Principles, IEEE S&P 2022", "claims_and_evidence": "The contributions are well supported with sound designs and evaluations.", "methods_and_evaluation_criteria": "I have some confusions about TULA-DR that need further clarification:(1) How to determine the convergence of TULA-DR? The authors need to further clarify the metrics or criteria for assessing convergence. (2) In Algorithm 3, the authors should add a definition of “Decoding” function.", "theoretical_claims": "The authors did not present any theoretical proofs.", "experimental_designs_or_analyses": "The experimental design is sound, and the results empirically demonstrate the effectiveness of the proposed methods.", "relation_to_broader_scientific_literature": "The authors rethink the security of unlearning mechanisms on language models, revealing the vulnerabilities of existing methods from auditing to deployment.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "No.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. How to determine the convergence of TULA-DR? What does the “Decoding” function in Algorithm 3 refer to? While this work focuses on classification tasks, I am curious that could the idea of U-LiRA+ potentially be applied to generation tasks for future works?\n\n2. Although the paper highlights the limitations of text unlearning methods, it lacks an in-depth analysis of the performance differences of existing methods across different data modalities, resulting in its innovation being primarily confined to phenomenological descriptions.  \n\n3. While the paper identifies privacy leakage risks in the text unlearning process, it does not employ quantitative analyses based on information entropy or mutual information to demonstrate that the unlearned model retains exploitable information residues. Incorporating information-theoretic approaches, such as computing mutual information between model outputs and the original text or analyzing entropy variations, would provide a more rigorous validation of the inevitability of privacy risks.  \n\n4. The paper mentions \"high-confidence detection\" but does not explicitly define a statistical significance threshold, which may affect the robustness of its conclusions. For instance, failing to adopt a stringent significance level in hypothesis testing could lead to an increased false positive rate.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UHeazAHyEh", "reviewer": "Reviewer_9rYC", "summary": "This paper critically demonstrates that current machine unlearning mechanisms give a false sense of effective unlearning. First, they propose U-LiRA+, a rigorous textual unlearning auditing method, and find that the unlearned texts can still be detected with very high confidence after the unlearning process. Further, they comprehensively investigate the privacy risks of textual unlearning mechanisms in deployment. By proposing TULA along with its variants in both black- and white-box scenarios, the authors critically reveal that the unlearning mechanism would instead expose more about the unlearned texts, facilitating both membership inference and reconstruction attacks. The experiments demonstrate that the proposed auditing method much more strictly measures the unlearning effectiveness compared to previous approaches. Besides, the proposed TULA-MI and TULA-DR attacks are capable of inferring unlearned samples with high confidence. \n\nOverall, in order to explore the vulnerabilities and privacy risks of machine unlearning in language models, the authors introduce a rigorous auditing method and novel attack paradigms. The experiments support the findings well. In addition, this paper is structured well and easy to follow.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 9rYC\nWe sincerely thank the reviewer 9rYC for your valuable and constructive feedback!\n\n## Q1: How about the proposed reconstruction attack (TULA-DR) against the exact unlearning method (including retraining)?\n\nWe mainly focus on TULA-DR against inexact unlearning because: (1) **Exact unlearning is relatively safe**, as its core idea is to delete the target sample and retrain the model from scratch. Due to the randomness in the training process, the difference in model weights before and after unlearning is less likely to be approximated by the adversary. (2) In practical deployment, **developers often use the inexact unlearning methods considering computational efficiency**. Thus, exploring reconstruction attacks against inexact unlearning could provide more practical insights.\n\n## Q2: How do the findings inspire the development of unlearning mechanism on language models?\n\nThanks for the insightful question. We hope our findings could inspire future research in the following three aspects:\n\n(1) **Developing more precise unlearning mechanisms**. Considering the difficulty of existing unlearning methods to completely erase target samples, future work could explore *efficient exact unlearning methods* (e.g., retraining) or *certified inexact approaches* capable of accurately identifying target knowledge. \n\n(2) **Strengthening unlearning auditing before real-world deployment**. Future research should establish rigorous auditing frameworks for unlearning mechanisms *from broader perspectives*. Beyond verifying whether an unlearned model successfully erases target data, it is also essential to assess and mitigate potential risks, such as reduced robustness.\n\n(3) **Rethinking the secure deployment of unlearning mechanisms**. While prior research has primarily focused on *how to unlearn*, less attention has been given to *how to deploy unlearning*. As demonstrated in this work, future research should pay more attention to the *secure deployment of unlearning mechanisms* into real-world scenarios while mitigating potential new risks.", "claims_and_evidence": "Yes, the claims are clearly supported in the paper.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense for the problem.", "theoretical_claims": "There are no theoretical evaluations in this paper.", "experimental_designs_or_analyses": "Yes, the experimental designs and analyses effectively support the proposed methods.", "relation_to_broader_scientific_literature": "1. This work highlights that previous unlearning auditing approaches have overestimated the effectiveness of existing unlearning techniques. The authors propose a novel and rigorous auditing method, U-LiRA+, which could inspire the development of more thorough and reliable unlearning techniques.\n2. The work reveals the privacy risks of deploying textual unlearning mechanisms in both black-box and white-box contexts. This serves as a call to further investigate the potential new and additional privacy risks before widespread application of unlearning mechanisms.", "essential_references_not_discussed": "Related works are adequately discussed.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "It would be more informative for the authors to further discuss how your findings inspire the development of unlearning mechanism on language models.", "questions_for_authors": "1. How about the proposed reconstruction attack (TULA-DR) against the exact unlearning method (including retraining) ?\n2. How do your findings inspire the development of unlearning mechanism on language models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "vdPBB7QwEo", "reviewer": "Reviewer_PnWC", "summary": "The paper investigates the effectiveness of machine unlearning (MU) in LMs and introduces new auditing and attacking methods to evaluate its reliability and privacy risks. They propose U-LiRA+, which uses mislabeled samples to rigorously audit unlearning effectiveness. The results reveal that over 70% of unlearned texts remain detectable. The TULA-MI shows that attackers can infer whether a text was unlearned by comparing model outputs before and after unlearning, even in strict black-box scenarios. The TULA-DR exploits model weight differences to reconstruct unlearned texts with over 80% accuracy in some cases, showing that different unlearning methods leave distinct traces in the model. The findings demonstrate that existing unlearning methods do not ensure true erasure and may even increase privacy risks, highlighting the need for more robust and secure unlearning mechanisms.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer PnWC\nWe sincerely thank the reviewer PnWC for your valuable and constructive feedback!\n## Q1: Concerns about our assumptions and their practicality.\nWe would like to provide further explanations on our assumptions:\n\n(1) In the **black-box scenario**, we assume that an adversary can *query the model* with target samples and *utilize the outputs to infer their membership*. This is **the most fundamental assumption for Member Inference Attack (MIA)**[1] and is **widely considered in literature as well as real-world applications**. To further align with real-world scenarios, we further investigate **the strictest black-box scenarios**, where the adversary can *only obtain output scores* (e.g., confidence scores) for MIA.\n\n(2) In the **white-box scenario**, we assume that adversary has *access to the model weights before and after unlearning*. This assumption could also be realistic in practice. Here, the adversary could be a **malicious collaborator**, such as *a company contracting with the model developer, with access to model weights for local deployment*. According to the unlearning policy, **the collaborator’s model is also required to be updated after unlearning**, enabling the adversary to have the model weights before and after unlearning. Additionally, this assumption is **commonly studied in literature[2][3]**, serving as **an \"upper bound\" for exploring real-world risks**.\n## Q2: Suggestions on utilizing real-world data.\nWe utilize synthetic datasets to rigorously ensure that **the pre-trained models have not seen these samples during the pre-training phase**. However, *we are concerned that real-world data can not meet our requirement currently*: (1) Pre-trained models are trained on *vast amounts of real-world data*, which is typically **closed-sourced**. (2) Recent studies indicate that **we can not yet robustly detect whether a specific sample was used among massive pre-training data[4]**.\n\nTherefore, we utilize synthetic samples, which are **entirely based on fictional scenarios**, and generated by **GPT-4-based agents released after the models we utilized**. Through these efforts, we hope to ensure that the setups are rigorous and thus avoid being potentially misleading.\n## Q3: The concern on computational cost (training multiple shadow models) on the proposed U-LiRA+ for LLM.\nWe train multiple shadow models to **ensure the rigorous auditing on unlearning methods**. With them, we could *closely approximate the distributions of the model outputs* and *rigorously determine samples' membership through statistical tests*.\n\nThough this approach could incur much computational cost, **it could also be applied to LLMs**: (1) We could utilize **Parameter-Efficient Fine-Tuning (PEFT)** methods to effectively fine-tune and unlearn few audit samples. Since only few parameters are trainable, the computational cost would be significantly reduced. (2) **Our approach is MIA-agnostic**, as its core idea is to properly construct and inject rigorous audit samples before unlearning. **If more efficient yet equally rigorous MIAs for LLMs emerge, our approach can be flexibly adapted to them.**\n## Q4: Do we choose the models mainly for computational efficiency? \n**The selection of *1.5B*-parameter models balances *practicality* and *computational efficiency***: (1) **Lightweight LLMs are widely adopted in practice**, such as *autonomous driving* and *mobile AI assistants*. (2) As mentioned, **we aim to ensure rigorous auditing and evaluation**, which requires some consuming operations, e.g., *training multiple shadow models*. Choosing an appropriately sized model enables us to **conduct broader evaluations efficiently**.\n\nFurthermore, we highlight that **the model scale is unlikely to compromise our findings**: (1) We demonstrate that existing auditing methods are not rigorous because *they fail to correctly select audit samples*, **which is independent of the target model**. (2) We reveal that current unlearning methods fail to completely erase target samples because *they cannot accurately identify and remove parameters encoding target knowledge from the large parameter space*, **which would persist across models of different scales.**\n## Q5: The choice of TPR@LowFPR.\nTPR@LowFPR is the **most commonly used and rigorous metric** for evaluating MIA, quantifying MIA's effectiveness and confidence by measuring **the proportion of correctly inferred samples at a low false rate**[5].\n## References\n[1] Membership inference attacks against machine learning models, IEEE S&P 2017\n\n[2] Machine Unlearning Enables Camouflaged Poisoning Attacks, NeurIPS 2023\n\n[3] Adversarial Machine Unlearning Requests Destroy Model Accuracy, ICLR 2025\n\n[4] Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data, IEEE SaTML 2025\n\n[5] Membership Inference Attacks From First Principles, IEEE S&P 2022", "claims_and_evidence": "The claims are mostly well-supported by empirical evidence. Some claims could benefit from additional validation. However, both TULA-MI and TULA-DR assume that an attacker can access outputs (or even internal weights in the white-box case) from the model before and after unlearning. In practice, such access might be limited, so the generalization of the claims is concerned.", "methods_and_evaluation_criteria": "- The choice of using synthetic data may undermine the claims. I understand the concern that models might have seen the public data. However, the assumptions of this paper are strong (e.g., access to the model weights), which is unlikely to happen in real-world scenarios; it is better to test the model on real-world datasets. There are many other real-world datasets for MU and existing methods to test whether certain data are used to train the model, which could help to construct new datasets if the authors want a pure dataset for unlearning.\n- The authors trained 100 shadow models for each shadow example. This can be very time-consuming when their method is extended to commonly used LLMs today. The only imaginable setting I can think of is when a new unlearning method comes out and then it is applied to a smaller model to see if leakage happens under attack. However, even if leakage does not exist, the effectiveness of the new methods remains unknown.", "theoretical_claims": "I am not aware of any proof for theoretical claims.", "experimental_designs_or_analyses": "The choice of using TPR@LowFPR is not justified. Not sure if this metric is used by other work.", "relation_to_broader_scientific_literature": "This paper challenges prior claims that inexact MU approximates full retraining. They show that unlearning leaves residual traces rather than ensuring true forgetting. The paper contributes to the broader scientific literature by calling for more robust privacy-preserving techniques.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Please refer to the methods section for the weaknesses.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Is the choice of Pythia and OPT mainly based on computational efficiency?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a rigorous auditing method for unlearning to demonstrate that current textual unlearning mechanisms cannot guarantee privacy as expected. In addition, two methods, i.e., TULA-MI and TULA-DR, are introduced to investigate the privacy risks of unlearned models. However, using synthetic data in the experiments may raise concerns about the practicality of the proposed approach. After reading the authors' response, I believe they can clearly explain the reason for using synthetic samples based on fictional scenarios in the camera-ready version. Therefore, I recommend acceptance of the paper.
Decision: Accept (poster)