Unlearning Personal Data from a Single Image
Authors: Thomas De Min
Massimiliano Mancini
Stéphane Lathuilière
Subhankar Roy
Elisa Ricci
Venue: Accepted by TMLR ()

Abstract: Machine unlearning aims to erase data from a model as if the latter never saw them during training. While existing approaches unlearn information from complete or partial access to the training data, this access can be limited over time due to privacy regulations. Currently, no setting or benchmark exists to probe the effectiveness of unlearning methods in such scenarios. To fill this gap, we propose a novel task we call One-Shot Unlearning of Personal Identities (1-SHUI) that evaluates unlearning models when the training data is not available. We focus on unlearning identity data, which is specifically relevant due to current regulations requiring personal data deletion after training. To cope with data absence, we expect users to provide a portraiting picture to aid unlearning. We design requests on CelebA, CelebA-HQ, and MUFAC with different unlearning set sizes to evaluate applicable methods in 1-SHUI. Moreover, we propose MetaUnlearn, an effective method that meta-learns to forget identities from a single image. Our findings indicate that existing approaches struggle when data availability is limited, especially when there is a dissimilarity between the provided samples and the training data.

Published in Transactions on Machine Learning Research (03/2025)
Unlearning Personal Data from a Single Image
Thomas De Min University of Trento, Italy doomas.demine @uniln.
Massimiliano Mancini University of Trento, Italy
Stéphane Lathuilière
LTCI, Télécom-Paris, Institut Polytechnique de Paris, France Inria Grenoble, Unic. Grenoble Alpes, Prance
Subhankar Roy
University of Trento, Italy
Elisa Ricci
University of Trento, Baayy
Pondazione Bruno Kessler, Italy
Reviewed on OpenReview: https: // openreview. net/ forun? id- V#C4P2711 Ym
Abstract
Machine unlearning aims to erase data from a model as if the latter never saw them during training. While existing approaches unlearn information from complete or partial access to the training data, this access can be limited over time due to privacy regulations. Currently, no setting benchmark exists to probe the effectiveness of unlearning methods in such scenarios. To fill this gap, propose novel task call One-Shot Unlearning of Personal Identities (1-SHUI) that evaluates unlearning models when the training data is not available. We focus on unlearning identity data, which is specifically relevant due to current regulations requiring personal data deletion after training. To cope with data absence, we expect users to provide a portraiting picture to aid unlearning. We design requests on CelebA, CelebA- HQ, and MUFAC with different unlearning set sizes evaluate applicable methods in 1-SHUI. Morcover, we propose MctaUnlearn, an effective method that meta-learns to forget itentities from single image. Our findings indicate that existing approaches struggle when data availability is limited, especially when there is dissimilarity between the provided samples and the training data. Source code available at github. com/tdemin16/ 1al////oo-ssus
1 Introduction
In the sci-fi movie franchise Men in Black, the secret service agents possess Neuralyzer, brainwashing device that, with a bright flash, selectively wipes out anyonc's memory of alien encounters. Not being limited to fiction, "Neuralyzer--like tool is also practical for machine learning practitioners in order to erase information from trained neural networks. Such a requirement can inevitably arise when data owners exercise their "right to be forgotten" (Voigt & Von dem Bussche 2017 Mantelero 2013 Goldman 2020). Given the practical implications, Machine Unlearning (MU) aims to forget the influence of torgeted information from trained model, if they not part of the training set.
Some approaches are designed to forget class subclass from the model (Chundawat et al. 2023a Chen et al.] 2023). However, more realistic methods are conceived to unlearn random subset of the
*Corresoonding Author.

---

Published in Transactions on Machine Learning Research (03/2025)
Standard Machine Unlearning
One-Shot Unlearning of Personal Identities
Model
Model
Unlearning
Unlearning
Unleamed Model
Unleamed Model
Forget Set
Support Sample
Retain Set (Optional)
One-Shot Unlearning of Personal Identities. Standard machine unlcarning approaches leverage the entire forget set to forget an identity. In 1-SHUI, the user provides a picture of themselves as the only input to the unlearning algorithm for forgetting their identity. 1-SHUT evaluates unlearning when the entire training set is inaccessible.
Figure 1:
training data (Jia et al. 2023 Fan et al.
2024 Cheng & Amiri 2023 Shen et al., 2024 He et al. 2024
Zhao et al., 2024b; Huang et al. 2024). These works require full (Fnn et al.) 2024 Jia et al. 2023 or partial (Foster et al. 2024b Hoang al. 2024 access the training data as they compute gradient. updates unlearn. Despite their promising results, assuming access to the original dataset unrealistic under current data protection regulations (Mantelero 2013 Goldman 2020), e.g. Article 6 of EU Directive 95/46/ 46/EC forbids to store personal data "for no longer than is necessary". To address this challenge, existing works (Chundawat et al. 2023b; Kravets & Namboodiri 2024 Tarun et al. 2023 propose to exploit reconstruction-base
with carefully designed loss functions or identify important paramcters for the information
algorithms to aid unlearning of categories without accessing the original data. However, these techniques can only be applied to unlearnneg scenarios, not being able to reconstruct specific data points. Thus, they cannot be used for targeted unlcarning (i.c., random data unlcarning), limiting their applicability.
The objective of this work is to allow for unlearning randomized samples in data absence scenarios. To achieve this, we propose novel task, One-Shot Unlearning of Personal Identities (1-SHU1), which evaluates unlearning when training data is unavailable. 1-SHUI focuses on identity unlcarning, rcalistic scenario that accounts for unlearning personal data to protect privacy, which is a special case of random unlearning. The target of 1-SHUT are machine learning models trained for an arbitrary downstream task (e.g., face attribute recognition, age classífication), performing unlearning at the identity level. To circumvent the lack of training data, we propose that unlearning requests from a user are accompanied by single "Support Sample" portraying their identity to the unlearning algorithm. Therefore, unlearning methods must unlearn the whole data associated with the identity without accessing the real forget data, but to this single sample.
Since unlearning using single image per identity is challenging, we propose a learning-to-unn unlearn framework, MetaUnlearn, that trains a meta-loss function to approximate an unlearning loss without needing the full training data. During training, MetaUnlearn exploits the available training data (before their removal) to simulate unlearning requests. By sampling random identities and their respective Support Sets, MetaUnleat iteratively exploits the Support Set to unlearn IDs while using the rest of the data compute the error estimate, which is backpropagated to optimize MctaUnlearn parameters. Although 1-SHUI is challenging, MetaUnlearn achieves consistent forgetting results compared to existing approaches across different datasets
(i.e., CelebA [Liu et al. 2015), CelebA-HQ [Karras et al. 2017 Lee et al. 2020). MUFAC (Choi & Na 2023 and unlearning set sizes.
Contributions. In summary, our contributions are the following:
We are the first to highlight the mismatch between realistic data availability and existing unlearning evaluation benchmarks.
We propose 1-SHUI, the first benchmark for methods that perform targeted unlearning in absence of training data.
2

---

Published in Transactions on Machine Learning Research (03/2025)
3. We evaluate existing applicable MU methods 1-SHUI, revealing that they struggle to tackle machine unlearning when training data is unavailable.
4. We propose MetaUnlearn the first approach tailored for unlearning in data absence, showing its effectiveness in 1-SHUL.
2 Related Work
Machine Unlearning. By retraining (part of) the network weights, exact machine unlearning meth-
ods (Aldaghri et al., 2021 Bourtoule et al.] 2021 Yan et al.)
2022) can guarantee that training data is completely forgotten. However, retraining even part of the model makes these approaches prohibitively
expensive (Chundawat et al. 2023a Foster et al. 2024b: Graves et al. 2021h. As an efficient alternative,
approximate unlearning algorithms (Chundawat et al. 2023a Trippa et al. 2024 Choi & Na. 2023 Fan
ct al.) 2024 Foster ct al., 2024b|a Chen et al. 2023 Chundawat ct al., 2023b Jia al.) (0233) propose
to remove sample influence by tuning model parameters for just few learning steps, relaxing guaranlee
constraints. Most approximate unlearning works are designed for class (Chen et al. 2023
Chundawat et al.
2023a: Cheng et al. 2024 Hoang et al., 2004 Zhao et al.] 2024a] or random (Jia et al. 2023; Fan et al.
2024 Kurmanji et al.) 2023 Cheng & Amiri 2023 He el al.] 2024 Huang et al.; 2024
Shen et al. 2024)
unlearning, thus, they either focus on forgetting a single class or a random subset of the training dataset.
However, both these lines of work assume full or partial access to the training dataset, which can be limited in time (Mantelero) 2013). To circumvent data requirements, Chundawat et al. (2023b) proposes recon- structing forget information via an error maximization noise generation process (Tarun et al. 2023 aiming to invert the learning process. In the same paper, Chundawat et al. (023b)) proposes another method that exploits knowledge distillation (Hinton) 2014) and a generator network to distill synthetic information from the original model to a learning student while bypassing forget-related information. Instead, Kravets & Namboodiri 20244 proposes to unlearn classes from CL.P (Rombach et al. 2022 by generating synthetic samples via gradient ascent (Szegedy 2013
Although these methods demonstrate unlearning capabilities in a data-absence scenario, they are limited by reconstruction-basee algorithms, which use the model output distribution the only source of information about training data. Therefore, they cannot unlearn information that is not linked to the output (e.g., random data, identity data). To circumvent this limitation, we propose a novel task called One-Shot Unlearning of Personal Identities. By requesting users for a picture portraying themselves, 1-SHU1 exploits the semantic similarity of images of the same identity to overcome the data absence constraint. The requested image aids unlearning and is discarded after use.
Meta-Learning. Contrary to standard machine learning, meta-learning distills knowledge from multiple learning episodes for improved learning performance, a paradigm known as learning to learn (Hospedales)
et al. 2021b. Preliminary meta-learning approaches (Finn et al.] 2017 Nichol et al. 2018 Antoniou et al. focused on learning the model initialization to facilitate few-shot learning, via e.g. first Nichol et al.
2018
2018 or second-order (Finn et al.] 2017), or specific training recipes (Antoniou et al., 2018). Beyond few-
shot learning, meta-learning has been applied to other tasks, such as domain gencralization (Li ct al. 2019). continual learning (Javed & White, 2019), and fast model adaptation (Bechtle al.) 2021). Related to our work, Huang et al. (2024) achieves better forgetting and retaining balance via meta-learning- beaseennnnn-needddddddeeeee approach that improves over existing baselines. Gao et al. (2024) proposes meta-learning framework to mitigate the degradation of benign concepts that are related to unlearned (harmful) ones. However, both works cannot be used in data absence scenarios as both require original data availability.
Following meta-learning principles, our proposed approach MetaUnlearn approximates the unlcarning pro- cess by simulating multiple unlearning requests. At the end of the training, MetaUnlearn can generalize unlearning by accessing only a single sample per identity.
3

---

Published in Transactions on Machine Learning Research (03/2025)
3 Problem Formulation
This section introduces the machine unlearning problem and the proposed One-Shot Unlearning of Personal Identities task, highlighting how the benchmark dataset is constructed in practice. Section 3.1 provides a machine unlearning overview, focusing on identity unlearning. Section 3.2 rntuues the novel 1-SHUT task, and describes its relevance in realistic scenarios. Finally, section 3.3 contains a schematic overview of how we split the training data to construct the benchmark dataset.
3.1 Machine Unlearning
Let fe X + y be a model parameterized by 0 that maps inputs from the image domain X' to the target domain y (e.g,, face attributes), with 0 trained on dataset D,,- The dataset is characterized by image- label pairs such that Drr = 8y)5 with N the dataset size. The goal of machinc unlcarning is to "Torget" a subset of the training dataset Dy C Dr, while preserving original model performance on the retain D, Drr \ Dj and test Die datasets. Given original model parameters 0 and forget set, an effective MU algorithm U outputs model weights 0n that have minimal distance w.r.t. the optimal paramcters 0, computed by retraining the model f(-) on D,:
O₂ arg min d( 8,)),
where d is a distance metric measuring the closeness between the two model (0,, and 0,) distributions et al. 2024b).
Zhao
While in random sample unlcarning (Foster et al.] 2024b Golatkar al., 2021) the forget set samples are
i..d., in the identity unlearning scenario (Choi & Naj 2023) samples of the same identity correlate. Let denote Ier the training, I the forget, and I, the retain identities, with Ij Iar \ I,. We construct Dj
from Iy:
Dj = ((xx Uss i is E /
(2)
where iz is the identity number, and Ny is the forget set size.
3.2 One-Shot Unlearning of Personal Identities
As section T shows, complete or partial access to the original dataset is not guaranteed in realistic sce- narios (Mantelero) 2013 Goldman 2020 To circumvent this limitation, this work proposes One-Shot Unlearning of Personal Identities (I-SHUT) a novel benchmark that requires unlearning methods to operate without access to training data at unlearning time. Specifically, we analyze the case of identity unlearning where the entire training dataset is discarded after training, as it might contain sensitive data. One-Shot Unlearning of Personal Identities exploits the semantic similarity of images the same identity to cope with the challenges of this setting. In practice, we require users asking to be unlearned to provide one image with their identity portrayed. We name this image the Support Sample. This sample aids unlearning and is discarded after use. We define the collection of Support Samples from the same unlearning request as:
S= [(x 1 E E (Ak E (1,.., - -1
Thus, the Support Set contains only one data point for each identity, where Ns is the number of identities to unlcarn. Given the Support Set S and the original model weights 8, the unlearning algorithm outputs Og to minimize equation 1 U1(0 S) 0,, Therefore, U must. unlearn Dj by having only access to S. This constraint makes the task extremely challenging (i) methods cannot access the retain the forget sets, and (ii) the Support Sample cannot capture the entire distribution of the forget samples we aim to remove.
3.3 Benchmark Construction
To evaluate methods in 1-SHUI, we split the full dataset D in train Der, validation D., and test Die sets with non-overlapping identities, figure illustrates. Then, randomly select. Ns identities from Drr
4

---

Published in Transactions on Machine Learning Research (03/2025)
It br
If
I
Sample one image for each forget identity
I
S
I,
Ite
Figure 2: Benchmark dataset construction. The dataset is split based on identities, dividing them into train Ior validation Iy and test Ite IDs. Following, we sample forgetting identitics from training ones, splitting Zor into forget I and retain I, IDs. Out of forget identities, we sample one image for each identity to form the Support Set S, which unavailable training time.
that serve the base to construct the forget dataset Dj. The remaining IDs are left for the retain set Dr. For each identity in the forget set, randomly remove one image to form the Support Set S, such that the same image cannot appear in both D, and S. We note that the Support Set is also removed from the training dataset, and thus unscen by the model: SnD, = Ø and consequently SnD, = Ø. After splitting the dataset, we train the model on Drr and use S to unlearn Dy. Instead, the evaluation performed on Dr. Dj, and Dic.
4 Learning to Unlearn for 1-SHUI
This section presents MctaUnlearn, our proposed approach tailorod for 1-SHUI that unlearns identitics using Support Samples only. MetaUnlearn exploits the dataset to learn an unlearning loss that can forget identities through the limited set of Support Samples, before discrding the training data. Thus, proposed approach consists of three stages: (i) model training the downstream task; (ii) learning of the unlearning loss; (iii) unlearning via the trained loss when data is not available anymore. Figure Bsmmmriizes these three steps intuively. Section 4.1 describes how we train MetaUnlearn in a tractable and scalable way to generalize unlearning without accessing original data. Scction 4.2 instead, presents how the trained MctaUnlearn can used to unlearn data absence.
4.1 MetaUnlearn training
The learnable loss h-(+) (step ii) requires simulating unlearning requests to estimate hz(.) unlearning error and optimize parameters. Before the data discarded, simulated unlearning requests are formed by randomly sampling Ns identitics from the training set and split D,, following the same procedure of sec- tion 3.3 We, therefore, forward simulated support samples through the original network and the meta-loss and compute gradient step w.r.t. model's parameters 0 to produce unlearned parameters 8,. We evaluate the effectiveness of unlearning and backpropagate the error to hz(+) parameters. We discard Ou, simulate another unlearning request, and iterate until convergence (further details in appendix A). Ns is kept fixed throughout training and, to ensure all identities are used, ID sampling is done without replacement.
Following previous works meta-learning (Li al., 2019 Bochtle et al. 2021). we design the learnable loss h-(-) as an MLP parameterized by é. We aim train the meta-loss in such a way that computing optimization step using the gradient of the meta-loss Voha(fo(S)) minimizes equation Therefore,
5

---

Published in Transactions on Machine Learning Research (03/2025)
D tr
S
f D
S
Legend
M Meta-loss value
fo
feu
fo fou
A Auxiliary loss
fo
Trainable params
h
A
h
Frozen params
+
Forward Pass
0 e- 0 BM
task
M
M
a t & aV aVOA
(i) Model training
(ii) MetaUnlearn optimization
(iii) Unlearning
D
D tr
Figure 3: MetaUnlearn pipeline. While training data are available (i) The original model fo is trained on Der using the task loss function (e.g., cross-entropy loss). (ii) Before being discarded, the training data can be used learn the proposed unlcarning loss (MetaUnlearn). MetaUnlearn simulates an unlcarning request S and use the meta-loss (ha()) to unlearn it al once. We evaluate the unlearned model fa, using equation 66 (A) the forget, and validation data and backpropagate to ha(-). fe, is discarded, simulate another unlearning request and iterate until convergence. (iii) Once the MetaUnlearn is trained, we can use it to unlearn identities via Support Samples, without accessing the original training set.
iteratively simulate unlearning requests and optimize h+(+) parameters with:
+ = arg min d (ooos)a(((()))
(4)
where 17 is the learning rate and Va computes the gradient the meta-loss w.r.t. the model parameters. Compared to equation I the minimization is now carried through the learnable parameters & and not 6..
Although directly optimizing equation is computationally tractable, minimization not scalable as it requires computing retrained parameters 0, for each simulated request. Therefore, we propose to evaluate the unlearning effectiveness by exploiting an auxiliary error function A that is agnostic to 9,. However, computing reliable error estimate without the retrained weights is challenging. Assuming that the unseen sample losses for retrained and original models are similar for sufficiently small forget sets, we propose that A minimizes the discrepancy between the average forget loss and the original model validation loss. Intuitively, forget loss values should be close to the loss of unseen samples the retrained model, on average. Thus, we optimize & as follows:
Φ arg min A arg min Crasse (Dy;Ba)) Lrask (De; (w;;ow)l + [[((tak(DD;(b)) Lrask (Do;
(5)
Forget and unleurning test losss aligument
Forget and original test loss alignment.
where 02 = Vih (Jo(S), and Crsk the loss used to train the model Se on the downstream task (c.g., cross-cntropy loss). The first term of equation 5 aligns forget and validation sets loss of the unlcarned model to ensure they are close. The second term aligns the forget loss of the unlearned and original model, preserving performance. Together, the two terms push forget data to be treated unseen (i.e., validation) w.r.t. both its current status (first term) and the original model (second term). We note that the gradient Vaha(fo(S)) is computed at once for all Support Samples in S. Therefore, equation evaluates unlearning after just one gradient update.
Although equation 5 aligns forget and validation set losses, we noticed that, some cases, this did not correspond to an alignment also in accuracy terms. To explicitly account for mismatches in the accuracy between forget and validation set, scale each loss term in equation 5 by the opposite of its corresponding

---

Published in Transactions on Machine Learning Research (03/2025)
accuracy:
Φ = arg min arg min [](1 mAP(DD;; Liask (eack(Dy; - (l Ltask 0w)ll
(6)
+ +[[1 (l1-AAP:Dy;))) Crask (Lask(Dy::>)) (Dy: 0u) (1-mAP(D.;:) Ctack (D::
To summarize, the first term of the auxiliary loss promotes similar loss values for the unlearned model between forget and unseen samples, making them less distinguishable. Then, the socond term ensures that original model loss values for unseen samples preserved, limiting performance degradation. Finally, scaling by the opposite of the accuracy improves accuracy alignment between forget and test sets. Once the meta-loss is trained, the dataset can be discarded.
4.2 MetaUnlearn unlearning
By simulating unlearning requests multiple times, the meta-loss learns to forget all identities in the training dataset by only relying on Support Samples S. In order forget identities at unlearning time (step (ii)). it is sufficient to forward S through (ff((-)) and update & with single gradient step: 02 Vohs
Similarly to equation 5 and D all identities are unlearned at once via one gradient update. Algorithm summarizes the training and unlearning procedures (step (ii) and (iii) of MetaUnlearn.
5 Experiments
Section 5.1 describes 1-SHUI experimental set- ting, outlining datasets, metrics, and baselines used (see section A for implementation de- tails). Section 5.2 evaluates existing bascline approaches and our MetaUnlearn in 1-SHUI, while section 5.3 compares methods when the number of unlearning requests is extremely low. Instead, sections 5.4 and 5.5 investigate when unlearning the hardest and when un- learning harms model performance. Finally, section 5.6 provides complete ablation of MetaUnlearn.
Algorithm MetaUnlearn pseudocode.
def simulate unlearning(Z1r D..): I, ~ Ier sampling w/o replacement D, ((x).)i.1)) ig 1
D. Der \D
S. build, support, set (I,,2,) return S,D,,D.
def MetaUnlearn training((1r, Drr for epoch in range (num (aum_epochs): # iterate all IDs in batches of size Ns for it in range([N/Ns])
5.1 Experimental setting
# compute simulated unlearning step S S.D,,D. D, M = h)()o(S))
Datasets. To evaluate unlearning ap- proaches in 1-SHU1, we focus on datasets amnotated with identity information and for different downstream task (i.e., face attribute recognition, age classification). The goal is to train model on the downstream task and to perform identity-aware unlearning while preserving the original model accuracy on the test set. We identify three datasets that satisfy our requirements: CelebA-HQ (Karras et al. 2017 Lee et al.] 2020), CelebA (Liu ct al. The first. two datasets contain images of celebrities annotated with 40 facial attributes
simulate, simulate_unleartn www.arninennnnnnneemen (Irr D..))
nVaM # unlearn
step
# evaluate meta-loss
MSE (D(; (D,)
A += MSE(Crsa Leaet(De; 0))
# update using gradients V and ir a
$ Adam(ó, VaA, a)
return None
def MetaUnlearn Mocalllleemm_mnleaamnnggg.. M hs(fo()))
2015). and MUFAC (Choi & Na 2023)
0 = v. unlearn S in one step return
and identity information. The third, instcad, is built from family images, where cach identity is portrayed multiple times with photos spanning different ages (c.g., from newborn to teenager), and annotated for eight different age categories. We investigate two unlearning sizes: 20 and 50 identities for Cel

---

Published in Transactions on Machine Learning Research (03/2025)
namely CelebA-HQ/20 and CelcbA-HQ/50, and 5 and 10 IDs for CelebA and MUFAC, i.. CelebA/5, CelebA/10, MUFAC/5, and MUFAC/10. We set lower number of identities for CelebA and MUFAC, as each counts approximately 20 and 10 hamples/identity
respectively, compared to CelebA-HQQss 5
Metrics. We report retain, forget, and test accuracy, for each method to estimate the unlearning effec- tiveness. As ranking approaches over three different metrics is challenging, we summarize the three reported accuracies using the Tug of War (ToW) metric (Zhao et al. 2024b). ToW captures the performance discrep- ancy with the retrained model on the three datasets in a single score, simplifying evaluation:
ToW = [1 mAP(D,; 0.) mAP (D,: 8,)1]
[1 |mAP(D,; f: (w) mA 0,)1]
[1 ImAP( 0.)) mAP 0,)11
Like previous approaches (Kurmanji et. al.
2023 Hoang el. al. 2024): we additionally report the membership
inference attack (MIA) (Shokri et al. 2017 Carlini et al.
2022 Yeom et al. 2018 Zarifzadeh et al. 2024]
mcasuring the true positive rate in inferring sample membership at 0.01% falsc positive rate (further details in section
B We denote in bold the best method and underline the second best on each metric.
Baselines. Although several machine unlearning approaches exist, only a handful can be applied to 1- SHUI. Methods that only assume access to the forget data are the only applicable algorithms for our setting, we can carry unlearning via Support Samples only. Thus, compare MetaUnlearn with SSD (Foster ett al. 2024b), , PGU (Hoang et al. 2024), and JiT (Foster et al. 2024a). While SSD unlearns via parameter dampening, PGU and JiT unlearn the model via gradient updates. Note that not performing meta learning and replacing (-) with the losses proposed in Hoang et al. (2024) or Foster et al. (202a)), would be equivalent to using PGU, and JiT respectively, with the only difference that MetaUnlearn forgets all identities in a single gradient update. Thus the comparisons with these methods already validate the effectiveness of our meta-learning approach. We also report SCRUB (Kurmanji et al.) 2023] and Bad Teacher (Chundawat et al.] (023a)) that perform unlearning using the entire training dataset: their performance is used reference.
5.2 Results in One-Shot Unlearning of Personal Identities
Tables I to 3 evaluate cxisting methods and MetaUnlearn in One-Shot Unlcarning of Personal Identities. We divide both tables into two sections corresponding to different unlearning set sizes. From top to bottom, we report pretrain and retrained model performance for each size, as previous works (Jia et al. 2023 Fan
ct al. 2024). Following, we detail Bad Teacher (Chundawat et al. 2023a) and SCRUB (Kurmanji et al. results as reference. Finally, in the second part, we report methods that can operate with Support Samples only, including MetaUnlearn.
2023
On CelcbA-HQ/20 (table D only MetaUnlearn and PGU achieve higher ToW score than the pretrain model. When unlearning 20 identities, they score 95.7 and 95.6 against 95.5 of the pretrain model, showing the best unlearning performance from the accuracy perspective against tested methods. However, no method, except for Meta Unlearn, substantially improves over the pretrain model on CelebA-HQ/550 (table D which scores a ToW of 95.9 vs. 94.8 of pretrain. Interestingly, SSD decreases the forgetting mAP to lower values (82.0 and 70.9) compared to MetaUnlearn (82.6 and 82.5) and PGU (82.8 and 83.7) in both settings, suggesting better forgetting. Yet, it uniformly degrades retain and test sets performance, achicving a lower ToW score (94.3 and 73.7) than MetaUnlearn and PGU. In both Cel experiments, JiT performance degradation makes it inapplicable robustness to membership inference attacks, SSD always achieves the best protection, with MetaUnlearn achieving comparable results (e.g,, 0.03% vs. 0.04% in CelebA-HQ/220) Instead, JIT and PGU, show worse results, offering a weaker irotection ion to MIAs (i.e., 0.97% and 0.17% in CelebA-HQ//0). By, factoring in the better accuracy alignment with the retrained model, MetaUnlean Unlearn achieves the best performance overall.
test mAPs drop by respectively 5.7 and 16.4. By looking at the
Compared to the above case, JiT achieves the best alignment with the retrained model on CelebA/5 (table in accuracy terms (scoring 97.6) while being slightly worse than the best result (MetaUnlearn and PGU) in
8

---

Published in Transactions on Machine Learning Research (03/2025)
Table Unlearning on CelebA-HQ dataset. The first two columns report the method name and whether it uses retain and forget sets. Following, we show the average mAP on the retain, forget, and test set, while we show the ToW metric and MIA in the last two columns.
Access dataset
mAP
Method
ToW f
MIA +
D.
Dj
Din
20 IDENTITIES
Pretrain
84.8#0.11
83.0
80.7=0.0
95.5
0.97%
Retrain
84.7#0.2
78,643.5
80.8=0.1
Bad Teacher (Chundawat al.
2023a
84.410.1
82.312.9
80.3.0.2
95.610.9
0.01%
SCRUB (Kurmanji et al.
2023
<
88.110.1
80.212.9
81.2.10.
94.711177
0.02%
JiT
(Foster et al.
2024a
X
78,2+1.5
75.9+14
75.0=1.2
85.7=3.1
0.97%
PGU
Hoang el
2021
X
84.6.10.0
82.812.9
80.6
95.6
0.17%
SSD
Foster et al.
2024b
X
83.4110.6
82.0:3.1
79.6=0.55
94.311.3
0.03%
Meta nlearn
X
84.5:10.2
82.642.5
80.7=0.1
95.7:10.8
0.04%
50 IDENTITIES
Pretrain
84.9#0.1
83.
80.8=0.1
94.8=0.4
6.10%
Retrain
84.6:10.2
79.1+1.2
80.6=0.1
Bad Teacher [Chundawat et al.
2023a
83.6:t0.1
81.9#0.4
79.4=0.1
95.1
0.49%
SCRUB (Kurmanji et 2023
87.0
81.410.7
81.2-0.3
94.721.0
0.02%
JiT
Foster al. 2024a
66.3
65.340.9
64.7=1.3
59.2+2.4
0.41%
PGU Hoang et al
2024
X
84.6#20.3
83.7+1.6
80.5=0.3
94.9+0.6
0.41%
SSD
Foster
2024b
X
72.7+5.0
70.9+5.7
70.9=1.5
73.7+11.3
0.01%
Meta nlearn
84.1.0.4
82.5.1..
80.6.0.33
95.910.6
0.01%
Unlearning on CelebA dataset. The first two columns report the method name and whether it uses retain and forget sets. Following, we show the average mAP on the retain, forget, and test set, while show the ToW metric and MIA in the last two columns.
Table 2:
Access to dataset
mAP
Method
ToW 1
MIA 1
D.
Dj
Dec
IDENTTTIES
Pretrain
84.4+0.0
81.8+2.2
80.9=0.1
97.2+1.0
13.25%
Retrain
84.5+0000
79.143.2
80.9
Bad Teacher (Chundawat et al.
2023a
84.3+0.0
79.543.4
80.7=0.1
99.1
22.89%
SCRUB (Kurmanji et al. 2023
87.9+0.1
77.543.4
80.6=0.2
94.8+0.7
3.61%
JiT (ooterr et al. 2024a
X
84.210.1
81.11222
80.7=0.2
97.611..
8.43%
PGU
Hoang et
2021
84.4.0.0
82.112.1
80.9
96.911.5
7.23%
SSD
Foster et al.
2021b
X
23.1:10.
30.912.0
23.1
8.4.0.9
13.25%
Meta ullarn
84.4+0.0
81.942.2
80.9=0.1
97.2+1.0
9.64%
10 IDENTITIES
Pretrain
84.5.0..1
83.0.1..1
80.9
97.3
29.76%
Retrain
84.4.0.1
80.414.5
80.9
Bad Teacher (Chundawat et al.
2023a
84.2+0.1
82.243.7
80.5=0.1
97.6+1.0
33.93%
SCRUB (Kurmanji et al. 2023
87.9+0.1
79.9
80.6=0.1
95.8
1.20%
JHT
(ootee al.
2024a
X
83.9#1.4
82.5+14.4
80.4=0.3
96.9
23.21%
PGU
Hoang el al. 2024
X
84.5:t0.0
82.9+433
80.9=0.1
97.4
25.00%
SSD
Foster al.
2024b
26.944.33
30.115.2
26.9
10.043.2
10.12%
Metal Ulearn
X
84.4+0.0
82.944.3
80.9
97.4400.3
19.05%
CelebA/10 (table
SSD performance degradation (test mAP of 23.1 and 26.9) makes it inapplicable real- world scenarios, although it achieves the best MTA in CelebA/11 MetaUnlearn and PGU perform similarly
9

---

Published in Transactions on Machine Learning Research (03/2025)
Table 3: Unlearning on MUFAC dataset. The first two columns report the method name and whether it uses retain and forget sets. Following, we show the average Accuracy on the retain, forget, and test set, while we show the ToW metric and MIA in the last two columns.
Access to dataset
Accuracy
Method
ToW t
MIA +
D.
Dj
Din
IDENTTTIES
Pretrain
99.8
99.3+1.0
78.2=0.7
70.3+10.3
0.04%
Retrain
99.7#0011
70.0+11.4
78,1=0.4
Bad Teacher (Chundawat al.
2023a
99.710..0
67.318.3
77.511.1
94.114.4
0.00%
SCRUB (Kurmanji et al.
2023
<
100.011..0
89.013.8
78.0
80.119.9
0.02%
JiT
(Foster et al.
2024a
X
99.7=0.11
99.3#1.0
77.8=0.8
70.1+10.3
0.02%
PGU
Hoang el
2021
X
99.8.10.1
99.311.0
78.1
70.3
0.04%
SSD
Foster et al.
2024b
X
1..2::1.6
14.41400
15.0=1.1
2.5=0.5
0.01% %
Meta nlearn
X
99.6:10.3
98.5+1.0
78.0=0.9
70.8
0.03%
10 IDENTITIES
Pretrain
99.9+0.00
99.7
77.4=0.7
76.7
0.03%
Retrain
99.6:10.4
76.847.4
77.4=0.7
Bad Teacher /Chundawat et al.
2023a
99.7:t0.0
67.948.0
77.2=0.1
90.2#0.7
0.00%
SCRUB (Kurmanji et 2023
99.8+0.00
95.742.7
78.3-0.3
80.2
0.02%
JiT
Foster al. 2024a
99.7+0.11
99.7
76.9=0.9
76.4
0.03%
PGU
Hoang et al
2024
99.5#t0.3 X
96.8+2.3
76,9=1.1
79.0+9.1
0.04%
SSD
Foster al.
2024b
X
12.3+11.
15.3+24
12.5=3.5
1.7=0.6
0.01%
Metal Unlearn
X
96.3.13.5
91.817.11
74.1
79.1
0.04%
in both CelebA tests, scoring the best ToW CelebA/10 (97.4). However, MetaUnlearn slightly outperforms PGU CelebA/5 (97.2 vs. 96.9), achieving better aligmment with the retrained model. By investigating the MIA, PGU and JiT achieve the best and second best results (7.23% and 8.43%) CelebA/5, while MetaUnlearn is slightly outperformed (9.64%). Yet, MctaUnlearn achieves the best MIA CelcbA/10 among methods that preserve sulficiently high mAP, scoring 19.05% vs. 23.21% of JiT (the second best). MetaUnlean is overall more consistent in CelebA experiments than existing baselines. It shows the same average ToW with JiT (97.3 vs. 97.25) across the two experiments but a lower average MIA 14.3% against 15.8% PGU.
Finally, the MUFAC dataset demonstrates the most challenging of the three (table 3). We believe its difficulty is caused by the large age gap between different pictures of the same identity, especially for young people. All tested methods struggle to remove target identities effectively from the ToW perspective. SSD cannot identify important parameters for the forget set, scoring a ToW of 2.5 and 1.7 in MUFAC/5 and MUFAC/10. Instead, JiT always fails to improve the ToW over the pretrain model, suggesting little-to-no unlearning, On the contrary, PGU and MetaUnlearn both improve over the pretrain model on MUFAC/10 by 2.3 and 2.4 points respectively. However, only MetaUnlearn manages to improve over the original model scoring ToW of 70.8. Additionally, among methods that preserve model utility (JiT, PGU, MctaUnlearn), only shows a reduction in the forget accuracy -0.8% on MUFAC/5, 7.9% on MUFAC/10), further suggesting that identities are at least partly unlearned. Regardless, the attack success rate (MIA) is very low for all methods, where values are very close to random guessing (0.01%) %), which suggests good privacy protection from membership inference attacks.
Although MetaUnlearn struggles to outperform existing baselines in CelebA, it is more consistent than tested approaches, showing the best or comparable performance in all six configurations. The next section (section 5.3p, compares MetaUnlearn performance against other methods when unlearning extremely small forget set sets, i.e. when the number of unlearning identities equals one.
10

---

Published in Transactions on Machine Learning Research (03/2025)
Table 4: Unlearning one identity on CelebA-HQ and CelebA dataset. The first two columns report the method name and whether it uses retain and forget sets. Following, we show the average mAP on the retain, forget, and test set, while we show the ToW metric and MIA in the last two columns.
Access dataset
mAP
Method
ToW t
MIA +
Dr
Dy
Die
BELEBA-HQ
Pretrain
84.5:#0.3
90.948.9
80.7=0.1
95.8
0.01%
Retrain
84.6:0.22
87.2+9.9
80.7=0.1
Bad Teacher (Chundawat al.
2023a
84.510.3
19.9
80.7.0.1
96.812.9
0.03%
SCRUB (Kurmanji et al.
2023
88.010.2
86.818.9
81.1.102
95.110.4
0.01%
JiT
(Foster et al.
2024a
X
84,4#0.2
91.1
80.5=0.2
95.6#2.8
0.14%
PGU
Hoang el
2021
X
84.411..3
89.918.5
80.620.0
96.8
0.01%
SSD
Foster et al.
2024b
X
25.2:10.5
70.8
25.6=0.4
15.2
0.05%
Meta alearn
X
83.7:10.0
88,219.1
80.1=0.3
97.4::0.8
0.01%
CELEBA
Pretrain
84.5#t0.1
83.5+7.1
80.9=0.1
96.6+1.5
5.26%
Retrain
84.5:10.1
80.2
80.9=0.1
Bad Teacher [Chundawat et al.
2023a
84.4+t011
83.7:6.2
80.9=0.1
96.5+1.0
0.32%
SCRUB (Kurmanji et 2023
88.0+0.0
80.415.5
80.6-0.1
95.220.6
0.01%
JiT
Foster al. 2024a
84.5.1..1
82.5.1.9
80.9=0.1
97.6+1..4
5.26%
PGU Hoang et al
2024
X
84,4#0.1
83,4+7.0
80.9=0.1
96,8+1.2
0.26%
SSD
Foster al.
2024b
X
22.7+0.2
56.1+188
22.7=0.3
12.1#1.7
0.01%
Meta Unlarn
nlearn
X
84.510.11
83.517.2
80.9
96.711.5
0.01% %
5.3 Evaluation on extremely small forget set
Tables D and compare MetaUnlean and baselines in the proposed 1-SHUI task by varying the number of unlearning identities. In all experiments, always considered the number of forgetting identities to be greater than one. This section tests the robustness of the approaches when only one user asks to be forgotten, therefore, experimenting in the case where Ng = 1. Results are computed following the same evaluation procedure as tables [ and 2 and are reported in table Our method achieves the best ToW score (97.4) in CelebA-HQ, showing great alignment with the retrained model. Yet, it performs close to the second-best PGU (Hoang al.) 2024) (96.7 vs 96.8) in CelebA. Nonetheless, it strongly preserves privacy by scoring the lowest MIA rate both datascts. JiT (Foster et al., 2024a) scores the best ToW in CelebA (97.6), while achieving the same MIA as the original model worse, failing to unlearn the sensitive information. PGU, instead, achieves worse or comparable results on both datasets from both metrics perspectives. Finally, SSD strongly degrades model performance after unlearning, making it unusable. Overall, our method achieves the best ToW or is comparable with cxisting basclincs whilc always achieving the best MIA, therefore better preserving the privacy of the sensitive data.
5.4 When is unlearning the hardest?
In evaluating UetaUnlearn and existing methods in 1-SHUI, we noticed how some identities were easier to unlearn than others. We conjecture that this behavior relates to the dissimilarity between Support Samples and forget set images of the same identity (see figure G), iee. unlearning an identity that shows large gap between the Support Samples and images in the forget set might be harder. To quantify this, we computed the accuracy difference between the unlearned and retrained models for each unlearning identity, averaging the results of the top-3 unlearning methods on three seeds. We then compute the Euclidean distance in the feature space between each Support Sample and the centroid of the forget data of corresponding identity. Finally, we produced bar plot, grouping identities with similar forget data-support sample distance and sorting the distance from left. to right in increasing order.
11

---

Published in Transactions on Machine Learning Research (03/2025)
mAP gap on Dj
mAP drop D, after unlearning
2.5
2.5
2.0
DAP difference with O.
2.0
1.5
mAP difference with
1.0
1.5
0.5
1.0
0.0
0.5
0.5
-1.0
0.0
24
32
40
48
57
66
75
83
91
99
31
36
39
42
46
50
53
58
63
Euclidean distance from unlearning ID centroid
Support sample Euclidean distance from ID centroid
Figure 5: Performance drop vs. distance from forget set. As retain identities get closer forget identities, retain samples mAP drops compared to the pretrain model.
Figure Unlearning hardness vs. Support Sample distance. As the Support Sample dis- tance from the identity centroid increases, the ac- curacy gap with the retrained model grows.
Figure $ shows the results of our analysis. The unlearned model accuracy on forget data struggles to achieve retrained model values as the distance increases, going from difference of -1.1 to 2.4 in mAP terms. We also outline that if the Support Samples are close enough to identity centroids, the forget accuracy can be lower than the retrain model value, suggesting a possible Streisand Effect (Chundawat et al.) 2023a Foster et al. 2024b). The Streisand Effect occurs when unlearning is too aggressive and the accuracy of the lorget set drops way below the retrained model one. When the model confidence on unlearned samples too low compared to unseen samples, it suggest to the attacker that the forget set was previously part of the training set and it was unlearned, leaking membership information.
5.5 When does unlearning harm performance?
Analogously to the previous case (section 5.4). ; we noticed that, in both datasets, most methods demonstrated a drop in performance from the original model on the retain data, especially on CelebA-HQ We hypothesize that the closer a retain identity is to a forget one, the higher the chance of an accuracy drop on retain data compared to the original model. Thus, we analyzed whether the distance between forget identities' centroids and retain ones correlates with performance drop. We computed the mAP difference from the pretrain model for each retain identity, averaging three seeds and using the top-3 methods on CelebA-HQ/50 We binned accuracy drops based on identity distances, as in section 5.4 Figure 4 shows the result of our experiment. The closer a retain sample is to a forget sample, the more the mAP drops from the original model, which ranges from 2.4 to 1.5 as the distance grows. Consequently, preserving the original performance becomes easier as the distance between forget and retain samples increases.
5.6 Ablation studies
This section shows comprehensive ablation study of MetaUnlearn auxiliary loss terms, inputs to the meta-loss, and simulated requests size. We show the performance in ToW terms, due to its much cheaper computational cost than MIA while achieving correlated results (Zhao et al. 2024b
Auxiliary loss function. 5 the ablation study of the auxiliary loss used to train MetaUnleam. For completeness, we report the performances of our approach in all four configurations, averaging over three different scods. In the first row, we show ToW scores when naively applying SCRUB (Kurmanji al. (023) loss function to measure the unlearning effectiveness. As result, either the approach performs sub-optimally fails to achieve meaningful ToW scores. We hypothesize that, SCRUB loss numerically unstable if
12

---

Published in Transactions on Machine Learning Research (03/2025)
Table Ablating MetaUnlearn auxiliary loss. The first column reports the loss function used,
i.c.SCRUB loss or different combinations of equation [] Then, we report ToW scores for settings we describe in section Note that OOM short for "out-of-ment" m-meoooy.."
ToW f
Loss Type
CelebA-HQ//0
CelebA-HQ/50
CelebA/5
CelebA/10
SCRUB (Kurmanji et al. 2023
94.3+0.2
0OM
51.7#314
55.4+15.0
First term
8.3=0.7
32.4=33.4
7,620.88
6.940.6
First term + Accuracy
8.1=0.7
8.2+0.2
7.6+0.8
6.940.7
Second term
95.710.9
95.110.3
97.1i1.1
97.410.
Second term + Accuracy
95.7
95.711.0
97.211.0
97.310.3
First & Second terms
95.6+1.0
95.4+0.3
97.1+100
97.3+0.4
First & Second terms + Accuracy
95.8+0.7
95.9=0.6
97.2+1.0
97.4+0.3
Table 6: Ablating the inputs to MetaUnlearn. The first column lists all possible inputs to the meta- loss, while the remaining four we report the ToW metric on the described settings.
Input
ToW
Logits
Features
IDs
Targets
CelebaA-HQQ/0
CelebaA-HQ/50
CelebA/5
CelebaA/10
95.0+1.4
93.0=3.0
97.3+1.2
97.3+0.4
96.1=0.9
95.4
97.2+1.1
97.3#0.3
95.770.77
95.8+10.
97.241.1
97.240.3
95.7
95.9+006
97.2+1.0
97.4+00.3
not optimized in alternate way Kurmannji et al. 2023 Chen & Yang 2023 using it to measure the meta-loss error similarly be problematic. In one case (CelebA-HQ/50 experienced out-of-memory issues as SCRUB requires the model to forward retain and forget images through the original and unlearned model, largely increasing the memory footprint in a second-order optimization such as ours.
In the second and third rows, we show MetaUnlearn performance when using only the first term of equation 5 where TOW scores are close zero. The forget error gets progressively closer to the validation error when the second term is omitted. However, the validation empirical error is not bounded, which causes performance degradation (e.g., ToW of 6.9 in CelebA/10). Instead, using the sccond term of the loss offers more stability it aligns the forget error with the loss the validation set obtained using the original model, whose weights are frozen (97.4 of ToW in CelebA/10). When scaling losses by their mAP values (see section 4.1 using both terms provides consistently better results among all four settings, always achieving the best performance.
Input to the meta-loss. Table 6 reports the results of ablation study about the inputs provided to the meta-loss function. The first row shows results by forwarding only output logits to the meta-loss. MetaUnlearn struggles in Ce dataset, achieving a ToW score of 95.0 and 93.0 (against 95.5 and 94.8 of the pretrain model), while we find sufficiently good CelebA. As the second row highlights, by concatenating model features to logits, CelebA-HQ performances improve to 96.1 and 95.4, surpassing the pretrain model. When also concatenating identitics and ground truth annotations, the ToW slightly grows average. Therefore, in our experiments, we use all four inputs to MetaUnlearn.
Section 4.1 describes how simulate multiple unlearning requests using fixed size Ns. which corresponds to the one used during evaluation (i.c. we set the simulated unlearning size to 20 for the CelebA-HQ/P experiment). This choice follows practical considerations e.g. if the unlearning set size larger, one can split it to match the desired size while, if it is smaller, one can queue the requests until the desired number is reached. While this choice sound in scenarios where the user can manage the unlcarning set size, there might be cases where the size of real unlearning requests is unknown a priori. In this section, we test whether Meta Unlearn is affected by shifts between differences in number of requests between training and test. To
13

---

Published in Transactions on Machine Learning Research (03/2025)
assess this, we test our method when the size of the simulated and real unlearning requests differ. We perform four experiments: on CelebA-HQ 20 using Ns- = 50 during training, and vice versa (i.e. Ns 20 for He 50). We also test it on CelebA-10 with Ns = 5 during training and vice versa (i.e., Ns 10
for Ccl Table summarizes the results, where Reference refers to the number of simulated requests
matching those seen al evaluation time. In general, MetaUnlearn shows a very low discrepancy with the reference, L.c. with differences between 0.1 (CelebA-HQ 20 identities, CelebA 10 identities) and 0.3 (CelebA 5 identities) ToW. Although the best results are always achieved when the size of the simulated requests during training matches those of the test, MetaUnlearn achieves comparable performance also in case of discrepancies, showing its robustness to the choice of Ns.
6 Conclusions
This work is the first to investigate the unlearning of randomized samples in the data absence scenario. We present a novel lask we call One-Shot Unlearning of Personal Identities (1-SHUI) to eval- uate algorithms that perform unlcarning when the original dataset is inaccessible. 1-SHUT focuses on identity unlearning, specific case of random unlearning where methods must forget all data associated with the unlearning identity. To circum- vent the lack of training data, we assume that users asking to be forgotten, pro- vide one of their pictures (Support Sam- ple) to aid unlearning. To tackle this challenging task, propose learning- to-unlearn approach (MetaUnlearn) that
Table 7: Robustness to different unlearning set sizes. Even when the simulated and real unlearning set sizes do not match, Meta Unlearn performance is comparable to the best case where they match.
CelebA-HQ
CelebA
Training
ToW f
Training
ToW
20 IDENTITIES
IDENTITIES
Reference
95.7=0.8
Reference
97.2+1.0
50 ldentities
95.6#1.0
10 Identities
96.9+1.0
50 IDENTITIES
10 IDENTITIES
Reference
95.910.6
Reference Identities
97.4+0.3
20 Identities
95.7+0.66
97.3+0.3
learns to forget identities by relying on the provided Support Sample. MetaUnlearn simulates multiple unlearning requests to optimize a paramcterized loss function while data still available. This results in a loss function that can forget identities without relying on the original training data. Experiments on three datasets show that MetaUnlearn achieves good forgetting results on 1-SHUI, being a first step toward tackling this challenging task.
Limitations and future works. One limitation of our work is that we focus on a single unlcarning request. As confirmed by our experiments, 1-SHUT is extremely challenging, and accounting for multiple unlearning requests could have made the evaluation tough. Therefore, exploring unlearning algorithms that account for multiple unlearning requests in the data absence scenario is natural extension of this work.
Due to their wide availability and fine-grained attribute annotation, we constrained our analysis to face datasets. At the same time, it would be interesting to test MetaUnlearn and basclines in more challenging settings, such as images of the full body or even images annotated with multiple identities. Moreover, 1-SHUI could be transferred to cases beyond identities, such as specific objects and/or animals. While these are less interesting use cases from a regulation point of view, they might be considered in future works.
Acknowledgements. We acknowledge the CINECA award under the ISCRA initiative for the availability of high-performance computing resources and support. Elisa Ricci and Massimiliano Mancini are supported by the MUR PNRR project FAIR- Future AI Research (PE00000013), funded by NextGeneration EU. Elisa Ricci is also supported by the EU projects AHTRUSS (No. 1010709)))) and ELIAS (No.0112023) and the PRIN project LEGO-AI (Prot.2020TA3K0)
Thomas De Min is funded by NextGeneration EU. This work has been supported by the French National Research Ageney (ANR) with the ANR-20-CF2230022
14

---

Published in Transactions on Machine Learning Research (03/2025)
References
Nasser Aldaghri, Hessam Mahdavifar, and Ahmad Beirami. Coded machine unlearning. In IEEE Access, 2021.
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your maml. In ICLR, 2018
Jimmy Lei Ba. Layer normalization. arXiv preprint orXi::1607.06450 2016.
Sarah Bechtle, Artem Molchanov, Yevgen Chebotar, Edward Grefenstette, Ludovic Righetti, Gaurav Sukhatme, and Franziska Meier. Meta learning via learned loss. In ICPR, 2021.
Lucas Bourtoule, Varum Chandrasekaran, Christopher A Choquette-Choo Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In SP, 2021.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In SP, 2022.
Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for Ilms. EMNLP, 2023.
Min Chen, Weizhuo Gao, Gaoyang Liu, Kai Peng, and Chen Wang, Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary. In CVPR, 2023.
Jiali Cheng and Hadi Amiri. Multimodal machine unlearning. arXiv preprint arXiv:2311.1900 2023.
Xinwen Cheng, Zhehao Huang, and Xiaolin Huang. Machine unlearning by suppressing sample contribution. arXiv preprint ar Xiv:244002 02.. 15109, 2024.
Dasol Choi and Dongbin Na. Towards machine unlearning benchmarks: Forgetting the personal identities in facial recognition systems. AAAI Workshop, 2023.
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlcarning in deep networks using an incompetent teacher. In AAAI, 2023a.
Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Zero-shot machine un- lcarning. IEEE TIFS, 2023b.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa. Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. TCLR, 2021.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. Salun: Empowering machine unlearning via gradient- braseent-based weight saliency in both image classification and generation. In ICLR, 2024.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Jack Foster, Kyle Fogarty, Stefan Schoepf, Cengiz Öztireli, and Alexandra Brintrup. Zero-shot machine unlcarning scale via lipschitz regularization. arXiv preprint orXiv:2402.01401, 2024a.
Jack Foster, Stefan Schoopf, and Alexandra Brintrup. Fast machine unlcarning without retraining through selective synaptic dampening. In AAAI, 2024b.
Hongcheng Gao, Tianyu Pang, Chao Du, Taihang Hu, Zhijic Deng, and Min Lin. Mcta-unlearning on diffusion models: Preventing relearning unlearned concepts. arXiv preprinl ar 10. 2024.
Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed- privacy forgetting in deep networks. In CVPR, 2021.
Eric Goldman. An introduction to the california consumer privacy act (ccpa). Santa Clara Univ. Legal Studies Research Paper, 2020.
15

---

Published in Transactions on Machine Learning Research (03/2025)
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In AAAI, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Zhengbao He, Tao Li, Xinwen Cheng, Zhehao Huang, and Xiaolin Huang. Towards natural machine un- learning. arXiv preprint 1xXi:2245.154p00 2024.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint (rXiv:1606.08411 2016.
Geoffrey Hinton. Distilling the knowledge in a neural network. NewrIPSW, 2014.
Tuan Hoang, Santu Rana, Sunil Gupta, and Svetha Venkatesh. Learn to unlearn for deep neural networks: Minimizing unlearning interference with gradient projection. In WACV, 2024.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. TPAMI, 2021.
Mark He Huang, Lin Geng Foo, and Jun Liu. Learning to unlearn for robust machine unlcarning. In ECCV, 2024.
Khurram Javed and Martha White. Meta-learning representations for continual learning. In NeurIPS, 2019.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. In NewrIPS, 2023.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10190 2017.
Diederik P Kingma. Adam: A method for stochastic optimization. arXio preprint arXiv:1412.6000 2014.
Alexey Kravets and Vinay Namboodiri. Zero-shot class unlearning in clip with synthetic samples. arXiv preprint 20x 2024.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Fleni Triantafillou. Towards unbounded machine unlearning. In NewrIPS, 2023.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In CVPR, 2020.
Yiying Li, Yongxin Yang, Wei Zhou, and Timothy Hospodales. Feature-critic networks for heterogencous domain generalization. In ICML, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning lace attributes in the wild. ICCV, 2015.
Alessandro Mantelero. The eu proposal for a general data protection regulation and the roots of the 'right to be forgotten'. Computer Law & Security Review, 2013.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. orXin preprint arXiv:1803.0200 2018.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In ICLR, 2018
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Sathoesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Imagenet large scale visual recognition challenge. In 1JCV, 2015.
16

---

Published in Transactions on Machine Learning Research (03/2025)
Shaofei Shen, Chenhao Zhang, Yawen Zhao, Alina Bialkowski, Weitong Chen, and Miao Xu. Label-agnostic forgetting: A supervision-free unlearning in deep models. ICLR, 2024.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In SP, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Nya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, In JMLR, 2014.
C Szegody, Intriguing properties of neural networks. arXiv preprint 1312.6199, 2013.
Ayush K Tarun, Vikram Chundawat, Murari Mandal, and Mohan Kankanhalli, Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Daniel Trippa, Cesare Campagnano, Maria Sofia Bucarelli, Gabriele Tolomei, and Fabrizio Silvestri. Vr: Gradient-baad based and task-agnostic machine unlearning, arXiv preprint arXiv:2403.14389 2024.
Paul Voigt and Axel Von dem Bussche. The general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 2017.
Haonan Yan, Xiaoguang Li, Ziyao Guo, Hui Li, Fenghua Li, and Xiaodong Lin. Arcane: An efficient architecture for exact machine unlearning. In IJCAI, 2022.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Ana- lyzing the connection overfitting. In CSF, 2018.
Sajjad Zarifzadeh, Philippe Cheng-Jie Mare Liu, and Reza Shokri. Low-cost high-power membership infer- ence by boosting relativity. In 1CML, 2024.
Hongbo Zhao, Bolin Ni, Junsong Fan, Yuxi Wang, Yuntao Chen, Gaoleng Meng, and Zhaoxiang Zhang, Continual forgetting for pre-trained vision models. In CVPR, 2024a.
Kairan Zhao, Meghdad Kurmanji, George-Octavian Bärbulescu, Fleni Triantafillou, and Peter Triantafillou.
What makes unlearning hard and what to do about NewrIPS, 2024b.
17

---

Published in Transactions on Machine Learning Research (03/2025)
A Implementation details
For all experiments, we used a ViT-B/16 (Dosovitskiy et al. 2021 pre-trained on ImageNet (Russakovsky ct al. 2015). We fine-tuned VIT on CelebA and CelcbA-HQ for 30 epochs, using SGD with a learning rate of x 10-3 and momentum 0.9. The learning rate is initially warm-upped for the first two training epochs and decayed following a cosine annealing schedule for the rest of the optimization. We regularize the optimization using weight docay with penalties of 10-3 and x 10-4 for CelebA-HQ and CelebA. Additionally, we augment input images with RandommesizedCrrop and RandomnHorizontalFFii to further regularize (He al., 2016). The same optimization configuration is used for both model pretraining and retraining. We used mixed precision for all experiments using the "brain floating point 16" to speed up training. Furthermore, single A100 Nvidia GPU was used for all experiments.
The meta-loss was trained for 3 epochs, except for one experiment in MUFAC, where each epoch consists of iteratively unlearning all identities in batches of Ns IDs. Therefore, at each iteration, we sampled without replacement Ns IDs, unlearned those identitics using hg, computed the auxiliary loss function A (equation 6 and backpropagated the error back to hig Then, we discarded the unlearned network and repeated the above operations until all identities were unlearned, and cycle for three epochs. We used Adam optimizer (Kingma 2014) with AMSgrad (Reddi et al., 20188) and no weight decay. The learning rate (a) was choosed from {10-4, 1,033, 10000.000 3 and was decayed following cosine annealing schedule. Instead, the meta-learning rate (n), used when computing the unlearning step, was chosen from (10-3, 0.13. The meta-loss architecture was composed of a linear encoder that mixes input features and projects them from the input size [ + D + 64 + Di to the hidden size (512), where [) is the cardinality of the output space, is the backbone outpul. size, and 64 is the chosen size of the identity embedding table. Then, we subsequently applied LayerNorm (Ba 2016) layer, a lincar projection, and a non-lincar activation function (Hendrycks & Gimpel 2016). We used Dropout (Srivastava et al. 2014) for regularization with probability of 0.5 and another LayerNorm before computing the output projection from the hidden size to scalar value. Finally, as the loss function must output positive values only, we applied a softplus transformation. Tables 8a and 8b summarize the training details of MetaUnlearn and the architecture used.
On the absence of D, in A. We empirically noticed that directly enforcing performance alignment with D, hinders unlearning. We argue that since the size of the retain set is usually much bigger than the forget set (Foster et al. 2024b). N, >> Ny; at each simulation D. differs from Drr by small number of samples. In other words, the probability that one sample is chosen for the unlearning request is close to zero. Consequently, while forcing an aligmment on Dr, he finds the aligument with Der as a shortcut, hindering unlearning. Empirically, we noticed that adding an alignment constraint on D, resulted in an unlearned network that performed closely to the original one, hardly showing signs of unlearning.
Table 8: MetaUnlearn training details and architecture. Table Sa report the value assigned tol each hyperparameter. Table 8b instead, presents MetaUnlearn architecture.
(a) Training details.
(b) Meta-loss architecture.
Component
Value
Layer
Component
input_size
D + D + 512
64 + |
Linear size, hidden size) LayerNorm(hidden (lizddd_sddee)
hidden_size
identity_eabed_dar
64
Lincar(hidden size, hidden size) GeLU() Dropout(0.5)
dropout_prob
0.5
regression_
amooth_li_lost
meta_lr (1)
{10-3, 0.1}
LayerNorm(hiddden size)
optimizer
Adam
Lincar(hidden size, 1) Softplus()
1r (a)
(10 10 10-3, 10-2)
epachs
{3, 10)
scheduler
cosine
amsgrad
True
amp
bfloat16
18

---

Published in Transactions on Machine Learning Research (03/2025)
B Membership Inference Attack
To evaluate the privacy protection level of unlcarning methods, we rely on the recently proposed RMIA (Zariizadeh et al. 2024). Contrary to previous membership inference attacks (Carlini et al. 2022
Shokri et al. (017) that are less efficient and require different shadow models to achieve satisfactory results, RMIA performs attacks with just one shadow model its cheapest formulation. To test the unlearning capabilities of each approach, we retrained the original model using only the retain data. Then, we ran each unlearning algorithm the retrained model to account for parameter changes introduced by specific algorithms. We compute the membership score detailed by Zarifzadeh et al. (2024):
LRe, (I, 2I = Pr(x (w)) Pr(x (0))
Pri( (a))
(8)
Pr(z 0%,)
ScOreMIA (m) = Pr (LRe,, (z, 2) 2 7),
(9)
2nD
where target sample we want to infer membership on, 0g is the model retrained and unlearned (reference model), is a random sample drawn from the dataset, and y is hyperparameter. Following Zarifzadeh
et al. 2024h, we predict membership as:
MIA(x; (ScoremIA 0w) 2 3)
(10)
We set 8 such that the FPR 0.01% (False Positive Rate). and report the corresponding TPR. (True Positive Rate) in the results. We refer to the original paper (Zarikzadeh el al. 2024 for further details.
19

---

Published in Transactions on Machine Learning Research (03/2025)
Support Sample
Forget Set
Figure Visualizing unlearning hardness. This Figure portrays the five identities with the highest embedding distance between the Support Sample and the forget set. Images are taken from CelebA-HQ.
20


Reviews:
[{"review_id": "fFzyfdaJ4Y", "reviewer": "Reviewer_YfPq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe paper proposes an unlearning method called MetaUnlearn that removes the influence of some sets from the model. They construct a benchmark to evaluate the performance. The paper also lists the privacy issues of holding data after training, which can be problematic in most real-world settings.\n\n\n**Weaknesses**\n1. I believe this is meant to be learning-to-unlearn in the introduction section: \n\"Since unlearning using a single image per identity is challenging, we propose a learning-to-learn framework,\nMetaUnlearn, ...\"\n\n2. This contribution term is misleading: \"We propose 1-SHUI the first benchmark that evaluates machine unlearning methods in data absence.\" As far as I understood the evaluation, you still need the training set when creating the benchmark.\n\n3. More explanation regarding this is also necessary: \"We evaluate existing applicable MU methods on 1-SHUI, revealing that they struggle to tackle machine unlearning when training data is unavailable.\" Based on your results, some of the baselines, like PGU, perform very similarly to the proposed method and better in some cases.\n\n4. The paper uses ToW metric as the main metric instead of the accuracies of the sets. I don't believe this is the best metric here as it may punish positive improvements compared to $\\theta_r$ because of the absolute value. For example, in Table 2 (5 identities), PGU has 84.4, 82.1, and 80.9 mAP on the retain, forget, and test sets, respectively, while MetaUnlearn has 84.4, 81.9, and 80.9. On average, PGU is better but in ToW MetaUnlearn is better. The same goes for the JiT model.\n\n5. Some results seem to have a high standard deviation. How many times were the results computed to get the mean and std?\n\nIn general, the paper is not well written, and it could be improved to make it more understandable and have fewer errors. The results have also shown very marginal improvements. I will update the claims and evidence part based on your response.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes an unlearning algorithm that tries to remove the effect of unwanted data from the training set on a model without retraining it. Given some random subsets of training data, they propose to unlearn those sets and output a new $\\theta_u\\$ parameters of the model. They also propose an evaluation benchmark called 1-SHUI that is meant to compare unlearning methods in the absence of full training data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the issues in the **Weaknesses** section.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lThBHXIR20", "reviewer": "Reviewer_noQy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. **Novel and practical problem setting**  \n   The paper focuses on a scenario where personal data must be deleted for compliance reasons, making it unavailable after training. This extreme yet realistic setup aligns well with real-world privacy concerns.\n\n2. **A general, extensible framework**  \n   By adopting a meta-learning paradigm, the proposed method trains a meta-loss that can quickly adapt to unlearn specific data at inference time. This approach is naturally extensible to various tasks and model architectures.\n\n3. **Comprehensive experiments**  \n   The authors evaluate their approach on CelebA and CelebA-HQ using multiple unlearning scales (i.e., varying numbers of identities). They measure both the trade-off between forgetting and retaining, as well as membership inference attacks (MIA). The results support the main claims.\n\n4. **Comparison with multiple baselines**  \n   The paper compares MetaUnlearn with other unlearning algorithms (e.g., SSD, PGU, JiT) that do not require the original training set at inference time, thereby highlighting MetaUnlearn’s advantages.\n\n---\n\n### Weaknesses\n\n1. **Insufficient ablation on the meta-loss importance**  \n   Although the meta-learning approach is shown to be effective, the paper does not thoroughly evaluate how much of the improvement is due to the learnable meta-loss $\\(h_\\phi(\\cdot)\\)$. An ablation study using a standard loss function instead of $ \\(h_\\phi(\\cdot)\\)$ would clarify its specific contribution.\n\n2. **Unclear sampling procedure in Section 4.1**  \nI am a bit confused about whether identities  are sampled  or whether the support set \\(S\\) is  sampled. If \\(S\\) is sampled, how does this affect generalizability? For example, if a different forget subset is used, would the same mechanism remain valid? On the other hand, if identities are sampled, does this require access to all associated data, and how does this impact generalizability? Providing more details on the sampling strategy and its implications for generalizability would enhance the clarity of this section.\n\n\n3. **Rationale and effect of the mAP need more elaboration**  \nWhile the paper mentions using mAP for alignment or scaling, the performance gain appears modest, and it is not clear how mAP is computed during training.\n\n4. **Pseudo-code and meta-loss training procedure lack detail**  \n   Whether training the meta-loss parameters is done once or iteratively. Likewise, it is not specified whether the support samples in \\(S\\) are forgotten all at once or one by one. More details would make it easier for others to reproduce the results.\n\n5. **Concern Regarding Formula (5)**  \nI would like to ask if \n$\\[\nL_{\\text{task}}(D_f; \\theta_u) - L_{\\text{task}}(D_v; \\theta)\n\\]$\nin the second term can be replaced by \n$\\[\nL_{\\text{task}}(D_v; \\theta_u) - L_{\\text{task}}(D_v; \\theta).\n\\]$\nIf not, could you please provide an explanation?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel task called **1-SHUI (One-Shot Unlearning of Personal Identities)**, focusing on random-sample unlearning (particularly personal identities) under the scenario where the original training data is inaccessible. The key contributions are:\n- It introduces a **one-shot unlearning** setup, where each identity to be forgotten is represented by only a single “Support Sample,” forming a new benchmark and evaluation protocol.\n- It presents a **meta-learning**-based approach (**MetaUnlearn**), which learns a meta-loss to facilitate forgetting under these extremely limited conditions.\n- Experimental results on CelebA and CelebA-HQ show that many existing unlearning methods struggle when the training data is inaccessible, whereas MetaUnlearn achieves a better balance between forgetting and retaining.\n\nOverall, the paper addresses a very practical unlearning scenario and provides an initial feasible solution. This work has implications for privacy and regulatory compliance in machine learning.\n\n---", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Add more ablation studies on the meta-loss**  \n   In response to Weakness (1), conduct a comparison where a standard loss function replaces $\\(h_\\phi(\\cdot)\\)$. This will clarify the meta-loss's impact.\n\n2. **Clarify the sampling description in Section 4.1**  \n   Address Weakness (2) by explaining how identities or support samples are chosen, and how it affects the method’s generalizability.\n\n3. **Further elaborate the motivation for introducing mAP**  \n   In response to Weakness (3), describe how mAP is computed during training or why it is crucial for the proposed method.\n\n4. **Refine the pseudo-code training loop and forgetting mechanism**  \n   In response to Weakness (4), provide more information on the number of iterations for meta-loss training, and how exactly the MetaUnlearn_unlearning process is carried out (in batches, one by one, etc.). Including more detailed descriptions in the methods section would make it easier for readers to understand.\n\n5. **Justify Formula (5) more thoroughly**  \n   In response to Weakness (5), explain the reasoning behind useing $\\(L_{\\text{task}}(D_f; \\theta_u)\\)$ not $\\(L_{\\text{task}}(D_v; \\theta_u)\\$).", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QwLebOS3Ba", "reviewer": "Reviewer_Hsay", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. This new benchmark is closer to the realistic scenario where the model training data is not available but the model still needs to be evaluated for the machine unlearning. \nMoreover, they can control the unlearned information according to the given support image, which seems to be reasonable in reality. \n\n2. The proposed method seems to be effective based on the experiment results. The detailed experiment discussion and analysis are beneficial to understanding the method better.\n\nWeakness:\n1. The Related Work section [1,2,3] mentions some methods for unlearning data in scenarios where the original data is not available. \nHowever, these methods are not included in the main discussion of the paper. \nSince these methods are important, the authors should mention them in the Introduction and clearly explain how the newly proposed benchmark is different. \nThis will help readers understand how this work fits into the existing research and what makes it unique. \n\n2. I am a bit confused about the setting. The paper emphasizes that the original training set is not needed for the machine unlearning. \nHowever, in Figure 3 of the method pipeline, the forget sets $D_f$ appear for the loss calculation. In my understanding, the forget setting is part of the training data, so it is also not available in training.\nIf my understanding is not correct, please add more clarification in the method section to make it clear which part is available in the machine unlearning or not. \n\n3. This method is not specifically designed for face attribute recognition, so I think it is better to include more various datasets to evaluate the method's effectiveness. This can also benefit the benchmark, which can be proven to have more applications other than face recognition.\n\n4. It is better to clarify what is $L_{\\rm task}$ in Eq. 5 for better understanding at least in the Appendix.\n\n\nReferences:\n\n[1] Zero-Shot Machine Unlearning\n\n[2] Fast Yet Effective Machine Unlearning\n\n[3] Zero-Shot Class Unlearning in CLIP with Synthetic Samples", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper emphasizes the problem of unavailable realistic training data which is often required in existing unlearning evaluation benchmarks. \n1-SHUI benchmark is proposed to evaluate machine unlearning without requiring the training data. \nExisting MU methods struggle to tackle the 1-SHUI task, while this paper designs the MetaUnlearn method to tackle the problem, showing its effectiveness on the new benchmark 1-SHUI.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the most important part is the point 2 of the weakness. In the new setting, what data is available for the machine unlearning?", "broader_impact_concerns": "Besides the facial recognition application, this work will be more impactful if there are any other possible applications of such one-shot  machine unlearning tasks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Before rebuttal, all reviewers were positive about the submission, though they raised concerns regarding the clarification of contributions and technology, as well as evaluation. The authors addressed these concerns. All reviewers agreed that the proposed method is effective, with solid benchmark contributions, and recommended acceptance. Thus, the AE also recommends accepting the paper as is.

However, one reviewer does not recommend transferring this paper from ICLR journal track to conference track, reasoning that the proposed unlearning algorithm shows only minimal improvement. The AE concurs with this comment.
Decision: Accept (poster)