Efficient Source-free Unlearning via Energy-Guided Data Synthesis and Discrimination-Aware Multitask Optimization
Authors: Xiuyuan Wang
Chaochao Chen
Weiming Liu
Xinting Liao
Fan Wang
Xiaolin Zheng
Venue: ICML 2025 spotlightposter (2025)
Keywords: source-free unlearning
machine unlearning
unlearn
Primary area: social_aspects->privacy

Abstract: With growing privacy concerns and the enforcement of data protection regulations, machine unlearning has emerged as a promising approach for removing the influence of forget data while maintaining model performance on retain data. However, most existing unlearning methods require access to the original training data, which is often impractical due to privacy policies, storage constraints, and other limitations. This gives rise to the challenging task of source-free unlearning, where unlearning must be accomplished without accessing the original training data. Few existing source-free unlearning methods rely on knowledge distillation and model retraining, which impose substantial computational costs. In this work, we propose the Data Synthesis-based Discrimination-Aware (DSDA) unlearning framework, which enables efficient source-free unlearning in two stages: (1) Accelerated Energy-Guided Data Synthesis (AEGDS), which employs Langevin dynamics to model the training data distribution while integrating Runge–Kutta methods and momentum to enhance efficiency. (2) Discrimination-Aware Multitask Optimization (DAMO), which refines the feature distribution of retain data and mitigates the gradient conflicts among multiple unlearning objectives. Extensive experiments on three benchmark datasets demonstrate that DSDA outperforms existing unlearning methods, validating its effectiveness and efficiency in source-free unlearning.

Efficient Source-free Unlearning via Energy-Guided Data Synthesis and Diserimination-Aware Multitask Optimization
Xiuyuan Wang' Chaochao Chen
Weiming Liu"
Xinting Liao
Fan Wang' Xiaolin Zheng'
Abstract
private information, posing significant privacy risks (Cao & Yang, 2015; Nguyen et al., 2022; Liu et al., 2023a;b). To mitigate these risks, multiple regulations, e.g., the European Union's 's General Data Protection Regulation (GDPR) (Voigt & Von dem Bussche, 2017) and the California Consumer Privacy Act (CCPA) (Pardau, 2018), mandate companies and organizations to implement data deletion mechanisms and grant individuals the right to be forgotten. This gives rise to the field of machine unlearning, whose primary goal is to ensure that the model eliminates all influence of the data requested for deletion (i.e., forget data) while preserv- ing model integrity and performance on the remaining data (i.e., retain data).
With growing privacy concerns and the enforce- ment of data protection regulations, machine un- learning has emerged as promising approach for removing the influence of forget data while maintaining model performance retain data. However, most existing unlearning methods re- quire access to the original training data, which is often impractical due to privacy policies, stor- age constraints, and other limitations. This gives rise the challenging task of source-free unlearning, where unlearning must be accom- plished without accessing the original training data. Few existing source-free unlearning meth- ods rely on knowledge distillation and model retraining, which impose substantial computa- tional costs. this work, we propose the Data Synthesis-based Discrimination-Awaree (DSDA) unlearning framework, which enables efficient source-free free unlearning in two stages: (1) Acceler- ated Energy- Guided Data Synthesis (AEGDS), which employs Langevin dynamics to model the training data distribution while integrating Runge-Kull Kutta methods and momentum enhance efficiency. (2) Discrimination-Ae Aware Multitask Optimization (DAMO), which refines the feature distribution of retain data and mitigates the gra- dient conflicts among multiple unlearning objec- tives. Extensive experiments on three benchmark datasets demonstrate that DSDA outperforms ex- isting unlearning methods, validating its effective- ness and efficiency in source-free unlearning.
Existing methods typically assume access to the training data, utilizing gradient computation or weight adjustment techniques to erase specific knowledge. Some methods estimate the influence of training data on model param- eters, using the Fisher Information Matrix (FIM) (Foster et al., 2024; Golatkar al., 2020) and Hessian (Mehta et al., 2022), which are prohibitively expensive due the high dimensionality of the parameter space. Other methods re- train (Bourtoule et al., 2021; Chundawat et al., 2023a) or fine-tune (Tarun et al., 2023) the model with the training data, intentionally degrading model performance on forgot data while preserving its performance on retain data.
However, access to the original training data cannot be guar- anteed in practical scenarios, due privacy concerns, data retention policies, or other constraints. For example, many cloud platforms delete training data immediately after use to address privacy concerns and storage limitations. Similarly, in streaming data environments, real-time processing over- writes old data with new inputs, preventing historical data retention. Under these cireumstances, inaccessible training data makes most existing unlearning methods infeasible. Consequently, there is a pressing need for unlearning with- out the training data, relying solely the original model and limited auxiliary information (class labels) to perform unlearning, referred to as source-free unlearning.
1. Introduction
Modern Machine Learning (ML) models rely vast amounts of data for training, which may contain sensitive
'Zhejiang University, China. Correspondence Chaochao
Chen
Source-free unlearning is an emerging yet underexplored area, with only few existing methods attempting to tackle its challenges. Specifically, GKT (Chundawat et al., 2023b) and ISPF (Zhang al., 2024) both adopt the Data-Free
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DSDA: Source-free Unlcarning via Data Synthesis and Multitask Learning
Knowledge Distillation (DFKD) technique with a filtering mechanism to selectively transfer knowledge from the origi- nal model to randomly initialized model. However, they both require training a new model from scratch during the distillation process, resulting in significant computational costs.
2. Related Work
2.1. Deep Machine Unlearning
Existing deep unlearning methods can be categorized based on their dependency on training data.
Methods Requiring Full Training Data. Many existing methods depend on access to the complete training data. Exact unlearning methods (Bourtoule et al., 2021; Yan et al., 2022; Kim & Woo, 2022) remove the forget data from the original dataset and retrain the model, incurring high com- putational costs, Approximate unlearning methods bypass complete retraining, with some methods leveraging Hessian matrices (Sekhari et al., 2021; Mehta et al., 2022; Li et al., 2023b) or Fisher Information Matrices (FIM) (Foster et al., 2024; Golatkar et al., 2020) to estimate and reverse the in- fluence of forget data on model parameters, while others (Chundawat et al., 2023a; Thudi et al., 2022; Chen et al., 2024; Li et al., 2023a) fine-tune the original model directly using the training data.
Motivated by the limitations of existing methods, we propose the Data Synthesis-based Discrimination-AW Aware (DSDA) unlearning framework to achieve efficient and effec- tive source-free unlearning, which consists of the following two key stages:
(i) In the first stage, we overcome the unavailability of train- ing data by proposing the accelerated mnerg-guided data synthesis (AEGDS) method generate synthetic datasets. Specifically, we derive an energy function by reinterpreting the output logits of the original model and employ Langevin dynamics to implicitly model the training data distribution. To further improve efficiency, we incorporate high-order Runge- Kutta methods and momentum- based updates into the sampling process, reducing redundant sampling steps while preserving effectiveness.
Methods Requiring Partial Training Data. When retain data is unavailable, Cha et al. (2024); Kim et al. (2024) leverage Projected Gradient Descent (PGD) (Mądry et al., 2017) to generate adversarial samples for the forget classes. SCAR Bonato et al. (2025) substitutes external datasets for the original data, and performs unlearning via Knowledge Distillation (KD). When forget data is unavailable, UNSIR (Tarun al., 2023) generates noise matrices for the forget classes with error- maximizaiiiionn mechanism to induce class-level unlearning. However, all these methods rely on full or partial access to training data, which is often infeasible in real-world scenarios.
(ii) In the second stage, we propose the Discrimination- Aware Multitask Optimization (DAMO) method, which uti- lizes the synthetic datasets for effective unlearning. Through feature space visualization, we observe that traditional un- learning losses disrupt feature distributions, causing retain class samples to become widely dispersed, leading to per- formance degradation. In light of this, we introduce novel unlearning objective, discriminative feature alignment ob- jective, to improve intra-class compactness and inter-class separability of retain classes, thereby improving model per- formance. Additionally, to resolve gradient conflicts arising from optimizing the triple objectives, i.e., forget, retain, and feature alignment, we develop a multitask optimization strategy, ensuring stability and balance in unlearning.
Source-Free Unlearning. In the strictest setting of source- free unlearning, where neither the forget nor retain data is accessible, there are only few existing methods. GKT (Chundawat et al., 2023b) represents the first source-free unlearning method, applying DFKD within an adversarial inversion-and and-distillation framework. Building on GKT, ISFP (Zhang et al., 2024) addresses the over-filtering issue in DFKD by introducing inhibited synthesis to reduce the generation of forgetting- related information. However, both methods rely on DFKD, which necessitates costly and time- consuming model retraining, limiting their practicality for large-scale or real-time unlearning. In this work, we propose the DSDA framework, which overcomes these limitations by leveraging energy-guided data synthesis and efficient fine- tuning through discrimination-ave -wwrre aware multitask optimization.
Our main contributions are summarized as follows:
We propose DSDA, novel two-stage framework for effi- cient, source-free unlcarning, addressing the limitations of methods that rely on original data or incur high computa- tional costs.
For the first stage, we propose AEGDS to efficiently gen- erate synthetic datasets as substitutes for the original data. For the second stage, motivated by insights from feature space analysis, we propose DAMO, an unlearning optimiza- tion method that enhances feature distribution and resolves gradient confficts.
2.2. Model Inversion
Extensive experiments on three benchmark datasets demonstrate DSDA's superiority over existing methods across multiple tasks.
Model inversion (MI) aims to reconstruct training data by exploiting the model's outputs, gradients, or internal repre- sentations (Mahendran & Vedaldi, 2015). Gradient-based MI methods optimize synthetic data using techniques such
2

---

DSDA: Source-free Unlcarning via Data Synthesis and Multitask Learning
as Momentum SGD or Adam (Kingma & Ba, 2014), mini- mizing the loss between model predictions and ground truth (Struppek et al., 2022: Yuan et al., 2023). Alternatively, GMI (Zhang et al., 2020) employs Generative Adversar- ial Networks (GANs) (Goodfellow et al., 2014) to guide data generation, with subsequent works (Yuan et al., 2023; Nguyen et al., 2023; Chen et al., 2021) enhancing generator performance by integrating additional information from the target model. However, these methods are computation- ally expensive, requiring either frequent gradient evalua- tions or training of an auxiliary generator. In contrast, the proposed AEGDS manipulates the model's likelihood land- scape through implicit distribution modeling (Chen et al., 2021), achieving efficient model inversion for source-free unlearning.
3.2. Accelerated Energy-Guided Data Synthesis
Since the original training data is inaccessible, we first in- troduce an energy mechanism to generate synthetic datasets that approximate the distributions of the original data. To further improve efficiency, we propose the AEGDS mechanism, which integrates high-order numerical meth- ods with momentum-basee based updates to reduce computational overhead while preserving the integrity of the generated samples.
Formalizing the Energy Function Energy- models (EBMs) define probability distribution over data samples using an energy function E(x), where the likelihood of sample is inversely proportional to its energy (Grathwohl et al., 2019; LeCun et al., 2006). For an input E X, the energy function E : X' - R. maps each data sample into an energy value, which can be interpreted as an unnormalized probability (Du & Mordatch, 2019). To approximate the original data distribution for each class, the energy-based based model can be constructed from the original M(, 6.) based the observation that discriminative models inherently follow an energy-based framework (Grathwohl et al., 2019). Specifically, the energy function is derived from the logits of the classifier, defined as:
3. Method
In this section, we define the problem of source-free unlearn- ing and introduce two key stages of the DSDA framework, as illustrated in Figure 1. In the first stage, we propose the AEGDS, a method that efficiently generates synthetic datasets as substitutes for the original training data. In the second stage, we propose the DAMO method to perform unlearning using the synthetic datasets. DAMO incorpo- rates a novel discriminative feature alignment objective and resolves gradient conflicts from optimizing multiple unlearn- ing objectives simultancously through multitask optimiza- tion, ensuring stable and balanced unlearning.
Fa v) = log Mxx, 0v)[v),
where M (xx (6)ly) is the predicted probability of the sample x belonging to class y.
3.1. Preliminaries and Notations
First, we formulate machine unlearning problem as follows. Let Drrain [(Itrain ]train)} E X' y denote the train- ing dataset, where X' and y = {1, 2, .... K} are input and class label space, respectively. Let Cy denote the set of classes we intend to forget in a pre-trained ML model and Dy denote the subset of training data corresponding to the forget classes. Similarly, C, denote the set of classes we intend to retain, with the corresponding data subset DD,, satisfying D, = Dirain'Y \Dy. A ML model, represented by ((:6), generates classification probabilities for each class, where 0 is the set of model parameters. M can be de- composed into components: M1, which represents the collection of feature extraction layers, and MS, which repre- sents the final classification layer. Let (-; (.) denote the original model trained on the complete dataset Dtrain- In this paper, we address a challenging yet practical scenario of source-free unlearning, where the unlearned model is gen- erated directly from 0, and class information in C,, without requiring access to D Dirain.
Energy-Guided Data Synthesis (EGDS) By specifying an energy function Egg (2r, y) for a target class y. we can then perform data synthesis following the principles of Stochas- tic Gradient Langevin Dynamics (SGLD) (Welling & Teh, 2011), which generates samples that align with the energy- based model by iterative gradient-guidee updates combined with noise perturbations.
Theorem 3.1. Given the initial data probability g(xr) and the target probability p(x), one can determine the trans- formation process via T(x) x + E by minimizing KL- Divergence as shown:
min Langevin = KL (cr(a))))())
where: T(x) = +
where e denotes the searching direction and the optimal solution on e is given as E =ov, [KL ((r())])(())) where denotes the step size on gradient descend. The result can be further calculated as T(x) = x aBa-g(a) a log)()))
Proof. By taking the differentiation on KT. (1))))))((2
3

---

DSDA: Source-free Unlcarning via Data Synthesis and Multitask Learning
Class labels
Accelerated Energy Guided Datu Synthesis
Synthetic datasets
Iterative sampling 1..
AEGOS Mechanism
20 C,
C
momentum
Dr
Energy function (a,(2
Original model M(:0,)
Random Initialization
Synthetic data sample b
Cr
2nd order approsimation
Dj
Sg
Discrimination- Multitask Optimization
Parameter update
LF
BF
Multitask Optimization
Unlearning model ((:
Dual Unlearning Objectives
max, min, (S.. d), s.t. Bri $ cl
Discriminative Feature Alignment Objective
Ve
Dr
BR
Eu
Iciaa 4)
Feature space
M
h
M
io
V EDioe
Weighted gradient
D
futc owr Futtertter
Gradient conficts!
Figure 1. Overview of the DSDA framework, comprising two stages: (I) Given the original model and forget class labels, construct an energy function and generate synthetic datasets through AEGDS. (2) Synthetic datasets are then used to perform unlearning, incorporating dual unlcarning objectives and novel discriminative feature alignment objective, with multitask optimization for model updates.
w.r.t e, we can obtain the following results:
Accelerated Energy-Guided Data Synthesis (AEGDS) Traditional Langevin dynamics sampling, which fixed time step sequence to update data states iteratively, can be computationally expensive due to frequent gradient evaluations and noise perturbations. While this stepwise progression ensures theoretical soundness, often results in inefficient computations due to computational cost of performing gradient evaluations and noise perturbations at every time step. To address this limitation, propose the AEGDS mechanism, which skips redundant time steps while preserving sampling accuracy.
V. (oT = V. [KL (1=z)]ler->
- -Een((a))
V. log P (a)) det (=T())]
(==
V. log (T))) + V. log det [=T())]
For the first term Langevin Dynamies, calculated V. log (xT())) Va log (() (xx)) and V.T(x:) = I. Therefore, by considering the searching di- rection e with minimizing KL (9T(x) (1)()))l(())) we can con- clude the result as: T(x) x - V. [KL (yT ( (x)))(())) x alE=)=((=) VV log p(=)]
be further
We begin by randomly initializing the synthetic data ai and selecting sequence of time steps {tz [t where f+41-t > O. Starting with the previous data state i at time step 4. the exact solution of 2 the subsequent time step 4++1 is given by:
Inspired by the valuable insights into Langevin dynam- ies presented in Theorem 3.1, propose a novel energy- guided data synthesis mechanism, termed the EGDS mech- anism. Specifically, we randomly initialize the synthetic data & for a target class and then update them iteratively according to the principles of SGLD, where the sampling process is carried out by constructing a Markov chain:
1/ = 2t
-at,
V Eg( (1 dt
Jr:
(4)
that
a 2 at V Eo v) + 20rer
(3)
byy at.
so so . yt.
where 2² and &, are the sample and Gaussian noise at iter- ation time step 1, [o is the sequence of step size. As demonstrated in (Welling & Teh, 2011), by appropriately introducing noise and gradually reducing the step size, this procedure will converge to the distribution defined by the energy function. The pseudocode are shown in Algorithm
where sa(x, v) Bo y). In summary, the process is equal to progressively moving the sample from the ini- tial time step to (x, skipping intermediate steps while still capturing the target distribution.
To numerically approximate the integral, we leverage two-

---

DSDA: Source-free Unlcarning via Data Synthesis and Multitask Learning
stage second-order Runge-Kutta method, which improves accuracy by incorporating intermediate gradient evaluations, reducing the errors of first-order approximations. Specif- ically, the synthetic data at each time step & is updated following:
Algorithm Data synthesis with EGDS/AEGISS
Input: Target class y, score function Sø, sequence of time
steps {t.) step size [ar:)=o momentum decay factor
1: Initialize: 2 ~ (0, I), vo =0
2: for i = to N 1 do
x' = 24, + +ntrd; + 12a,, Cto
if adopt EGDS then
a/ Hi, + onl At: (11
1 di
1
de
(5)
4:
Sample c; ~ (0, 5: Update data: - 4u. + a,,0 + 2azei
21
2m
6: else if adopt AEDGS then
+ v Ft.
7:
Sample ct, NO,,
where d, = so y) (black arrow in Figure 1)). d; = sa v) (grey arrow in Figure 1)), Al, = 1++1- - 4, and 1y is weight parameter. We set 17 1. corresponding to Heun's second-order method (Ascher & Petzold, 1998).
8:
de s 8-80(1)) 50( ,y)
9:
x' < 8t + -1jd1 + v20,, Atre,
10:
di f s0(x',
11:
0.+1 t 1/44 + 3dd
12:
Update momentum: 0+41 s- y0; + or, 00,0
To further accelerate the sampling process, we incorporate Nesterov's 's momentum (Nesterov, 1983) which can improve convergence rates in gradient-based optimization. The mo- mentum U;+1 at each time step is updated as:
13:
Update data:
F4 f
5% + A4u441 +
V2a, At,et,
14: end if
15: end for
Output: Generated sample a% (EDGS) or d (AEDGS)
0++1 = 7/0; + Ole, 0s+1:
(6)
where 8. = 1 + 1d (blue arrow in Figure 1)), is the previous momentum (yellow arrow in Figure 1)) and y a momentum decay factor. The momentum term ef- fectively leverages past gradients to accelerate convergence and smooth the updates. We formulate the overall algo- rithm for AEGDS in Algorithm 1. We leverage the AEGDS mechanism to perform class-wise sampling with parallel computing, generating synthetic data for each class in C, and Cy to form D, and Dy.
the forget classes. This objective is expressed through a forget loss function Lr = ( 0), Ms), where & and yy represent the synthetic forget data and their class labels. The overall unlearning objective is then for- weighted combination of these two objectives: Lr (D, Dy; and A2 are
mulated
y; 0) A Cn 0) + (2 0). where Å1
Discriminative Feature Alignment Objective However, the above objectives exhibit notable limitations when di- rectly optimized. Specifically, we fine-tune the original model using Lr to achieve unlearning. By comparing the feature distributions of fine-tuned ResNetl 8 model on CIFAR-10 10 (shown in Figure 2 (c)) with those of the original and retrained ResNet18 models (shown in Figure 2 (a) and (b)), we draw two key observations: model, samples from the forgot classes scattered near the boundary of retain classes. Moreover, compared to the original model, the retained class samples form compact clusters with clearer separations between classes in the re- trained model. (2) Conversely, in the unlearned model, the forget class samples are more widely dispersed across mul- tiple retained classes, which weakens both the inter-class separability and the intra-class compactness of the retain classes. As a result, the model's 's performance declines due to the disruption in feature distributions.
3.3. Discrimination-Awee Aware Multitask Optimization
After Section 3.2, we have obtained synthetic retain and forget datasets D, and D Then in this section, we propose the DAMO method to fulfill the unlearning task. To begin with, we observe from visualization of future space that the separability and compactness of the retain classes are dis- rupted during unlearning, which motivates the introduction of the discriminative feature alignment objective. To further resolve gradient conflicts among multiple objectives, we develop a multitask optimization strategy for more balanced and effective unlcarning.
(1) In the retrained
Dual unlcarning objectives Following existing studies (Golatkar et al., 2020), We decompose the unlearning pro- cess into two distinct objectives, formalized as follows: The retain objective ensures the model maintains performance on the retain classes. This is expressed by the retain loss function [D( 0) = CCE( M (z,:; 0), )r where ă, rep- resents the synthetic data samples in D, and y, represents the corresponding retain class labels. In contrast, the for- get objective aims to degrade the model's performance on
In light of the above observations and existing studies (Wang et al., 2025), we propose the discriminative feature align- ment objective to improve unlcarning performance. This objective is designed to simultaneously improve intra-class
5

---

DSDA: Source-free Unlcarning via Data Synthesis and Multitask Learning
Class (forget)
Class
Class
Class
(a) Original model
(b) Retrained model
(c) by fine-tuned model
(d) DAMO unlearned model
Figure 2. Feature space visualization of the original, retrained, fine-tuned (using Cr) and unlearned (using DAMO) ResNet18 models on the CIFAR-10 10 datasct. Red dots indicate the samples from Cy, while dots in other colors represent retained samples from the Cr.
compactness by clustering samples of the same class and enhance inter-class separability by pushing apart samples of different classes in the feature space.
Theorem 3.2. Given the gradients {g.), {R, F. Disc). one can find an update direction d that minimizes gradi- ent conflicts while remaining close to the initial combined gradient gr as:
Specifically, given data sample ê, belonging to class y. we define the inter-class similarity as 8/2 o3 h/ /|[]]]]]]|) and the intra-class similarity as Sp = 2g [1 where j C, G. (y) Here, h M (Er; 0) denotes the feature vector of the data sample, and e; denotes the feature center of class i. To achieve discriminative alignment, we introduce the following loss function:
arg max min ((gR d)), (gF, d), (giie: d) d
(8)
s.1. - gr S elggT|.
where e E [0, 1) a hyper-paramene that controls the extent of deviation from the weighted average gradient gr.
To further reduce computational complexity, introduce auxiliary weight variables (WR, WF, WDise) R2, rep- resenting the contributions of each gradient. These weights satisfy E; Wi = 1 and 2 0. Equation (8) can now be reformulated as:
1c-1-1
DDise = log 1
Σ j=1
exp (8ahsh) exp (opppp))
(7)
where d is scale factor, 4% Op are weight factors con- trolling the contribution of inter-class and intra-class simi- larities, respectively. To ensure stability and prevent simi- larity scores from deviating far from their optimal values, we define the weight factors as ag = [+] - Onl+.ap = [Op - spl+, where [+ is the "cut-off at zero" operation that ensures non- non-negativity.
max min Bu d), s.t.ldd gr S clggr (9) w
where gw wRgR + wFgF + WiscegDise represents the weighted gradient direction.
Theorem 3.3. Given the optimization problem in Equa-
tion (9), one can obtain the optimal d by solving the follow- ing Lagrangian:
Multitask Optimization for Unlearning Optimizing the three unlearning objectives- retain, and discrimina- tive feature alignment- simultaneously is challenging due to inherent confficting gradients (Yu et al., 2020). These conflicts prevent all objectives from being optimized si- multaneously, leading to suboptimal performance, such as over-forgetting or under-forgetting. Let gæ VeLR, gF VoLr, and gDise VoL be the gradients of the retain, forget, and discriminative objectives, respectively. The com- bined gradient is defined as gr Aıga+ + Aagr AagDise- A gradient conflict occurs when the combined gradient gr misaligns with any individual gradient, expressed as (g., (8..T) < 0 E { RR F, Disc]. To address this conflict, we propose multitask optimization strategy for unlearn- ing, which resolves gradient conflicts and balances three unlearning objectives effectively.
min A20. max T.T.
2 21BT
d +
(10)
2
Ee 21
where Φ- c² IgT represents constraint the update
magnitude. The optimal solution can be expressed as d' gr + gw />", where w² = arg min w. g. +
Voll [ 1|, X [|g.. 1//0//2.
Proof. We can derive the Lagrangian of Equation (9) as:
> (lgr d| Ø), (11)
max d'
min X Σ. 1,
g
2
where o = e I| represents a constraint on the up- date magnitude. Base on Equation (11), we can derive
6

---

DSDA: Source-free Unlcarning via Data Synthesis and Multitask Learning
Equation (10) due to the concavity of the objective with respect to d and the linearity of its constraints, allowing the interchange of max and min operations. Then, to solve Equation (10), we first fix À and w, and optimize the inner maximization problem. The optimal solution d' is obtained by setting the derivative with respect to d to zero:
Baselines We compare the proposed DSDA with the origi- nal model and the following unlearning methods: (1) Re- train refers to training a model from scratch on the retain data only. (2) SSD (Foster et al., 2024) selectively dampens model parameters according to their importance to the for- get data, as determined by FIM. (3) UNSIR (Tarun et al., 2023) uses error-maxiiziing noise to approximate D, and combines the noise with D to fine-tune the original model. (4) ADV+IMP (Cha et al., 2024) fine-tunes the original model using adversarial examples. (5) LAU (Kim et al., 2024) applys Partial-PGDD and KD to the classification layer. (6) SCAR (Bonato et al., 2025) modifies the feature vector projections of surrogate forget data using metric learning and KD. (7) GKT (Chundawat et al., 2023b) employs a generator to maximize the information gap between teacher and student model, and refines model weights through gated knowledge transfer. (8) ISPF (Zhang et al., 2024) improves upon GKT by introducing the Inhibited Synthetic and Post- Filter methods.
A ad Bu + Ag - 1 |
= gu + AgT Ad= O. (12)
Therefore, we obtain d' = gr + Bw/A X. Substituting d' back into the objective function, the outer minimization problem simplifies to:
ggll | giggl
A
min A20
gT
2 +
(13)
2A
2
23 weg 2,0
Then, we compute X by taking the derivative and solving the resulting condition:
8
|g 12
1 AΦ)
gw 12 2xa
|gTl|
(14)
ax
2x
Implementation Details We implement all experiments in Python 3.9 and use the PyTorch library (Paszke et al., 2019). All experiments are conducted on two NVIDIA RTX 3090 GPUs and repeated three times with different random seeds. Both the original and retrained models are trained from scratch using a multi-step learning rate scheduler, which begins with a learning rate of 0.01, and optimized with the Adam optimizer (Kingma & Ba, 2014). For a fair com- parison, the batch sizes of all methods are set to 256 in R and 32 in ViT. We carefully tune all comparison methods to achieve their best performance.
Setting the derivative to zero, the optimal X is given by x" = -llg l/o. Substituting X back into the objective, we obtain the optimal w' = arg ming w,=1,w20 gT Wl validating the correctness of Theorem 3.3.
Finally, according to Theorem 3.3, the unlearned model can be obtained by applying the optimal update direction to 6, iteratively, which resolves gradient conflicts and maintains balance among the unlearning objectives, thereby improv- ing the overall unlearning performance. Additionally, to clarify the shift in feature distributions after introducing the discriminative feature aligmment objective, we visualize the feature space of the unlearned model using the proposed DAMO method, as shown in Figure 2 (d). The results reveal that the retained class samples exhibit both clear inter-class separability and intra-class compactness, closely aligning with the retrained model.
Evaluation Metrics Following existing studies (Tarun et al., 2023; Cha et al., 2024; Foster et al., 2024), we adopt the following four metrics to measure the overall performance of an unlearning method. (1) Accuracy on forget data (Ay) evaluates the unlearned model's is performance on the forget test data. (2) Accuracy on retain data (A,) measures the unlearned model's performance the retain test data. A, and A, of an unlearned model are expected to get close ac- curacy with the retrained model. (3) Membership inference attack (MIA) evaluates whether any information about the forget data persists in the model. We follow the logistic regression-based MIA implementation proposed in (Chun- dawat al., 2023a; Foster et al., 2024). (4) Execution time (ET) measures the time (in seconds) required produce the unlearned model and complete the evaluation, assessing the timeliness of the unlearning process.
4. Experiments
In this section, we evaluate the effectiveness of DSDA across three benchmark datasets and two model architectures. Ad- ditionally, we conduct ablation experiments to analyze the contribution of its three key components. Finally, we visual- ize the synthetic data to verify its integrity and privacy.
4.1. Experiment Settings
Datasets and Tasks We conduct experiments on CIFAR-10, CIFAR- 100 (Krizhevsky al., 2009) and PinsFaceRecogni- tion (Hereis, 2024) datasets. Following existing studies (Cha
4.2. Comparison with Baselines
To comprehensively evaluate DSDA's performance, we conduct single-class unlearning experiments across three datasets (shown in Table 1) and multi-class unlcarning exper- iments on the CIFAR- 10 dataset (shown in Table 2). Based
et al., 2024; Foster et al., 2024), we adopt ResNet-18 (He et al., 2016) as the backbone for CIFAR-10 and CIFAR-100, and Vision Transformer (ViT) (Dosovitskiy et al., 2021) for FinsFaceReconnition.
7

---

DSDA: Source-free Unlcarning via Data Synthesis and Multitask Learning
Table Evaluation results single-class unlearning. Bolding indicates the best result and underlining indicates the second best result.
Dj
CIFARIO
CIFARI00
PinsFiaceRecognition
Method
free
free
A.(%)
(y6%)
MIA (%)
(s)
A,(%)
Ay(%)
MIA (%6
(s)
A.(%)
Ay(%)
MIA (%)
ET (s)
Original Retrained
79.80
88.83
81.40
58.66
45.00
78.66
90.85
88.57
70.09
77.90
0.00
11.46
1108.10
53.98
0.00
13.22
2243.88
90.89
0.00
0.82
1664.66
SSD
X
x
54.44
0.00
19.72
168.73
43.59
0.00
16.80
160.47
86.76
0.000
30.00
$98.75
UNSIR
V
75.74
0.00
70.48
659.08
40.77
0.00
60.60
173.30
78.21
9.23
20.49
566.96
ADV+IMP
V
x
49.30
0.000
40.68
143.55
47.39
2.20
45.42
180.64
21.01
0.00
68.
731.61
LAU
X
78.37
0.20
33.78
113.10
42.95
0.00
24.00
112.67
87.84
0.00
64.10
201.37
SCAR
V
76.13
1.69
12.50
468.29
48.94
10.00
7.80
452.40
60.35
0.00
21.15
1248.56
GKT
V
44.23
2.19
24.98
393.58
40.74
0.00
38.38
461.31
54.14
2.11
34.87
1941.10
ISPF
/
66.46
0.00
31.70
303.97
23.28
367.50
55.30
32.61
1303.22
DSDA (ours)
V
<
77.91
0.00
11.80
133.40
49.96
0.00
13.80
155.00
88.36
0.00
18.39
549.48
Table Evaluation results for multi-class unlearning on CIFAR-100. h Cforgez denotes the number of forget classes. Bolding indicates the best result and underlining indicates the second best result.
k=2
k=4
k=8
Method
(%6)
Ay6%)
MIA (%)
ET (s)
(%
(%)
MIA (%)
ET (s)
A.(%)
(y6)
MIA (%)
ET (s)
Original
58.35
65.00
92.12
58.84
48.
81.57
57.76
67.51
88.86
Retrained
56.23
0.00
33.21
1960.15
56.98
0.00
14.48
1592.25
57.15
0.00
31.69
1093.86
UNSIR
41.87
0.00
53.02
178.65
42.25
0.00
45.89
374.99
41.66 11.64
0.00
50.08
229.71
ADV+IMP
41.40
3,91
36.60
191.10
0.00
68.33
148.97
34.73
0.00
$3.15
174.93
LAU
45.28
0,50
23.30
174.29
30.51
0.00
67.75
166.06
17.03
0.00
18.55
168.39
SCAR
52.01
13.50
20
136.79
42.39
13.50
34,00
302.61
40.89
25.00
35.18
322.40
GKT
40.57
0.00
33.65
568.81
32.74
37.95
895.39
25.39
39,38
973.95
ISPF
44.81
0.00
24.69
446.70
45.47
0.00
24.95
480.37
44.00
0.00
38.83
$71.62
DSDA (ours)
49.55
0.00
33.28
157.25
50,28
0.00
17.80
147.01
49.56
0.00
28.29
162.55
on the objective of machine unlearning, we expect the per- formance of a desirable unlearning method to be close to the performance of the Retrain baseline. From the experimen- tal results, we draw the following conclusions, considering both unlearning effectiveness and efficiency.
90
(%6
50
60
4,
CKT
30
CKI
SPP
ESF
DSCA
DSDA
Unlearn Effectiveness. (1) DSDA achieves complete re- moval of forget data information, with A, reaching 0% across all unlearning tasks, while also attaining the highest A, among all source-free unlearning methods. Additionally, DSDA demonstrates the best or second- best accuracy per- formance compared to all baselines. Note that this is not a fair comparison, as source-free free methods function without the original training data, making their unlearning task more challenging. (2) Several baselines, including ADV+IMP and UNSIR, exhibit significantly higher MIA values than Retrain, indicating potential leakage of forget data, despite achieving desirable accuracy. DSDA achieves the best MIA results, closely aligning with the retrained model, further demonstrating its effectiveness in mitigating privacy risks. (3) Moreover, as the number of forgotten classes increases, LAU and GKT experience notable performance degradation. In contrast, DSDA maintains robust performance regardless of the number of forget classes, underscoring its scalability and reliability in more complex unlearning scenarios.
120
240
360
600
1200
1800
ET (s)
(S)
(a) CIFAR-10
(b) Pinnoaaceeooonniiion
Figure 3. The results ET A, under CIFAR-10 with ResNet18 and PinsFaceRecognition with VIT settings.
Furthermore, DSDA also surpasses most ses--suur--eeee baselines in efficiency, despite the additional step of gener- ating synthetic data to substitute the original training data.
Moreover, we analyze the relationship between A, and ET for source-free unlearning methods, shown in Figure 3. In the figure, the A, values of DSDA remain constant before certain point, which is due to the model being frozen during the data synthesis stage. The results demonstrate that DSDA consistently achieves higher A, than GKT and ISPF, while reaching optimal performance more efficiently.
4.3. Ablation Study
Unlearn Efficiency. The ET results demonstrate that DSDA significantly outperforms source-free baselines in unlearn- ing efficiency, with an average improvement of 68.50%.
We conduct ablation experiments on three key components in the DSDA, to elucidate their contributions respectively. Specifically, DSDA-w-EGA represents DSDA without data
8

---

DSDA: Source-free Unlcarning via Data Synthesis and Multitask Learning
cisc
AEGDS effectively models the original data distribution.
80
250
Visualization of Synthetic Data. To further evaluate the privacy implications of the synthetic data, we visualize its appearance in Figure (b). The synthetic samples are visu- ally indistinguishable, impossible for human observers to extract any meaningful information, which ensures that the synthetic data poses no privacy risks.
Ar. 1%6)
20000
50-
CIFAR-10
CIFAR-100
CIFAR-10
CIFAR-100
Figure 4. The ablation results of A, and ET on CIFAR-I0 and
5. Conclusion
CIFAR-100 using ResNet18.
We propose DSDA, a novel source-free unlearning frame- work, which addresses the critical challenges of inaccessible training data and computational cost. The two key compo- nents of DSDA, i.e. AEGDS and DAMO, enable the genera- tion of synthetic data and refinement of feature distributions, thereby ensuring both the removal of forgot knowledge and the preservation of performance retain data. Extensive periments on multiple benchmark datasets demonstrate that DSDA outperforms existing unlearning methods in terms of both efficiency and effectiveness. While DSDA effective in removing class-level information, remains limited in finer- grarnerained unlearning scenarios, such as instance- wise or attribute- wise unlearning. Exploring source-free unlearn- ing methods tailored to such fine-grained unlearning tasks presents an important direction for future work.
tng org ong dlsss ong dasp clee das dlo
(a) Feature distribution synthetic data original data
(b) Visualization of synthetic data
Figure 5. Visualization of feature distribution and synthetic data for CIFAR-10. 10. Round dots represent original data, square dots represent synthetic data.
synthesis acceleration. DSDA-wo-dis represents DSDA without the discrimination feature aligmment objective, us- ing only Lr and L. DSDA-wo-mi represents DSDAA with- out multi-task optimization, where model parameters are updated directly using gr.
A ckkownleggement
This work was supported part by the National Natural Science Foundation of China (No.62172362).
The results of A, and ET on CIFAR-10 and CIFAR-100 datasets, as shown in Figure 4, provide several important insights: (1) ASDAA--EGGDS requires significantly longer ET than DSDA, highlighting the importance of AEGDS in improving efficiency. Additionally, DSDA outperforms DSDA-w-EGDS in A,, due to the second-order Runge- Kutta method reducing approximation errors. (2) DSDA- wo-disc shows lower A, compared to DSDA, underscoring the critical role the alignment objective in preserving intra-class compactness and inter-class separability, which (3) DSDA-wo- mt leads lower A, and higher ET, demonstrating that multi-task task optimization not only balances the unlearning objectives but also accelerates the unlearning process.
Impact Statement
This paper presents a novel approach to source-free un- learning, advancing the field of machine learning with the proposed DSDA framework, which enhances both efficiency and privacy unlearning tasks. Our work has the potential to positively impact the field of Machine Learning, particu- larly in scenarios of data privacy.
References
in turn leads to better model performance.
Ascher, U. M. and Petzold, L. R. Computer methods for ordinary differential equations and differentiallalgebrric equations. SIAM, 1998.
Bonato, J., Cotogni, M., and Sabetta, I.. retain set all you need in machine unlearning? restoring performance of un- learned models with out-of-distribution images. In Euro- pean Conference Computer Vision, pp. 1-19. Springer, 2025.
4.4. Additional Analysis
Feature distribution of Synthetic Data. We visualize the feature distribution of both the synthetic and original CIFAR- 10 dataset using the original model, as shown in Figure 5. The results reveal that the synthetic data closely overlaps with the original data in feature space, exhibiting nearly identical distributions, which suggests that the proposed
Bourtoule, L., Chandrasckaran, V., Choquette-Chneee C. A.,
Jia, H., Travers, A., Zhang, B., Lie, D., and Papernot,

---

DSDA: Source-free Unlcarning via Data Synthesis and Multitask Learning
N. Machine unlearning- In 202/ JEEE Symposium on Security and Privacy (SP), pp. 141-159, 2021.
27th International Conference on Neural Information Processing Systems, pp. 2672-26801 2014.
Cao, Y. and Yang. J. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy. pp. 463 463-480, 2015.
Grathwohl, W., Wang, K.- Jacobsen, J.-H., Duvenaud, D., Norouzi, M., and Swersky, K. Your classifier secretly an energy based model and you should treat like one. In International Conference on Learning Representations, pp. 1-23, 2019.
Cha, S., Cho, S., Hwang, D., Lee, H., Moon, T., and Lee, M. Learning to unlearn: Instance-wine unlearning for pre- trained classifiers. Proceedings the AAAI Conference on Artificial Intelligence, pp. 11186-111944 2024.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778,201
Chen, C., Zhang, Y., Li, Y., Wang, J., Qi, L., Xu, X., Zheng, X., and Yin, J. Post-training attribute unlearning in rec- ommender systems. ACM Transactions on Information Systems, 43(1): 1-), 2024.
Hereis, B. Pins face recognition dataset, 2024. URL https://wwwt hwwps////wwww.gllleee.e wtt.ccc///a/////e /etetssss hereisburak/pins face recognition.
Chen, S., Kahla, M., Jia, R., and Qi, G.-J. Knowledge- enriched distributional model inversion attacks. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. (6178-161877 2021.
Kim, H., Lee, S., and Woo, S. S. Layer attack unlearning: Fast and accurate machine unlearning via layer level at- tack and knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 2024.
Chundawat, V. S., Tarun, A. K., Mandal, M., and Kankan- halli, M. Can bad teaching induce forgetting? unlearn- ing in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pp. 7210-7217, 2023a.
pp.
Kim, J. and Woo, S. S. Efficient two-stage model retraining for machine unlearning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3361-4369, 2022.
Chundawat, V. S., Tarun, A. K., Mandal, M., and Kankan- halli, M. Zero-shot machine unlearning. IEEE Transac- tions on Information Forensics and Security, pp. 2345- 2354, 2023b.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, pp. 1-15, 2014.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition scale. In 9th International Conference on Learning Representations, ICLR 2021, pp. 1 1-21, 2021.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers features from tiny images. Master's thesis, University of Tront, pp. 1-60, 2009.
LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., Huang, F., et al. A tutorial on energy- energy-based learning. Predicting structured data, 1(0):1-59, 2006.
Du, Y. and Mordatch, I. Implicit generation and modeling with energy- based models. In Proceedings of the 33rd International Conference on Neural Information Process- ing Systems, pp- 3608-3618, 2019.
Li, Y., Chen, C., Zhang, Y., Liu, W., Lyu, L., Zheng, X., Meng, D., and Wang, J. Ultrare: Enhancing receraser for recommendation unlearning via error decomposition. Advances in Neural Information Processing Systems, 36: 12611-12622 2023a.
Foster, J., Schoepf, S., and Brintrup, A. Fast machine un- learning without retraining through selective synaptic dampening. Proceedings of the AAAI Conference on
Li, Y.. Chen, C., Zheng, X., Zhang, Y., Gong, B., Wang, J., and Chen, L. Selective and collaborative influence function for efficient recommendation unlearning. Expert Systems with Applications, 234:121025, 2023b.
Artificial Intelligence, pp.
2024.
Golatkar, A., Achille, A., and Soatto, S. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9312, 2020.
Liu, W. Chen, C., Liao, X., Hu, M., Yin, J., Tan, Y., and Zheng, L. Federated probabilistic preference distribution modelling with compactness co-clustering for privacy- preserving multi-domain recommendation. In Proceed- ings of the Thirty-Second International Joint Conference on Artificial Intelligence, pp. 2206-2211 2023a.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Farle, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. In Proceedings of the
10

---

DSDA: Source-free Unlcarning via Data Synthesis and Multitask Learning
Liu, W. Zheng, X., Chen, C., Hu, M., Liao, X., Wang, E,, Tan, Y., Meng, D., and Wang, J. Differentially pri- vate sparse mapping for privacy-preserving cross domain recommendation. In Proceedings of the 31st ACM in- ternational conference on mulsimedia, pp. 6243-6252, 2023b.
Tarun, A. K., Chundawat, V. S., Mandal, M., and Kankan- halli, M. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-10, 2023.
Thudi, A., Deza, G., Chandrasekaran, V., and Papernot, N. Unrolling sgd: Understanding factors influencing ma- chine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), PP. 303-319, 2022.
Madry, A., Makelov, A., Schmidt, L.., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. stat, 1050:9, 2017.
Voigt, P. and Von dem Bussche, A. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10:10-5555, 2017.
Mahendran, A. and Vedaldi, A. Understanding deep im- age representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5188-5196, 2015.
Wang. F., Qi, L., W., Yu, B., Chen, J., and Xu, Y. Inter- and intra- similarity preserved counterfactual incentive effect estimation for recommendation systems. ACM Transactions on Information Systems, 2025.
Mehta, R., Pal, S., Singh, V., and Ravi, S. N. Deep unlearn- ing via randomized conditionally independent hessians. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 10422-10431, 2022.
Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning, pp. 681- 688. Citeseer, 2011.
Nesterov, A method for solving the convex programming problem with convergence rate o (1/k2). In Dokl akad nauk Sssr, volume 269. pp. 543, 1983.
Yan, H., Li, X., Guo, Z., Li, Li, and Lin, X. Arcane An efficient architecture for exact machine unlearning. In Proceedings of the Thirty-First International Joint Con- ference on Artificial Intelligence, pp. 4006-4013, 2022.
Nguyen, N.-B., Chandrasegaran, K., Abdollahzadeh, M., and Cheung, N.-M. Label-only model inversion attacks via knowledge transfer. In Proceedings of the 37th Inter- national Conference on Neural Information Processing Systems, pp.
2023.
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824-5836, 2020.
Nguyen, T., Huynh, T. T., Nguyen, P. L., Liew, A. W.-C., Yin, H., and Nguyen, Q. V. H. A survey of machine unlearning. arXiv preprint arXiv:2209.02299 pp. 1 1-24, 2022.
Yuan, X., Chen, K., Zhang. J., Zhang, W., Yu, N., and Zhang, Y. Pseudo label-guided model inversion attack via con- ditional generative adversarial network. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 37, pp. (3349-335] 2023.
Pardau, S. L. The california consumer privacy act: Towards european-style privacy regime in the united states. J. Tech. L. & Pol'y, 23:68, 2018.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, et al. Pytorch: imperative style, performefo deep learning library. In Proceedingy of the 33rd Inter- national Conference on Neural Information Processing Systems, pp. 8026-8037, 2019.
Zhang, C., Shen, S., Chen, W., and Xu, M. Toward efficient data-free unlearning. arXiv preprint arXiv:2412.13790. pp. 1-15, 2024.
Zhang, Y., Jia, R., Pei, H., Wang, W., Li, B., and Song, D. The secret revealer: Generative model-inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 253-261, 2020.
Sckhari, A., Acharya, J., Kamath, G., and Suresh, A. T. Re- member what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, pp. 18075-18086, 2021.
Struppek, L.,, Hintersdorf, D., Correira, A. D. A., Adler, A., and Kersting, K. Plug & play attacks: Towards robust and flexible model inversion attacks. In International Con- ference on Machine Learning, pp. 2022.
PMLR,


Reviews:
[{"review_id": "fyxaDTmfMS", "reviewer": "Reviewer_gZDa", "summary": "The authors present a well-structured and innovative approach to source-free unlearning, where a trained model must forget specific data without access to the original training dataset. To achieve this, the authors propose a novel two-stage framework DSDA. Firstly, the proposed AEGDS generates synthetic dataset as a substitute for the original ones using Langevin dynamics, enhanced with Runge–Kutta and momentum-based acceleration. Secondly, based on findings of the unlearned feature distribution, the authors introduce a novel discrimination-aware unlearning objective and perform balanced optimization to achieve unlearning. The authors conduct adequate experiments on three datasets and multiple unlearning tasks. Results show that DSDA outperforms existing source-free methods and is comparable to general methods, in terms of efficiency and effectiveness.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***W1: Adding a notation table will make the paper clearer.***\n\nWe thank the reviewer for pointing out this issue. We will add a notation table in our final version.\n\n***Q1: Could the synthetic data cause potential privacy linkage?***\n\nThe reviewer raises a critical concern. However, our work addresses this concern through an empirical analysis presented in Section 4.4. Specifically, we visualize the generated synthetic samples and demonstrate that they are visually indistinguishable from random noise, making it impossible for human observers to extract any meaningful information. This observation confirms that the synthetic data does not retain identifiable features of the original training data, thereby mitigating potential privacy risks.\n\n***Q2: Parameter sensitivity experiments w.r.t. weighting factors.***\n\nThe weighting factors in our MAMO method are set to ensure that the three losses are on a similar scale. To analyze their sensitivity, we conduct experiments of one-class unlearning task on CIFAR-10. We fix the weight of the retain loss $w_R$ at 0.1 and vary $w_F$ and $w_{Disc}$ within {0.01, 0.1, 0.5, 1}, observing the unlearned model’s $A_r$ and $A_f$. The results in the table below show that when $w_F$ and $w_{Disc}$ are within the range of 0.1–0.5, the model's performance remains stable. However, an excessively large $w_F$ or a small $w_{Disc}$ leads to a noticeable decline in $A_r$.\n\n| $w_R$| $w_F$ | $w_{Disc}$ | $A_r$ | $A_f$ |\n| ---- | ---- | ---- | ---- | ---- |\n| 0.1 | 0.01 | 0.01 | 71.68 | 0.00 |\n| 0.1 | 0.1 | 0.01 | 74.60 | 0.00 |\n| 0.1 | 0.5 | 0.01 | 74.23 | 0.00 |\n| 0.1 | 1 | 0.01 | 73.14 | 0.00 |\n| 0.1 | 0.01 | 0.1 | 76.92 | 0.00 |\n| 0.1 | 0.1 | 0.1 | 79.25 | 0.00 |\n| 0.1 | 0.5 | 0.1 | 78.07 | 0.00 |\n| 0.1 | 1 | 0.1 | 77.03 | 0.00 |\n| 0.1 | 0.01 | 0.5 | 74.38 | 0.00 |\n| 0.1 | 0.1 | 0.5 | 78.35 | 0.00 |\n| 0.1 | 0.5 | 0.5 | 78.05 | 0.00 |\n| 0.1 | 1 | 0.5 | 77.52 | 0.00 |\n| 0.1 | 0.01 | 1 | 72.46 | 0.00 |\n| 0.1 | 0.1 | 1 | 77.67 | 0.00 |\n| 0.1 | 0.5 | 1 | 78.02 | 0.00 |\n| 0.1 | 1 | 1 | 77.26 | 0.00 |", "claims_and_evidence": "Yes, the claims are well supported.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem or application at hand.", "theoretical_claims": "Yes (e.g. theorem 3.1 and 3.3).", "experimental_designs_or_analyses": "Yes. Detailed in “Other strengths and weaknesses”.", "relation_to_broader_scientific_literature": "The paper also contribute to the fields of model inversion and multitask learning.", "essential_references_not_discussed": "There are no related works not discussed.", "other_strengths_and_weaknesses": "Strengths :\n\n1.\tThe authors presents and addresses the pressing problem of source-free unlearning, a critical challenge in real-world machine unlearning scenarios where access to training data is restricted. The proposed framework offers a novel and efficient solution, surpassing existing source-free methods that rely on knowledge distillation.\n2.\tThe authors provides insightful visualizations of feature space to illustrate the impact of unlearning, highlighting how traditional methods disrupt feature distributions. These insights empirically justify the need for discriminative feature alignment, strengthening the theoretical motivation.\n3.\tThe metrics in the experiment are comprehensive, including accuracy, efficiency and defensive capability.\n4.\tThe authors conduct ablation experiments to isolate the contributions of each component of DSDA, clearly demonstrating the importance of each part of the framework in achieving the overall performance improvements.\n5.\tThe writing is clear and well-structured. In particular, the authors clearly introduce the core components AEGDS and DAMO with intuitive figures, algorithmic pseudocode, and detailed explanations, ensuring that the methodology is easy to understand and implement.\n\nWeaknesses :\n\n1.\tBecause the METHOD is written with many symbols and equations, adding a notation table will make the paper clearer.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "1. As the proposed AEGDS generates synthetic data that approximates the original training distribution, I am concerned that could the synthetic data cause potential privacy linkage?\n2. The authors mention weighting factors for multitask optimization, but provide limited discussion on their impact. Could they provide insights or experiments showing how these factors affect the framework’s performance and balance between unlearning objectives?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hfhFAWpZ05", "reviewer": "Reviewer_nSY3", "summary": "The paper addresses the challenge of source-free unlearning for image classification ML models, where the training data cannot be accessed after initial model training. The paper proposes a novel framework called DSDA, which utilizes Langevin dynamics, Runge–Kutta methods and gradient-based multitask optimization to achieve source-free unlearning. The paper demonstrate that DSDA achieves superior efficiency and effectiveness through experiments on three datasets. The paper also conducts ablation studies for key components and presents visualization analysis of the synthetic data.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***W1: A deeper discussion on the synthetic data's limitations would strengthen the argument.***\n\nWe appreciate the reviewer’s suggestion. Our experiments on three datasets with 10, 100, and 105 classes demonstrate that the synthetic data effectively supports unlearning across diverse distributions. As shown in Section 4.4, the visualized feature distributions further confirm that the generated data closely approximates real data across various classes. However, we acknowledge that for certain distributions, such as highly sparse or imbalanced data, our method may not fully capture fine-grained structural details. Addressing this limitation is an important direction for future research.\n\n***W2: The legend in Figure 5 (a) appears to be small, which may affect readability. Additionally, the color contrast could be improved to enhance clarity.***\n\nWe thank the reviewer for pointing out this issue. We will optimize Figure 5 (a) in our final version.", "claims_and_evidence": "The authors provide adequate theoretical and experimental evidence for their claims.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are well-suited and relevant for the problem.", "theoretical_claims": "Yes. The theoretical claims regarding the AEGDS and DAMO are well-supported.", "experimental_designs_or_analyses": "Yes. The experimental design is adequate, including comparisons with sota methods, ablation studies and visualizations.", "relation_to_broader_scientific_literature": "The paper contributes to the emerging field of source-free unlearning, which aims to remove specific data influence from a trained model without access to the original training dataset. The findings on feature distribution changes after unlearning contribute to the growing body of work on explainable AI and model interpretability.", "essential_references_not_discussed": "The authors well discussed existing works on machine unlearning.", "other_strengths_and_weaknesses": "Strengths :\n\n1.\tThe paper addresses a practical and critical problem of source-free unlearning. Unlike existing methods that either require access to training data or incur high computational costs, the proposed framework introduces an efficient alternative by leveraging energy-guided data synthesis and multitask optimization.\n\n2.\tThe incorporation of Langevin dynamics-based sampling for synthetic data generation is an innovative approach that bridges model inversion, generative modeling, and unlearning. With the proposed AEGDS method, DSDA directly reconstructs data distributions without external generators, enhancing both privacy protection and computational efficiency.\n\n3.\tThe paper includes extensive experiments across multiple datasets and compare DSDA with sota baselines, demonstrating its superiority in both unlearning effectiveness and efficiency.\n\nWeaknesses :\n\n1.\tA more detailed discussion on the potential limitations of the synthetic data (e.g., its ability to generalize to different data distributions) would enhance the robustness of the argument.\n\n2.\tThe legend in Figure 5 (a) appears to be small, which may affect readability. Additionally, the color contrast could be improved to enhance clarity.", "other_comments_or_suggestions": "See Weaknesses for details.", "questions_for_authors": "Please response to the weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XOmfY4bbIC", "reviewer": "Reviewer_NZ7b", "summary": "This paper proposes a new framework, DSDA, for machine unlearning without access to the training data.\n\nSpecifically, DSDA first crafts synthetic data via energy-based models with Langevin dynamics and then performs unlearning using the generated synthetic data. \n\nThe authors observe that simply formulating the unlearning problem as $\\lambda\\_1 \\mathcal{L}\\_R + \\lambda\\_2 \\mathcal{L}\\_F$ would let retain class samples become dispersed and disrupt feature distributions; therefore, DSDA include a feature alignment objective $\\mathcal{L}\\_{Disc}$ to improve intra-class compactness and inter-class separability of retain classes.\n\nFurthermore, since the objective now involves $\\mathcal{L}\\_R, \\mathcal{L}\\_F$ and $\\mathcal{L}\\_{Disc}$, gradient conflicts may happen during the unlearning process. Hence, DSAD employ a multitask optimization strategy to make sure the update vector is close to the joint gradient.\n\nExperiments on CIFAR-10, CIFAR-100, and PinsFaceRecognition datasets across CNN and ViT demonstrate the effectiveness of the proposed framework DSDA compared to baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***W1: This mechanism may be limited on high-resolution data.*** \n\nThank you for the comment. However, we respectfully disagree. Our method is not restricted to high-resolution data, as demonstrated by our experiments on three datasets with varying resolutions—CIFAR-10 (32×32), CIFAR-100 (32×32), and PinsFaceRecognition (224×224). The consistent performance across both low- and high-resolution scenarios validates the broad applicability of our method.\n\n***W2&C3: The proposed framework is limited to class-wise forgetting.***\n\nWe appreciate the reviewer’s insight. Our method is designed for class-wise unlearning, as it aligns with common unlearning scenarios and is more practical in real-world applications. Most existing unlearning studies also focus on class-level or concept-level unlearning, as instance-level unlearning poses inherent challenges—removing a specific instance does not guarantee that the model will completely forget similar patterns, as retain data with similar features may still contribute to strong performance on the forgot data. While our method is effective in removing class-level information, we recognize its limitations in finer-grained unlearning, such as instance-wise or attribute-wise unlearning, which we will clarify in our final version.\n\n***C1: Why not adopting contrastive alignment for the discriminative feature alignment objective?***\n\nThe contrastive alignment methods in [1-2] employ a metric learning loss that is fundamentally similar to our $L_{disc}$ without the weight factors $\\alpha$. However, our method introduces $\\alpha$ to adaptively adjust gradient magnitudes based on how far a similarity score deviates from its optimal values.\nThis adaptive weighting mechanism provides two key advantages. First, it prioritizes large updates for highly misaligned features, accelerating convergence and improving optimization efficiency. Second, it prevents excessive penalization of already well-aligned features, reducing the risk of overcorrection. As a result, our method achieves more precise and stable discriminative feature alignment, ultimately enhancing unlearning effectiveness.\n\n***C2: Explain rationale behind the choice of the solution to gradient conflicts.***\n\nThe method in [3] primarily balances two losses and requires a predefined distinction between the main and auxiliary objectives, limiting its applicability to more complex multi-objective optimization. In contrast, our approach can effectively balance three or more objectives without the need for a manually designated main objective, enabling a more flexible and adaptive optimization process. This allows for the seamless integration of the Discriminative Feature Alignment Objective and ensures a better trade-off among multiple competing objectives in unlearning.\n\n***Q1: Regarding the objective LF, what if DSDA apply random labelling? Would it help with feature alignment?***\n\nWe appreciate the reviewer’s thought-provoking question. However, random labeling may not be an effective alternative for the forget objective $L_F$. As shown in Figure 2(b), most forget data samples are predicted as specific classes rather than randomly distributed across all classes. Additionally, prior research [1] has shown that fine-tuning a trained model with randomly labeled forget data can introduce unintended shifts in the decision boundaries of retain classes, ultimately degrading the model’s utility on the retain data. Therefore, the forget objective  $L_F$, which explicitly optimizes unlearning through gradient ascent, is more controlled and effective in achieving the desired unlearning behavior.\n\n[1] Chen, Min, et al. \"Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n***Q2: What is the data range for the crafted data?***\n\nThe crafted data follows a similar range to the preprocessed original data. For example, in the case of CIFAR-10, we apply a common normalization setting with CIFAR_MEAN = (0.5071, 0.4865, 0.4409) and CIFAR_STD = (0.2673, 0.2564, 0.2762), resulting in a transformed data range of approximately [−2,2]. Our synthetic data is generated within a slightly extended range of [−2.5,2.5], ensuring compatibility with the model while maintaining sufficient variability for effective unlearning.", "claims_and_evidence": "In general, the claims made in the submission are clear and convincing.\n\nHowever, regarding the observation that simply formulating the unlearning problem as $\\lambda\\_1 \\mathcal{L}\\_R + \\lambda\\_2 \\mathcal{L}\\_F$ would let retain class samples become dispersed and disrupt feature distributions, I recommend the authors provide more evidence to support that.\n\nIn the submission, an empirical result on CIFAR-10 when forgetting class 1 is provided. However, $\\lambda\\_1, \\lambda\\_2$ may affect the results, as well as the forgotten class, \\i.e., for scenarios where classes having hierarchical categories may affect the distributions and be more difficult.", "methods_and_evaluation_criteria": "Yes, the proposed framework DSDA is important to improve the practicality of machine unlearning as training data may not be available, and DSDA not need to collect auxiliary data.", "theoretical_claims": "The proof seems correct.", "experimental_designs_or_analyses": "The experimental setting is suitable and correct.\n\nPlease refer to Claims And Evidence* for the concerns on the analysis of observations.", "relation_to_broader_scientific_literature": "Compared to existing source-free machine unlearning methods, the key contribution of this work is that there is no need to collect auxiliary data and no need to train an auxiliary model/model retraining.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The proposed DSDA crafts synthetic data via energy-based models with Langevin dynamics, providing a fresh perspective on source-free machine unlearning. However, this mechanism may be limited on high-resolution data.\nIn addition, the proposed framework is limited to class-wise forgetting.", "other_comments_or_suggestions": "- It would be better to explain why not adopting contrastive alignment [1-2] for the discriminative feature alignment objective, as the former can also obtain the objective and can drop weight factors $\\alpha$. \n\n- [3] discusses the gradient conflict issue in machine unlearning and employs an optimization method to resolve the issue. It would be better to explain the rationale behind the choice of the solution to gradient conflicts in the submission.\n\n- It would be better to admit and discuss limitations such as class-wise forgetting in the paper.\n\n\nI am willing to raise my score if the authors can address my main concerns.\n\n----\n> [1] Liu, Qingxiang, et al. \"Personalized Federated Learning for Spatio-Temporal Forecasting: A Dual Semantic Alignment-Based Contrastive Approach.\" arXiv preprint arXiv:2404.03702 (2024).\n>\n> [2] Tan, Yue, et al. \"Is heterogeneity notorious? Taming heterogeneity to handle test-time shift in federated learning.\" Advances in Neural Information Processing Systems 36 (2023): 27167-27180.\n>\n> [3] Wu, Jing, et al. \"Erasediff: Erasing data influence in diffusion models.\" arXiv preprint arXiv:2401.05779 (2024).", "questions_for_authors": "- Regarding the objective $\\mathcal{L}_F$, what if DSDA apply random labelling? Would it help with feature alignment?\n\n- What is the data range for the crafted data?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "WR4KoHT0of", "reviewer": "Reviewer_bm1b", "summary": "The authors propose the DSDA source-free unlearning framework the address the challenge of inaccessible original training data. DSDA consists of two key components: (1) AEGDS generates synthetic data using Langevin dynamics, and (2) DAMO balances unlearning objectives by resolving gradient conflicts. Extensive experiments demonstrate that DSDA outperforms existing source-free unlearning methods in terms of both efficacy and efficiency.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***W1: Incorporating quantitative metrics (such as FID) to evaluate the soundness of synthetic data could strengthen the paper.*** \n\nWe respectfully disagree with the reviewer’s suggestion. We emphasize that our goal is to generate synthetic data that approximates the original distribution while preserving privacy. Unlike model inversion methods, which aim to maximize image fidelity, our approach prioritizes unlearning effectiveness without reconstructing identifiable samples. Therefore, evaluating the synthetic data with metrics like FID, which emphasize visual quality, is not necessary for our objectives.\n\n***W2: Parameter sensitivity experiments w.r.t. weighting factors.***\n\nThe weighting factors in our MAMO method are set to ensure that the three losses are on a similar scale. To analyze their sensitivity, we conduct experiments of one-class unlearning task on CIFAR-10. We fix the weight of the retain loss $w_R$ at 0.1 and vary $w_F$ and $w_{Disc}$ within {0.01, 0.1, 0.5, 1}, observing the unlearned model’s $A_r$ and $A_f$. The results in the table below show that when $w_F$ and $w_{Disc}$ are within the range of 0.1–0.5, the model's performance remains stable. However, an excessively large $w_F$ or a small $w_{Disc}$ leads to a noticeable decline in $A_r$.\n\n| $w_R$| $w_F$ | $w_{Disc}$ | $A_r$ | $A_f$ |\n| ---- | ---- | ---- | ---- | ---- |\n| 0.1 | 0.01 | 0.01 | 71.68 | 0.00 |\n| 0.1 | 0.1 | 0.01 | 74.60 | 0.00 |\n| 0.1 | 0.5 | 0.01 | 74.23 | 0.00 |\n| 0.1 | 1 | 0.01 | 73.14 | 0.00 |\n| 0.1 | 0.01 | 0.1 | 76.92 | 0.00 |\n| 0.1 | 0.1 | 0.1 | 79.25 | 0.00 |\n| 0.1 | 0.5 | 0.1 | 78.07 | 0.00 |\n| 0.1 | 1 | 0.1 | 77.03 | 0.00 |\n| 0.1 | 0.01 | 0.5 | 74.38 | 0.00 |\n| 0.1 | 0.1 | 0.5 | 78.35 | 0.00 |\n| 0.1 | 0.5 | 0.5 | 78.05 | 0.00 |\n| 0.1 | 1 | 0.5 | 77.52 | 0.00 |\n| 0.1 | 0.01 | 1 | 72.46 | 0.00 |\n| 0.1 | 0.1 | 1 | 77.67 | 0.00 |\n| 0.1 | 0.5 | 1 | 78.02 | 0.00 |\n| 0.1 | 1 | 1 | 77.26 | 0.00 |", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence", "methods_and_evaluation_criteria": "The proposed method is well-suited to source-free unlearning. The evaluation metrics and datasets are widely used", "theoretical_claims": "Yes. I check the proofs for key theorems (Theorem 3.1 on Langevin dynamics and Theorem 3.3 on multitask gradient optimization)", "experimental_designs_or_analyses": "Yes. The weaknesses are detailed below", "relation_to_broader_scientific_literature": "This paper contributes to the under-explored challenge of source-free unlearning, a topic with increasing relevance due to privacy concerns and regulatory demands", "essential_references_not_discussed": "The paper thoroughly cite and discuss the related works necessary to understand its key contributions.", "other_strengths_and_weaknesses": "Strengths\n\n1.\tThe authors propose an innovative and practical solution to a pressing and under-explored problem in machine learning.\n2.\tThe findings on feature distribution shifts after unlearning provide compelling evidence for the need for discriminative feature alignment. The observation that traditional unlearning methods disrupt intra-class compactness and inter-class separability highlights a previously overlooked issue.\n3.\tExtensive experiments on multiple benchmark datasets strongly support the proposed method. Additionally, the ablation studies illustrate the individual contributions of AEGDS, alignment objectives, and multitask optimization.\n4.\tThe paper is well-organized, with clear theoretical foundations and thorough experimental validation.\n\nWeaknesses\n1.\tThe authors claim that the synthetic data closely approximates the original data distribution using feature distribution overlap and visual verification. However, further incorporating quantitative metrics (such as FID) to evaluate the soundness of synthetic data could strengthen the paper.\n2.\tThe framework includes several hyperparameters, especially the weighting factors for multitask optimization, yet the paper provides limited discussion on their potential impact on performance. A deeper analysis of hyperparameter sensitivity would be valuable for understanding the framework’s robustness.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "Please refer to the weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Hi,

Draft has received overall positive reviews, with 4 accept.  Dear authors, please update the draft per recommendations and comments of the reviewers.

Congratulations.

regards

AC
Decision: Accept (spotlight poster)