Machine Unlearning under Retain–Forget Entanglement
Authors: Jingpu Cheng
Ping Liu
Qianxiao Li
CHI ZHANG
Venue: ICLR 2026 Poster (2026)
Keywords: Machine Unlearning
Constrained Optimization
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Forgetting a subset in machine unlearning is rarely an isolated task. Often, retained samples that are closely related to the forget set can be unintentionally affected, particularly when they share correlated features from pretraining or exhibit strong semantic similarities. To address this challenge, we propose a novel two-phase optimization framework specifically designed to handle such retain–forget entanglements. In the first phase, an augmented Lagrangian method increases the loss on the forget set while preserving accuracy on less-related retained samples. The second phase applies a gradient projection step, regularized by the Wasserstein-2 distance, to mitigate performance degradation on semantically related retained samples without compromising the unlearning objective. We validate our approach through comprehensive experiments on multiple unlearning tasks, standard benchmark datasets, and diverse neural architectures, demonstrating that it achieves effective and reliable unlearning while outperforming existing baselines in both accuracy retention and removal fidelity.

Published as a conference paper at ICLR 2026
MACHINE UNLEARNING UNDER RETAIN-FORGET EN- TANGLEMENT
Jingpu Cheng
Ping Liu
Department of Mathematics National University of Singapore chengj wiw.iugnnn... nus. edu
Department of Computer Science
University of Nevada Reno pino.ping it @m i1, com
Qianxiao Li
Chi Zhang
Department of Mathematics Institute for Functional Intelligent Materials National University of Singapore qi anxi aaeeuu nusuuuu. edu...du....... sg
Department of Mathematics National University of Singapore czhang2 4 4@nus. edu. sg
ABSTRACT
Forgetting subset in machine unlearning is rarely an isolated task. Often, retained samples that are closely related the forget set can unintentionally affected, particularly when they share correlated features from pretraining or exhibit strong semantic similarities. To address this challenge, we propose novel two-phase optimization framework specifically designed handle such retain-forget entan- glements. In the first phase, an augmented Lagrangian method increases the loss the forget set while preserving accuracy on less-related retained samples. The second phase applies a gradient projection step, regularized by the Wasserstein-2 distance, to mitigate performance degradation on semantically related retained samples without compromising the unlearning objective. We validate our approach through comprehensive experiments on multiple unlearning tasks, standard bench- mark datasets, and diverse neural architectures, demonstrating that it achieves effective and reliable unlearning while outperforming existing baselines in both accuracy retention and removal fidelity. Our code is available here
INTRODUCTION
The indelible memory of machine learning systems presents a paradoxical challenge: what happens when we need algorithms to forget? Consider face recognition system deployed for secure access. When an employee resigns, their biometric signature cannot simply be deactivated-in must be completely expunged from the underlying machine learning model. This ability to erase specific information extends to the broader concept known as "machine unlearning" (Cao and Yang) 2013),, which aims to selectively remove the impact of specific data from trained models. In fact, the importance of unlearning extends beyond the general concept itself, with critical applications in meeting legal obligations (Mantelero 2013), mitigating harmful representational biases (Mchrabi et al. 2021 and repairing models from mislabeled or poisoned training data (Northcutt et al. 2021).
Recent work has investigated range of unlearning scenarios, including random-sample unlearn- ing (Golatkar et al. 2020a; Izzo et al.] 202T), ), class-wise unlearning (Kurmanji et al., 2023), and concept-level unlearning, where the forget set does not necessarily align with class labels (Zhu et 2024). More recently, several methods (Foster et al., 2024 [Seo et al. 2025; Xu et al., 2024) have introduced efficient post-hoc or feature-space-avee aware solutions. Together, these approaches have significantly advanced our understanding of what it means for model to "forget".
Yet, forgetting is rarely an isolated task. Removing the influence of one group of data often directly affects another group that is closely correlated with it. For instance, forgetting toxic statements involving minority group may inadvertently alter the model's behavior on non-toxic statements
"Corresponding: rcchang24@nus edu.

---

Published as a conference paper at ICLR 2026
about the same group (Shen et al. 2024). Similarly, forgetting one subclass of images within broader category can disrupt predictions on closely related subclasses (Fan et al.] 2024a). Existing works typically assess retain performance by averaging over the entire retain set, paying little attention to these sensitive, correlated subsets, where performance is both more fragile and more consequential.
We therefore focus the challenge of rorgin-forget entanglement, where certain retained samples closely tied to the forget set and particularly susceptible to unintended degradation. To mitigate the resulting performance drops in these sensitive subsets, we propose two-stage framework based on constrained optimization. In the first stage, an augmented Lagrangian method enforces forgetting by increasing the loss on the forget set while preserving accuracy on less-correlated retained samples. In the second stage, the model is refined through gradient projection to restore performance on retained samples that are more strongly correlated with the forget set, without compromising the forgetting objective. To further stabilize the process and enhance generalization, we also regularize the loss distribution using the Wasserstein-2 distance during this stage.
We evaluate our method across variety of subclass-level unlearning scenarios, covering diverse forgetting tasks, multiple neural network architectures, and standard benchmark datasets. The results demonstrate that our approach consistently achieves effective forgetting while maintaining high accuracy on the retained data. In structured selective unlearning settings, it significantly outperforms prior methods, demonstrating robustness and reliability without compromising the intended forgetting effect. importantly, it preserves performance on retained samples that are closely related to the forget set, ensuring that sensitive subsets remain largely unaffected.
Our contributions can be summarized as follows:
Highlighting retain-forget entanglement: We focus correlation-awan unlearning setting, where the forget set is entangled with another group of data. This setting better reflects real-world unlearning demands and introduces significant distributional overlap with the retained data.
technical challenges due to
A novel two-stage unlearning framework: We propose a two-stage optimization- based framework to address this challenge. The first stage uses augmented Lagrangian method to enforce forgetting while preserving performance on less-correlated samples. The second stage applies gradient projection with Wasserstein-2 distance regularization to recover performance on sensitive retained samples without compromising the forgetting objective. Comprehensive evaluation: We provide comprehensive empirical evaluation across diverse tasks, architectures, and datasets, demonstrating that our method achieves strong forgetting performance while retaining accuracy on preserved data.
RELATED WORKS
Constrained Optimization in Machine Learning Constrained optimization is widely used in
machine learning to enforce domain-spedt specific requirements like fairness and safety (Cotter et al. 2019)
Zafar et al.) al.||20111 2019 Achiam et al. 2017: Liu et al. 2022). In fairness- awaae learning, these constraints prevent discriminatory predictions and are naturally framed as optimization problems (Donini et al.
2018 Zafar et al. 2019; Caton and Haas 2024). Classical techniques, such as penalty methods Berk
et al. (2017) and Lagrangian- baasd based approaches (Cruz et al.; Celis al., 2019: Cotter et al. 2019
Lokhande et al. 2020), have proven effective in these settings. Similarly y, in reinforcement learning.
safety constraints guide agents away from risky actions (Chow et al. 2018 Liu et al. 2022 often
handled through primal-dual optimization to penalize constraint violations (Achiam et al.
2017
Liang et al. 2018 Bohez et al. 2019))
Machine Unlearning The concept of machine unlearning was formalized by (Cao and Yang 2015
requiring model outputs indistinguishable from retraining without the deleted data. However, full retraining often infeasible for large-scale models, motivating approximate methods (Golatkar et al.
2020a by Izzo et al. (222T) Thuudi et al.] 2022 Mehta et al.) 2022). Many build on the framework
of Ginart et al. (2019), gradient-basee updates Gollatkar et al. 2020a| Fan et al. 2024bb Patel and Qiu 2025), sparsity- pruning (Jia et al. 2023), prompt editing (Liu et al. 2024). fisher and influence
based methods (Foster et al. [024]] Shi et al. 2024 Wu et al. 2022) and adversarial approaches (Di
et al. 2024). In particular, fine fine-tuning (Warnecke al. 2021 Zhang et al. 2024 2025) has been shown to be an effective post-training approach and is widely used in machine unlearning. For example, Kurmanji et al. 2023) proposed post-training methods by discouraging the model from
2

---

Published as a conference paper at ICLR 2026
correctly predicting labels forget set. Meanwhile, retain-forget entanglement has been show to impact unlcarning, where accuracy drops are often concentrated on retained examples most similar to the forget set (Zhao al. et al., 2024) Foster et al. 2024 Seo et al., 2025) considers settings where the forget subclass is semantically close other subclasses. Yet performance often reported as average over the entire retain set, which can mask degradation on the correlated subset. Similarly, in LLM unlearning (Maini al., 2024; Jin al., 2024 Chang and Lee, 20255 Choi et al. 2025), a neighbor set is often used for evaluation or regularizaiion in concept erasure for generative models (Gandikota et al. [024] Xie et al. 2025] Liu and Zhang 2025), preservation sets are used maintain overall performance.
2024: Chang and Lee 2025). For example, subclass-level forgetting (Zhu
Yet, these sets need not be closely related to the erased targets, whereas we explicitly decompose the retain set into adjacent and remote subsets and report the performance on both of them.
PROBLEM FORMULATION
Let D= ): be a dataset of N samples, where I; C X denotes an input and y/s E is its corresponding label. Let fep (xx) be model trained on D with parameters 0g. Given a subset D, D, the goal of machine unlearning is to obtain updated parameters 0 such that the resulting model fa(a) effectively forgets D,, while preserving performance on the remaining data D, == D D).
Classical formulations of machine unlearning typically do not assume further structures in the retain dataset. However, in many applications. forgetting D, affects not only average performance on D.. but disproportionately impacts a correlated portion inside D, (Fan el. (024a). We therefore conceptually split the retain set into two parts:
D, = Duij U Drem,
Duitj n Deem
Here, the adjacent retain set Dz consists of retained examples that are correlated with D and thus more sensitive to forgetting, while the remote retain set D.mm comprises the remaining. less-related retained examples, which we refer to as the remote samples.
In practice, the entanglement between the forget and retained samples can arise from different sources. One common scenario is subclass-level unlearning, where the forget set constitutes fine- grained subclass within a broader class. For example, if a model is trained on the 20 superelasses of CIFAR-106 100 and the forget set consists of one subelass, we can define Dr as the remaining samples from the same superclass and Drmm the rest of the dataset. Another scenario occurs when retained samples form a semantically related group with the forget set. For instance, in a language dataset containing normal and offensive sentences, comments referring to the same group of people may be strongly correlated with the forget set.
The goals of this retain-forget entangled machine unlearning therefore obtain updated model such that
1. The model retains its performance D,, especially samples belonging Dz that have strong correlation to D,,
2. The model forgets the forget set D, by removing or mitigating its influence.
It is important to note that the definition of "forgetting" can vary depending on the application. In privacy-focused contexts (Cao and Yang 2015). the objective is often for få to emulate model retrained from scratch on the retained set D,.. In contrast, there are scenarios that prioritize maximally reducing the model's performance on the forget set Dy. as studied in (Choi and Naj 2023). This approach is particularly relevant when D, contains undesirable patterns, such social biases, offensive content, or behaviors subject to withdrawal requests, where the goal is for the model to completely disregard the influence of these samples. In this work, adopt the latter perspective.
4 METHODS
Machine unlearning naturally poses multi-objective challenge: removing the influence of the forget set while maintaining overall performance. In the retain- retain-forget entangled setting, this becomes more difficult due to the semantic and distributional entanglement between the forget set Dy and the
3

---

Published as a conference paper at ICLR 2026
strongly correlated retain set D. To address this challenge, we introduce two-stage optimization framework this section.
4.1 STAGE 1: FORGETTING VIA CONTROLLED OPTIMIZATION
The first stage of our framework aims aggressively increase the loss on the forget set while preventing substantial degradation on the less-related retain set. Formally, let L = (f(;;))) == (0;
(0) (40(()
(0; D and L. (0) Lrm((: D denote the losses Dj. Dr and D. and
let 8g be the parameters of the original model. We formulate Stage 1 as the constrained optimization problem
min - (O) subject to
Arem (0) L. (0o).
We adopt an augmented Lagrangian formulation (Bertsekas) 2014) to provide an adaptive way of balancing the objective and the constraint:
Lang (0; A. p)) (0) A (0) C. (0o)) (c, C (0) C. (0o)) (2)
where is the Lagrange multiplier and > 0 penalty coefficient. We initialize A = 0 and iteratively update via gradient descent,
- rj V VoLug (0; X, 1s),
(3)
followed by updating the multiplier according to constraint violation:
A X+ ((
tem (0o)).
(4)
This iterative update scheme adaptively tightens or relaxes the penalty as needed, avoiding the need manually tune a fixed trade-off coefficient. The objective of Stage is to enforce unlearning on the forget set D, while preserving performance on the less-related retained subset Drem
4.2 STAGE 2: W2-DISTANCE GUIDED PROJECTED GRADIENT DESCENT (W-PGD)
Importantly, refrain from explicitly optimizing over the strongly correlated retain set Dr in Eq to avoid conflicting gradients (see Appendix [B.5p. As a result, the model achieves low accuracy on the forget set D, while maintaining strong performance the remote retain set Drem However, due to the semantic or distributional overlap between D, and Did performance on the adjacent retain set Di"" typically degrades. The objective of the second stage is to restore the model's accuracy on Dz while preserving the performance on D and Drem
4.2. Is CLASSICAL PROJECTED GRADIENT DESCENT GOOD ENOUGH?
We begin by aiming improve the performance on the adjacent retain set using the classical Projected Gradient Descent (PGD) framework (Bertsekas 1999), but adopt its first-order (linearized) projection
Sorget Acourac
0:22
0.88
15.0
100
Iequeed
mequeeec
0
Iorgel
2.00
0.22
60
38
4.33
40
33
(a)
(b)
(c)
(d)
Figure Training dynamics of PGD and cross-entropy loss distributions on Dy. (a) Loss and accuracy curves of PGD during the second stage; (b) Original loss distribution D, after the first stage; (c) Loss distribution on Dy after applying PGD the second stage; (d) Loss distribution on Dy after applying W-PGD. Comparing figure (b) and (c), PGD notably skews the loss distribution. with some samples attaining near-zero loss. In contrast, W-PGD (d) preserves a distribution closer to the original and effectively avoids assigning low loss forget set samples.
4

---

Published as a conference paper at ICLR 2026
variant, as widely used in multi-task learning (Yu et al. 2020 Farajtabar al. 2020). In this approach, the update modifies the gradient of C by removing its components aligned with the gradients of L.j and m
0+ 7 (VoL, Projv Vol."). where = span [VoL,, VeLr
(5)
Yet, this conventional optimization technique can exhibit significant performance degradation when applied to correlation-aware machine unlearning. As illustrated in Figure Ta although the average loss on the forget set Dj (blue line) remains stable under PGD, the prediction accuracy (red line) on Dy increases steadily. This counterintuitive behavior stems from the strong semantic and distribu- tional entanglement between Dy and the adjacent retained set D minimizing loss on the latter inadvertently reduces the loss similar samples in Dj- To compensate and maintain the mean loss on y, the model disproportionately increases the loss on less similar samples, resulting in polarized loss distribution, of standard PGD: it lacks the ability to control accuracy-level changes when loss redistribution is uneven. Indeed, preserving only the mean loss provides no guarantees on the proportion of samples with low loss values, often resulting in high accuracy on the forget set as many samples remain correctly predicted.
depicted in Figure Ic This observation exposes a critical limitation
4.2.2 GRADIENT PROJECTION WITH WASSERSTEIN DISTANCE REGULARIZATION
The failure of gradient projection using mean losses motivates the need for a more fine-grained control over the forgetting behavior. To this end, we propose to explicitly regularize the distributional shift in loss values on D by incorporating a Wasserstein-2 distance penalty.
The Wasserstein-2 distance, denoted W2, is principled metric for comparing probability distribu- tions /Vaserstein 1969). Given two probability distributions / and Q over IR the W2 distance is defined as
1/2
||u I
()(PQQ) =
inf (TT((P.) RR
v11 dy(,
(6)
where [(P, Q) denotes the set of joint distributions with marginals and Q. In our setting, P and Q represent empirical distributions of scalar loss values, admitting closed-form expression for the W2 distance. Specifically, given two collections of loss values fai,..., aN) and (bi....,by bby. the corresponding empirical distributions are defined p ~/4 E, 8a, and Q I Sb where d denotes the Dirac delta function. Then, after sorting the samples as āi S ... aN and by S b.. by. the Wasserstein-2 distance is simply given by
N
1/2
W2(P,Q)
b,)2
1=1
We define the empirical loss distribution over the forget set under parameters 0 as
p
forget
Σ
)e(f)(Z)))1)))
(8)
[Dgl
where é denotes the cross-entropy loss. Let denote the model parameters after the first stage. To constrain the mean and distributional shape of the loss over Dy, we define modified loss function:
[ = (1-0)) + oW? pfoorgtt
pforget
where a [0, 1] is a hyperparameter balancing the influence of the mean and distributional com- ponents. We then modify the gradient projection update to project the gradient of (dd) onto the orthogonal complement of the space spanned by the gradients of E, and Lrem:
0 - 0 (VoCel) (0) Projy V الوي (ad)( (0)) where V =sspan span (0), Vo (10)
This modified gradient projection method (W-PGD) enforces (, to be mostly unchanged during the update, while allowing the model to recover performance on the adjacent retain set D. as indicated by the following proposition.
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 Two-Stage Machine Unlearning
Input:
Forget set D, retain sets D. and D learning rates 1/ penalty coefficient and a, number of iterations K, M. Initialize = Bg. X O, compute D (6g)
Stage 1: Augmented Lagrangian optimization
for = 1 K do
Compute Laug (0; X, 11) -L; + (C,
LT (0o)) + 5 (C (0) L. (0o))
Update 0: f- 0 1/1 VoLaug (0; X, 14)
Update A: À + + u(rr
(6o))
end for
Stage 2: Wy-distance guided gradient projection optimization for j 1o M do
Compute (,) = (1-o) (0) + aW3 a
pforger pforget
Compute VoLy, VoL. VoL,
Update: 0 72 (oo Projy Vol."). where V =sspan (VeC,, Vol.m)
end for
Output: Unlearned model parameters
Proposition 4.1. Assume (0), Ladl (0) and (r" are twice continuously differentiable to O. Let A be the update of introduced by (TOp. Then, for sufficiently small > 0, we have;
(i) The change in E and f.em is at most second order in 1, i.e.
E + A0) ()) O(1)), Leem (B + 20) m. (0)
(ii) If V Ladl (0) is not in the span of (0) and V CrCrem()), then
اalll (0 + A0) الاl (0) -cn + 0
for some positive (depending on t). Hence, sufficiently small n, C:ll) strictly decreases.
Moreover, compared to the projected gradient descent where no guarantee on the accuracy of the forget set is provided, the following proposition provides bound on the accuracy of the forget set after the update. Specifically, let n be the number of superclasses in the classification task, and (f) denote the accuracy of the model on the forget set Dj. We have:
Proposition 4.2. Let m > log n and E > O. Suppose that (0) - 2,( < e, and ((f((z)). u) m.for all (xi, 9:) Dy. Then, the accuracy on Dj is upper bounded by:
1
(1- a
Acc,
+ [1
(11)
(m logn)n n2
The proposition indicates that when the minimum loss for model with parameter 0 is large and above zero, the accuracy of the forget set after W-PGD is bounded by small constant, ensuring the forgetting behavior of the model. Notice that for given e, the upper bound in the above proposition is minimized when = 1, i.e., when the Wasserstein distance is fully utilized. However, in practice where we assess each loss value in mini-batch sense, we observe that choosing & = may not achieve the best overall performance (see Ablation studies experiments, we set ( to be 0.5. As shown in Figure Tdl the loss distribution of the forget set after W-PGD more uniform compared that of PGD, maintaining zero accuracy on the forget set. In summary, the complete two-stage unlearning procedure is presented in Algorithm We evaluate our method in correlation-aware unlearning scenarios across multiple datasets and architectures.
in Appendix [B.5]. In our following
4.3 DISCUSSIONS ON THE TWO STAGES
The goal of the first stage is relatively straightforward, as the disentanglement between the forget set and the remote retain set makes the task less challenging. While alternative approaches, such as

---

Published as a conference paper at ICLR 2026
adding fixed-weight penalty terms, could in principle achieve similar trade-off with carefully tuned hyperparameters, the augmented Lagrangian formulation offers a key advantage: it introduces an adaptive multiplier that automatically balances the objective and constraint terms throughout training, resulting in a process that is more stable and less sensitive to hyperparameter choices.
A key component in the second stage is the of distributional constraints formulated via W2 distances. Prior work such as Golatkar et al. (2020a) also enforces the distributional constraints by estimating KL divergence between parameter distributions under a Gaussian prior. As comparison, W2 admits a closed-form solution for one-dimensional empirical distributions via sorting, whereas KL divergence generally requires density estimation or strong parametric assumptions, introducing approximation errors and additional computational cost (Lv et al., (024). Therefore, the use of W2 distances makes computation far more convenient, avoiding the approximations (e.g., kernel estimation) or prior assumptions typically needed for KI. divergence, while still providing principled and effective distributional constraint.
EXPERIMENTS
In this section, conduct comprehensive evaluation of our proposed method across various machine unlearning scenarios. To ensure the generality of our findings, we design experiments that span multiple unlearning tasks, various benchmark datasets, and different network architectures.
5.1 EXPERIMENTAI. SETUPS
Following prior work on machine unlearning Kurranji et al.] 2023). we conduct experiments on CIFAR-100 (Krizhevsky et al., 2009). Tiny Imageeee (Le and Yang 2015), and safety-critical language task on ToxiGie (Hartvigsen et al. (002). For the vision benchmarks, we adopt the superclass organization CCIFAR-10:: 20 superclasses x 5 subclasses;
Datasets:
ImtgeeeeNet:: 10 semantic groups; see Appendix B.I). Given a selected forget subset Dy (a labeled subclass within superclass), we define D as the remaining samples from the same superclass and Drrm as all other retained samples. For the language task, we use ToxiGen with a normal/toxic binary classifier, where DD consists of toxic sentences about the LGBTQ group, Drdl contains non-toxic sentences about the same group. and DrEm includes other sentences. This setup instantiates an unlearning scenario with retain-forget entanglement, where D. forms a semantic subgroup closely related to the forget set.
Baseline methods: We compare our approach against various unlearning methods, including: Gradient Ascent (GA) (Thudi et al. 2022): Train the model by maximizing the loss on the forget set. Fine-Tune (FT) (Warnecke et al. 2021 Golatkar al. 2020at: Fine-tune the model on the retained set. SCRUB (Kurmanji et al. 2023): perform gradient ascent on the forget set and descent on the retain set simultaneously with distillation from the original model. f-spaase (Jia et al.) 2023): ) fine- tune the model on the retain set with fn-rrrm regularization on the model. SSD (Selective Synaptic Dampening) (Foster et al. 2024): post-hoc parameter dampening guided by Fisher-style importance. SalUn (Fan et al. 2024b): : saliency-guided alternating updates. All the methods are run with 3 random seeds, except for SSD which is a deterministic algorithm. DELETE (Zhou al. 2025): decouples the forgetting and retention terms via distillation-bal based loss to perform class-centric machine unlearning. GDR (Lin et al. 2024): applies prrection-rectifice and segnitude-adjustee gradient updates to mitigate gradient conflicts between forget and retain objectives. Munba (Wu and Harandi] 2025): formulates unlearning as Nash bargaining game between forgetting and preservation players to find Pareto-optimal gradient direction.
5.2 MACHINE UNLEARNING ON 100 WITH RESNET-18
We begin our evaluation using the CIFAR- dataset. Specifically, we select the "aquarium fish" subclass from the "fish" superclass the forget set. The remaining 4 subclasses in the superclass are used as the adjacent retained set, and the other 95 classes are used as the remote retained set.
Table summarizes the overall performance of all evaluated algorithms. While fine-tuning and baased based methods effectively preserve performance on the retained set, they exhibit limited
sparsity-
'The forget set is chosen alphahetically.
7

---

Published as a conference paper at ICLR 2026
capability in removing information from the target forget set. Similar limitations are observed for gradient ascent algorithms such as GA and SCRUB. For SalUn, SSD, GDR and DELETE, although they achieve very low performance on the forget set, there is noticeable drop in accuracy the retained set, particularly on the adjacent retain subset. Munba achieves a relatively good ballance between forgetting and retention, but still suffers from non-negligible accuracy drop on the retain set, and its forgetting performance is not as strong as many other baselines. This underscores the strong entanglement between the forget and the adjacent retain set: effective forgetting can inadvertently degrade performance on related samples.
Table 1: Results for subclass-level unlearning on CIFAR-100 using ResNet- The forget set corresponds to the subclass "aquarium fish" within the "fish" superclass. SSD is deterministic algorithm, so standard deviations are O.
Training accuracy Desi
Test accuracy D
Method
D
D
D
D
Original
99.99
100.00
100.00
90.00
80.00
85.33
FT
76.67 7.76
99.47 1.002
99.47
62.33
77.8343..88
83.8940.41
GA
70.53
72.75.0. 76
91.33
56.00
59 59.00+0.61
80.57+0.27
4 Ep-spase
55.93
98.4844..0
96.92-..20
51.67
82.1242:24
81.6/4+0.27
Munba
33.80
92.17+2.67
92.68
31.67+4.77
69. 69.75433.7
75.32+1.98
SSD
37.40 10.00
43.75
76.02
33.00
39.25
67. 0..20
SCRUB
4.47
58.65
82.67
7.00
54.75
75.4212295
SalUn
3.20 20
52.27
86.3510.22
3.00+1.00
34.90+1.03
71.78+0.17
DELETE
0.00
3.57 20.18
98.37 20.29
0.67 40.44
2.83
82.09
GDR
1.87
31.92
96.10ga.32
8.67
22.33
79.93+40.09
Our method
0.00
98,17
98.44
2.33
78.1710.31
81,1010.18
Our algorithm successfully circumvents the trade-off between forgetting and retention. achieves complete unlearning, with 0.00% training accuracy on the forget class, while simultaneously maintain- ing high performance on both the retained data and the test set. These results highlight the capability of our method effectively eliminate memorization of the target class without compromising generalization or utility on the remaining data.
5.3 UNLEARNING ON TOXIGEN WITH ROBERTA-BANE BASE
We next evaluate correlation-awe aware unlearning on the ToxiGen dataset under a biased pretraining setting. Concretely, first simulate biased training process where all sentences mentioning LGBTQ groups are labeled as normal-thus the resulting model hg systematically misclassifies toxic LGBTQ samples normal. This simulates realistic scenario where deployed model is trained on incomplete or biased data and needs post-hoc correction.
We define the forget D, as the toxic sentences about LGBTQ groups that were incorrectly labeled during biased training. In this case, the normal comments on LGBTQ group are highly correlated to the forget set: they share similar semantic meaning and the same label during the training process. The adjacent retain set D. consists of the non-toxic sentences about LGBTQ groups (which we would like to preserve), and the remote retain set Drem contains all other sentences. The unlearning goal is thus to remove the effect of the biased labels on Dy, driving the model to predict them as toxic, while maintaining accuracy both Dzdl and Drem
Fine-tuning, SCRUB, Munba, and SSD preserve high accuracy on both the adjacent and remote retain sets, but only produce modest forgetting (see Table GA and the sssprse baseline reduce accuracy on D, slightly more, yet this comes with a notable drop performance on the remote retain set. SalUn attains very low forget-set accuracy (13.67%), but still causes a substantial decrease on the adjacent retain set. GDR is strong baseline, achieving low forget-set accuracy (20.54%) while maintaining high accuracy on both retain subsets. In comparison, our approach achieves the lowest forgetting accuracy-indication the most effective correction- while preserving high accuracy on D."l (88.88% 88%) and Dem (92.73%), and these gains generalize to the test set.
8

---

Published as a conference paper at ICLR 2026
Table Results for unlearning on ToxiGen dataset. The forget set contains toxic comments about LGBTQ groups that were mislabeled as normal. Lower accuracy on Dj means better correction.
Training accuracy D.
Test accuracy D.
Method
Dj
Drmm
D
Drm
Original
85.06
97.77
92.33
78.06
95.48
85.63
FT
50.04
99.87=0.0s
99.434..00
17. 47.73+24..57
92.37=0.59
84.73:00.12
GA
46, 46.2641000
70.25=0.05
79.43+1.557
43.78
66.6420.00
76.38
& -spaars
45.64448.13 3.333
86.33
80.4640.18
46.31 +9.82
85.87
79.52
Munba
51.09413.68
99.31 40.45
90.0640.11
49.27 1:425
93.53
85.361..200
SSD
67.78
91.83
90.7610.00 o0
86.90
86.90
84.51
SCRUB
56, 56.1542.41
91.95+1.41
84.7940.61
57.67+1-42
90.65
80.00
SalUn
13.6640.08
60.80
80.30
12.424...0
57.59
11.06110.1
DELETE
42.86
67.8540.07
79.0640.04
39.53
64.56.0.10
75.72+0.04
GDR
20.5445.88 86
86.15
91.30+0.71
19.83 19.83 .9
83.9225.49
85.52
Our method
11.95++0.0
88.88g0.01
92.73+..01
14.29
85.8620.00
85.23
5.4
UNLEARNING ON CELEBA WITH VIT-B
In addition, we evaluate on CelebA, large-scale face attributes dataset containing over 200K celebrity images annotated with 40 binary attributes. We construct a 4-class attribute- based classification task using the two binary attributes "Male" and "Smiling", treating each combination (female & smiling, female & not smiling, male smiling, male not smiling) as a separate class. The forget set D is defined images from the female & not smiling class that are also not Young and do not wear Eyeglasses. Within this class, the remaining samples (differing only in the "Young or Eyeglasses" attributes) form the adjacent retain set D while the other three gender/smiling classes constitute the remote retain set Dremi This construction yields a larger-scale vision benchmark where the forget and adjacent retain subsets share highly similar semantic attributes, making furin-forget entanglement particularly pronounced.
We provide the unlearning results for ViT-B on this CelebA superclass unlearning task in Table 3 Fine-tuning, & sparse, and SCRUB largely preserve accuracy on both retain subsets, but only achieve modest forgetting: test accuracy D, remains above 70%. Gradient Ascent and DELETE, on the other hand, drive the forget accuracy to essentially zero, but so by collapsing performance on the adjacent retain set to chance level, rendering the model unusable on the very samples aim to protect. SSD also degrades both adjacent and remote retain accuracy substantially. In contrast, our method achieves significantly lower test accuracy on the forget set (from 81.37% down 25.48%) while still maintaining high accuracy on D (75.05%) and Demm (92.38%), yielding the best overall balance between effective forgetting and retention in this more demanding scenario.
Table 3:
Results for CelebA superclass unlearning using ViT-B. The table shows the accuracy of the forget set and retained set for both training and test data. The forget set is the subclass not "not young & not wearing glasses" from "female & smiling" superclas.
Training accuracy Dud)
Test accuracy Dud)
Method
Dj
Drem
Dj
Drem
Origin
98.91
99.08
99.53
81.37
89.93
90.82
FT
69. 69.23
89.97
91.57
67.56
88, 4:2.8
89.77-27.1
GA
0.00
0.00
97 97.4640.41
0.00
0.00
91.16
s-spars
76.03-2..4
92.02++..9
90.48 40.66
75. 75.28-4442
91.87
89. 89.46
SCRUB
80. 30.73
96.81+14 82
98.38 98.384+.64 40.64
71.10
89, 89.222t2.11
90.57
SSD
23.07 23.07
41. 1:0.0
84.28
23. 23.1940.00
44.52+40.00
80.05
DELETE
0.00
0.00
99. 90.62++.00
0.00
0.00
93.97 93.9740.01
Ours
1.85
85.65
99.08
25. 25.48
75.05
92.38+0.06
5.5
GENERALIZATION TO A DIFFERENT ARCHITECTURE
We next evaluate our approach on the Tiny ImageNet dataset, targeting superclass- unlearning with Vision Transformer (ViT) architecture. This setting allows to assess the generalization of

---

Published as a conference paper at ICLR 2026
Table Results for Tiny-ImageNee superclass unlearning using VIT. The forget set is the subelass "dog" in "mammals" superclass.
Training accuracy D.
Test accuracy D
Method
D
D
D
Original
99.53
99.77
99.77
89.38
94.95
93.33
90.72.1.6 62
99.6920.18
99.57 00229
88.33
93.65-10.77
89.1910.33
GA
1.18
17.38=1.61
84.25
1.566100:2
16.57
75.85
& sparee
78. +5.79
99.00+0.50
97.73-0.22
78.11+4.80
89.27
78.49
Munba
80.00g7.58
97.8610.98
96.444.030
75.57 1:.35
90.41
83.0440.51
SCRUB
8. 4.70
84.1548.44
97 97.54410.99
7.22
81.97 4-6.22
88. 88.2911.96
SalUn
4.48
58.30+1.30
78.54
5.67
55.81
73.00+0.21
SSD
15.4340.00
20.33.0.00
97.74
44.00
77.05
87 87.90+0.00
DELETE
0.00=0.00
39.45.0.42
99.47
0.00
37.33
89.6440.01
Our method
0.00=0.00
98.95.0.08
98.49220.o
3.11
91.27+0.78
88.88
GDR
21.00
90.20
93.74
24.22
85.1445.30
85.42
Table 5: Ablation study on the We distance regularization. The table shows the accuracy of the forget set and retained set of CIFAR-100 subclass unlearning using ResNet18.
Training accuracy Dali
Test accuracy Deli
Dy
Dram
D
Denm
w/o W2 Regularization W2 Regularization
18.87
99.5520.04
98 28.04
14.33
87.00 87.00 78.171031
80.55 81.1010.18
0.00
98.1710a1
98.4410.08
2.33 _0.47
the proposed unlearning framework across both a different dataset and distinet model architecture. In this experiment, the forget set corresponds to the "dog" class within the broader "mammals" superclass. As shown in Table the results are consistent with previous findings, demonstrating that the effectiveness of our two-stage algorithm generalizes beyond a single dataset architecture.
5.6 ABLATION STUDY W2 DISTANCE REGULARIZATION
As alluded to earlier, the W2 distance regularization is crucial for preserving the forgetting behavior of the model. To validate this, conduct an ablation study by removing the W₂2 distance regularization from our method and comparing the results with the full method. Table [S) indicates that training without the W2 distance regularization also maintains strong performance on the retained set, but leads to an increase in the forget set accuracy with 18.87% on the training data and 14.33% on the test data. This indicates that the W2 distance regularization necessary for preserving the forgetting behavior of the model in the second stage.
5.7 ADDITIONAL EXPERIMENTS
To further assess the robustness and versatility of our approach, we include additional experiments in Appendix B covering range of learning tasks and model architectures. In addition, we report the MIA efficacy results, computational costs, along with more ablation studies and sensitivity evaluations on key hyperparameters.
CONCLUSION
In this work, we investigated the challenge of retain-forget entanglement in machine unlearning, where certain retained samples are strongly correlated with the forget set and thus particularly vulnerable to unintended performance degradation. We proposed two-stage optimization framework that first enforces forgetting on the target set while preserving accuracy on less-related retained samples, and then refines the model to recover performance strongly correlated retained samples using gradient projection with a Wasserstein-2-basee distributional constraint. Extensive experiments subelass-level vision tasks and safety-relevant language benchmarks demonstrated that our method effectively balances forgetting and retention, outperforming prior approaches in both removal fidelity and accuracy preservation. Our results emphasize the importance of correlation-awee unlcarning and provide principled approach for handling forgen-forget entanglement in practical machine unlearning scenarios.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This research is supported by the National Research Foundation, Singapore, under the NRF fellowship (project No. NRF-NRRFF13-2011-000)
REFERENCES
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In International conference machine learning, pages 22- 31. PMLR, 2017.
Richard Berk, Hoda Heidari, Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgen- ster,, Seth Neel, and Aaron Roth. A convex framework for fair regression. arXiv preprint arXiv:1706.02400, 2017.
Dimitri P Bertsekas. Nonlinear Programming. Athena Scientific, 1999.
Dimitri P Bertsekas. Constrained optimization and Lagrange multiplier methods. Academic press, 2014.
Steven Bohez, Abbas Abdolmaleki, Michael Neunert, Jonas Buchli, Nicolas Heess, and Raia Hadsell. Value constrained model-free continuous control. arXiv preprint arXiv: 04623, 2019.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015
IEEE symposium on security and privacy, pages 463-480. IEEE, 2015.
Simon Caton and Christian Haas. Fairness machine learning: A survey. ACM Computing Surveys, 56(7):1-38, 2024.
Elisa Celis, Lingxiao Huang, Vijay Keswani, and Nisheeth K Vishnoi. Classification with fairness constraints: A algorrihm with provable guarantees. In Proceedings of the conference fairness, accountability, and transparency, pages 319-328, 2019.
Hwan Chang and Hwanhee Lee. Which retain set matters for I1m unlearning? case study on entity unlearning. Findings of the Association for Computatioonal Linguistics; ACL 2025, pages 5966- 5982, 2025. URL.tttss: //acaaaantttt ogy 2005..2222..2 findi 31 0.ddf
Dasol Choi and Dongbin Na. Towards machine unlearning benchmarks: Forgetting the personal identities in facial recognition systems. arXiv preprint arXiv:2311.022444 2023.
Minseok Choi, Daniel Rim, Dohyun Lee, and Jaegul Choo. Opt-out: Investigating entity-level
unlearning for large language models via optimal transport. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisties, pages 28280-28297, 2025. URL
wttp:/gyy...gg/22..oooo long 1371. pdf
Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18 (167):1-51, 2018.
Andrew Cotter, Heinrich Jiang, Maya Gupta, Serena Wang, Taman Narayan, Seungil You, and Karthik Sridharan. Optimization with non-differentiable constraints with applications to fairness, recall, churn, and other goals. Journal of Machine Learning Research, 20(172):1-59, 2019.
André Cruz, Catarina G Belém, João Bravo, Pedro Saleiro, and Pedro Bizarro. Fairgbm: Gradi- ent boosting with fairness constraints. In The Eleventh International Conference on Learning Representations.
Zonglin Di, Sixie Yu, Yevgeniy Vorobeychik, and Yang Liu. Adversarial machine unlearning- arXiv preprint arXiv:2406.07687 2024.
Michele Donini, Luca Oneto, Shai Ben-David, John S Shawe-Tare Taylor, and Massimiliano Pontil. Empirical risk minimization under fairness constraints. Advances in neural information processing systems, 31, 2018
11

---

Published as a conference paper at ICLR 2026
Chongyu Fan, Jiancheng Liu, Alfred Hero, and Sijia Liu. Challenging forgets: Unveiling the worst- case forget sets in machine unlearning. In European Conference on Computer Vision, pages 278-297. Springer, 2024a.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Dennis Wei, Eric Wong, and Sijia Liu. Salun: Em- powering machine unlearning via gradient-basee weight saliency in both image classification and generation. In International Conference on Learning Representations, 2024b.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. In International conference on artificial intelligence and statistics, pages 3762-3773.
PMLR, 2020.
Jack Foster, Stefan Schoepf, and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 12043-12051, 2024.
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyáska, and David Bau. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5111-5120, 2024.
Antonio Ginart, Melody Guan, Gregory Valiant, and James Y Zou. Making ai forget you: Data deletion in machine learning. Advances in neural information processing systems, 32, 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net:
Selective forgetting deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
9312, 2020a.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Forgetting outside the box: Scrubbing deep networks of information accessible from input-output observations. In Computer Vision-ECCT 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pages 383-398. Springer, 2020b.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint urXiv:2203. 00500, 2022.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pages 2008-2015 PMLR, 2021.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity simplify machine unlearning. Advances in Neural Information Processing Systems, 36:5 56:51584-51605, 2023.
Zhuoran Jin, Pengfei Cao, Chenhao Wang. Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. Rwku: Benchmarking real-world knowledge unlearning for large language models. arXiv preprint arXiv:2406. 2024.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlcarning. Advances neural information processing systems, 36: 1997 2023.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. http::/cc23.n.stanffee ledu/tiny- waamy-immeeeet...mm 200. zip. 2015.
Qingkai Liang, Fanyu Que, and Eytan Modiano. Accelerated primal-dual policy optimization for 2018.
safe reinforcement learning. arXiv preprint
Shen Lin, Xiaoyu Zhang, Willy Susilo, Xiaofeng Chen, and Jun Liu. GDR-GMA: Machine Unlearn- ing via Direction- Rectiffee and Sienitudc-Adjuseen Gradients. In Proceedings of the 32nd ACM International Conference Multimedia, pages 9087-9095, 2024.
12

---

Published as a conference paper at ICLR 2026
Chris Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. Large language model unlearning via embedding cooruppte prompts. Advances in Neural Information Processing Systems, 37:1 18:988 118266, 2024.
Ping Liu and Chi Zhang. Erased or dormant? rethinking concept erasure through reversibility. arXiv preprint arXiv:2505. 2025.
Zuxin Liu, Zhepeng Cen, Vladislav Isenbaev, Wei Liu, Steven Wu, Bo Li, and Ding Zhao. Constrained variational policy optimization for safe reinforcement learning. In International Conference on Machine Learning, pages I 11644-13668. PMLR, 2022.
Vishnu Suresh Lokhande, Aditya Kumar Akash, Sathya N Ravi, and Vikas Singh. Fairalm: Aug- mented lagrangian method for training fair models with little regret. In European Conference on Computer Vision, pages 365-381, Springer, 2020.
Jiaming Lv, Haoyuan Yang, and Peihua Li. Wasserstein distance rivals kullback-leibler divergence for knowledge distillation. Advances in Neural Information Processing Systems, 37:65445-65575. 2024.
Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and Zico Kolter. Tofu: A task of fictitious unlearning for llss. arXiv preprint arXiv:2401.0mm 1.6401,0061,,,,, 2024.
Alessandro Mantelero. The eu proposal for general data protection regulation and the roots of the 'right to be forgotten'. Computer Law & Security Review, 29(3):22-2235, 2013.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6):1-35 2021.
Ronak Mehta, Sourav Pal, Vikas Singh, and Sathya N Ravi. Deep unlearning via randomized conditionally independent hessians. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10422-10431, 2022.
Curtis G Northeutt, Anish Athalye, and Jonas Mueller. Pervasive label errors test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2 14749, 2021.
Gaurav Patel and Qiang Qiu. Learning to unlearn while retaining: Combating gradient conflicts in machine unlcarning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 421 1--221, 2025.
Seonguk Seo, Dongwan Kim, and Bohyung Han. Revisiting machine unlearning with dimensional alignment. In 2025 JEEF/CVE Winter Conference on Applications of Computer Vision (WACV), pages 3206-3215. IEEE, 2025.
Shaofei Shen, Chenhao Zhang, Alina Bialkowski, Weitong Chen, and Miao Xu. Camu: disentangling causal effects in deep model unlearning. In Proceedings of the 2024 SIAM International Conference on Data Mining (SDM), pages 779-787. SIAM, 2024.
Jialei Shi, Kostis Gourgoulias, John F Buford, Sean Moran, and Najah Ghalyan. Deepelean: machine unlearning the cheap by resetting privacy sensitive weights using the fisher diagonal. In European Conference on Computer Vision, pages 1-16. Springer, 2024.
Anvith Thudi, Gabriel Deza, Varun Chandrasekaran, and Nicolas Papernot. Unrolling sgd: Under- standing factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P)) P), pages 303-319. IEEE, 2022.
Leonid Nisonovich Vaserstein. Markov processes over denumerable products of spaces, describing large systems of automata. Problemy Peredachi Informatsii, 5(3):64-72, 1969.
Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. arXiv preprint arXiv:2108.11577 101.110....1.,.... 2021.
Ga Wu, Masoud Hashemi, and Christopher Srinivasa. Puma: Performance unchanged model
augmentation for training data removal. Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 8675-8682, 2022.
13

---

Published as a conference paper at ICLR 2026
Jing Wu and Mehrtash Harandi. Munba: Machine unlearning via nash bargaining. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4754 4765, 2025.
Yiwei Xie, Ping Liu, and Zheng Zhang. Erasing concepts, steering generations: A comprehensive
survey of concept suppression. arXiv preprini arXiv:2505. 19398, 2025.
Heng Xu, Tianqing Zhu, Wanlei Zhou, and Wei Zhao. Don't forget too much: Towards machine unlearning on feature level. IEEE Transactions on Dependable and Secure Computing. 2024.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 2020.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez-Rodriguez, and Krishna P Gummadi. Fairness constraints: A flexible approach for fair classification. Journal of Machine Learning Research, 20 (75):1-42, 2019.
Chi Zhang, Cheng Jingpu, Yanyu Xu, and Qianxiao Li. Parameter-efficient fine-tuning with controls. In Forty-first International Conference on Machine Learning, 2024.
Chi Zhang, REN Lianhai, Jingpu Cheng, and Qianxiao Li. From weight- based state-based fine- tuming: Further memory reduction on lora with parallel control. In Forty-secont International Conference Machine Learning, 2025.
Kairan Zhao, Meghdad Kurmanji, George-Oetavian Barbulescu, Eleni Triantafillou, and Peter Tri- antafillou. What makes unlearning hard and what to do about it. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Yu Zhou, Dian Zheng, Qijie Mo, Renjie Lu, Kun- Yu Lin, and Wei-Shi Zheng. Decoupled distillation to erase: A general unlearning method for any class-centric tasks. In Proceedings of the Computer Vision and Puttern Recognition Conference, pages 20350-20359, 2025.
Jianing Zhu, Bo Han, Jiangchao Yao, Jianliang Xu, Gang Niu, and Masashi Sugiyama. Decoupling the class label and the target concept in machine unlearning. arXiv preprint 288.022406.08288. 2024.
14

---

Published as a conference paper at ICLR 2026
A
PROOF FOR PROPOSITIONS AND THEOREMS
Proof of Theorem
Let
AO = -17 (Vo27 - Projv Ve (o.
Consider the first-order Taylor expansion of Ly:
L + A0) (0) VoL, 2 1 A (0) AB + o(1140113)
Note that
VoL,(0) A0 -Voo((()) (VocT)
- Projv vol = 0,
by the definition of projection.
Thus, the first-order difference between ( + A0) and (,) vanishes; the dominant term is second-order in 17. giving
C A0) - , (0) = (n)))
The same argument applies to Ve indicating that the update 2. is also second-order in
For the change _.d0 (0). we also consider the Taylor expansion:
Lad + A0) (0) + (0) A + Ao
We have
V VoL. (0) AA - v VoL (0)
(oC (0) Proj (V (0))
Vo VoL (0)1 (V (0), Projy (V V (v (0)))
Vo Vo udi (0)]1 (0)11
Projy V VoCt (1)
When VoC
(0) & V. we have
(o - Projv (0)) >
(12)
ensuring strict decrease in Cadi
L
Proof of Theurem There a minor typo in the statement of Theorem 4.2 in the main text. The term مما should read (~= + This does not affect the validity of the theorem or the proof presented below.
( (0) < e implies that
(1 (1-))(C,(0) - (0) + aW2 Pfiigget pfarget) <.
(13)
According the inequality F[[X - Y1] < W2(P, Q) for any random variables X Pad Y
Q.
we have:
(PP pfurgel S (1 (1=)())) L,(()) S a
plogett plunge).
P
(14)
This indicates that
(1 (1- + a1 4a8
(P
pfinget p
(15)
2a
a
On the other hand, for an n-class classification problem, if a model's prediction is correct over a sample, then its cross-entropy loss for this sample is at most log n. Since (xff(z;), y:) m, we have the estimation:
< (P pforret S
1
(16)
α
a
which gives that
1
1-a
Acc(O) < (m 1ogn)²
(17)
a
a
15

---

Published as a conference paper at ICLR 2026
B EXPERIMENT DETAILS AND ADDITIONAL EXPERIMENTS.
B.1 EXPERIMENTAL DETAILS
We provide details of our experimental setup in this section, including model architectures, dataset descriptions, and hyperparameter configurations.
Base Models For CIFAR-100 experiments, we use the ResNer-18 architecture from PyTorch, initialized with ImageNet-preerrinee weights. The model is fine-tuned on the CIFAR-100 superclass classification task using the Adam optimizer (learning rate 2e-5, batch size 128) for 30 epochs. For TinyImageNet, employ the ViT-B-32 model from HuggingFace, also initialized with pretrained weights, and fine-tune it on the TinylmageNet superclass dataset with a learning rate of 2.5e-5, batch fine-tune the RoBERTa-baa model from HuggingFace
size 128, for 30 epochs. For ToxiGen, on the mislabeled ToxiGen dataset (all samples about group "Igbtq" are labeled as normal) using AdamW with learning rate of 2.5e-5, batch size 128, for 10 epochs.
Datasets For the CIFAR-100 dataset, we use the standard data split and class hierarchy provided on the official CIFAR-100 website. In particular, CIFAR-I00 is a labeled image dataset composed of 100 fin-grained object classes, each containing 600 color images. These 100 fine labels can be further grouped into 20 broader categories known as superclasses. Therefore, each image is annotated by both a "fine" label (the specific class) and "coarse" label (the superclass).
TinylmageNet (Le and Yang 2013) is subset of ImageNet, comprising 110,000 images across 200 classes. Each class contains 500 training images, 50 validation images, and 50 test images. The classes correspond to WordNet synset IDs, which are hicrarchically structured. For our experiments, group the 200 classes into 10 superclasses based on the WordNet hierarchy. The names of these superclasses and the number of classes each are summarized in Table 6
ToxiGen (Hartvigsen et al. 2022 is synthetically generated toxicity dataset containing approxi- mately 250k sentences covering 13 social groups (c.g., women, LGBTQ, mental disables). Each sentence is labeled as toxic or benign, with an approximately 1 1 ratio. We adopt the official dataset and perform a 9:1 split to construct our training and test sets. We relabeled the toxic samples about the group LGBTQ benign train model with bias.
Table 6: Superclasses and number of classes in TinylmageNet.
Class Names
Mammals 27
Other Vertebrates
Invertehrates
Vehicles
Tools/Machines
if of classes
10
23
21
42
Class Names
Furniture
Clothes
Food
Sports/Receeation
Geology Natures
of classes
23
Baseline Methods For the baseline methods, we summarize the main hyperparameter settings here and leave full implementation details to the released code. For fine-tuning (FT), we fine-tune on the retained set for 10 epochs using Adam, with learning rates of 2 x 10 for CIFAR-100. 5 10 for TinylmageNet, and 2 x 10-5 for ToxiGen. For Gradient Ascent (GA), we perform gradient-ascent updates the forget set: on CIFAR-100, we use SGD with learning rate x 10-5 for 7 epochs; on Tit, we use Adam with learning rate 1.5 X 10-6 for 10 epochs; on ToxiGen, we use SGD with learning rate 2.5 x 10 For E spasree, we follow the GA setup and add an e regularization term, with coefficient& 5 x 10-4 for CIFAR-100, 100, 2 x 10 for TinylmageNet, and 5 x 10-5 for ToxiGen. For optimization, we use SGD (learning rate x 10 4, momentum 0.9) on CIFAR-100 and Adam (learning rate 2 x 10 10-5) 5) on TinylmageNet; the remaining settings follow our GA-style setup in code. For SCRUB, we use 5 max steps and min steps in all experiments, and optimize with Adam using learning rates 5 x 10-5 (CIFAR-100)
x 10 (TinylmageNeet, and 1 x 10-5 (ToxiGen). The penalty coefficients a and 7 (see Kurmanji et al. (2033) ) are set to 0.1 and 0.9, respectively. For SalUn, we use sparsity threshold of 50% (see Fan et al. 2024b)) for all experiments, and train for 3 epochs with learning rate x 10 on CIFAR- TinylmageNet, and 2 epochs with learning rate 1 x 10 on ToxiGen. For SSD (Foster et al. 2024]. we set = and a = 10 for CIFAR- 100 and TinyImageNe and À = 1 and a 50 for ToxiGen.
epochs with learning rate 2 x 10 on
For GDR (Lin al. 2024), we use the default hyperparameters 7 100 and = 0.02 across all
16

---

Published as a conference paper at ICLR 2026
Training Accuracy
Test Accuracy
102
Accuracy (%)
Accuracy (%)
remote Retain
remate Retain
adjacent Retain
40
adjacent Retain
forget
forget
20
2
Epoch
Epoch
Figure 2: Learning dynamics of our method in the second stage on CIFARIO0 with ResNet18. The left figure shows the training accuracy while the right figure shows the test accuracy. The adjacent retain set contains all adjacent samples while the remote retain set contains all remote samples.
experiments, together with Adam W at learning rate 1 x 10 for CIFAR- 100 and TinyImageNet, and 5 x 10 for ToxiGen. For MUNBa (Wu and Harandi 2025). train for 10 epochs with learning rate 0.03 on CIFAR- 100 and ImngenaeNee, and for 10 epochs with learning rate 0.01 on ToxiGen; all other hyperparameters are kept their default values.
Implementation details for our method For experiments on CIFAR100 and TinyImageNet, we use epoch for the first stage and epochs for the second stage with our method. For ToxiGen, use epoch for the first stage and 2 epochs for the second stage.
Stage 1: We the Adam optimizer with learning rate of 2.5e-6 for CIFAR-100 and 5e-5 for Imageete.Net. Remote retain set batch size set to 128 for the TinylmageNet and CIFAR100, and 64 for ToxiGen. Forget set batch size ToxiGen. The penalty coefficient is fixed at 10 for all the datasets. To avoid excessively large loss values on individual samples, we use clipped cross-entropy loss for the forget set:
16 for the CIFARIOR 64 for TinyImagenet and
ClippedCE(r, y, C) min CE(a, v)},
(18)
where CE(r, y) is the standard cross-entropy loss and C is set to 10 for CIFAR100 and TinylmageNet, and is
for ToxiGen.
Stage 2: We use SGD with a learning rate of 2e-5 for CIFAR-100, 2e-4 for TinyImageNet and 1e-5 for ToxiGen. Batch sizes are 512 for the remote retain set, 128 for the adjacent retain set, and 128 for the forget set for CIFAR 100 and TinylmageNet. All batch sizes are set as 64 for ToxiGen. For ResNet-18 and ToxiGen, we apply gradient accumulation over 10 batches of the remote retain to stabilize the gradients. For ImageNet, gradients for the remote class retain set are computed using single batch.
B.2 LEARNING DYNAMICS OF OUR METHOD
We illustrate the learning dynamics of our method during the second stage on CIFAR-100 with ResNet18 in Figure [ The figure demonstrates that the accuracy on the forget remains at zero throughout training, while the accuracy on the remote class retain set stays consistently high. Meanwhile, the accuracy on the adjacent retain set steadily improves as training progresses.
B.3 COMPARISON OF TRAINING TIME AND MEMORY USAGE
We provide the running time of our method and other baselines in Table 7 All running times measured in minutes using an NVIDIA RTX 3090 GPU. SSD has a very short run time of 2.5 mins. GA, SCRUB and SalUn complete in under 10 minutes, whereas FT and method require slightly longer training times. Nonetheless, these methods remain significantly more efficient than full retraining.
We also report the memory usage in table 3 where all the methods use the same batch size of 128. Our method uses slightly more memory than fine-tuning, GA, and -s-sprs due the two-stage
17

---

Published as a conference paper at ICLR 2026
optimization process, but remains more memory- than SCRUB and SalUn. The results indicate our method does not impose significant additional memory overhead compared to other unlearning methods.
Table 7: Results of running time in minutes.
FT
GA
spssars
SCRUB
SalUn
SSD
Retrain
Our Method
Run time
12.4
5,4
11.9
9.4
6.1
2.5
70.1
14.2
Table 8: GPU memory usage (MB) for different unlearning methods with batch size 128.
Method
Retrain
FT
GA
SCRUB
f--sarse
SalUn
Ours
Memory (MB)
2949
2949
2860
3715
2974
4993
3297
B.4 MEMBERSHIP INFERENCE ATTACKS (MIA) EFFICACY
While the preceding results focus on classification accuracy, we further evaluate the effectiveness of the proposed method in terms of privacy, specifically through membership infer- ence attacks (MIA). We follow Jia al. (2023) by adopting a confidence-basee
Comparison of MIA Efficacy Acroas Mathods
0.99
5.50
5.7
MIA predictor, applied to the unlearned model, to assess its ability distinguish whether samples from the forget class were part of the training data. The resulting MIA efficacy quantifies the proportion of forget set samples conrectly identified non-members (i.e., not seen during train- ing) by the unlearned model. A higher MIA efficacy therefore indicates more successful removal of information related to the forget set Dy. As reported in Figure B our method achieves an MIA efficacy of 0.99, indicating that effectively removes the information about the forget set from the model.
0.8-
0.11
02
Figure 3: MIA efficacy of different unlcarning methods on CIFAR100 using ResNet-18.
B.5 ABLATION STUDIES
Combining adjacent and remote-class as Retain Sets in Stage 1 We provide additional ablation studies to assess the necessity of constraining only the remote class retain set loss in the first stage. Specifically, we compare two variants of the augmented Lagrangian method: one constrains only the remote class retain set loss, _rem = C,em while the other constrains the loss on the entire retain set (both adjacent and remote class), (0) (0o). Results are shown in Table When the constraint includes the adjacent retain set, the model's ability to forget is impaired, with training and test accuracy the forget set rising to 7.13% and 5.00%, respectively.
more noticeable decline is observed in the test accuracy adjacent retained samples, where accuracy drops to 72.75%. This demonstrates that separating the forget set from the adjacent retain set in the first stage is crucial for effective unlearning in our method.
Table 9: Ablation study on the constraints in the first stage. The table shows the accuracy of the forget and retained set of CIFAR-100 subclass unlearning using ResNet18.
Training accuracy Dadi
Test accuracy Dadj
Dj
Drem
Dj
Drem
w/o adjacent
7.1340.93
98.30+0.54 0.10+0.00
99.9940.0n 100.00
5.00+0.81
72.7543.21 0.00
84.0840.1e 84.73+0.01
adjacent
0.00+0.00
0.00
Sensitivity on the hyperparameter a We analyze the sensitivity of the parameter a in Equation 9 for method. Specifically, compare the performance of our approach for a: 0, 0.5, and
18

---

Published as a conference paper at ICLR 2026
reported in Table TO The results indicate that setting a = 0 fails to achieve effective forgetting, with forget set accuracy 19.67% on the training data and 16.33% on the test data. In contrast, both yield favorable outcomes, achieving low aceuracy on the forget set and high
0.5 and a accuracy on the retained set for both training and test data. Interestingly, using a = 1, which fully incorporates the Wy-distance term in É, does not necessarily lead to optimal performance. Compared to a 0.5, setting a 1 results in 0. 0.74% percentage point increase in accuracy on the training forget set and a 0.75% point increase on the out-of-class retain set, with only a marginal 0.17% point gain the adjacent retain set. In this case, find that a 0.5 offers more balanced overall performance.
Table 10: Sensitivity analysis on the hypeparameter a. The table shows the accuracy of the forget and retained set of CIFAR-I00 subclass unlearning using ResNet18. The forget subclass "bee" from "insects".
Training accuracy Dadi
Test accuracy Dadi
D
Drem
Dj
Drem
a 0
19.47
99.68
97.86=0.61
16.33
86.67 0.59
79. 19.79+0.19
a 0.5
0.73
90.10+0.10
+7.39
6.33
80.18 80.92
0
1.4740.09
98.87+00.13
96.16_0.12
6.33+0.47
81.25
79.68+0.19
B.6 SENSITIVITY STUDY ON THE PENALTY COEFFICIENT
We examine the sensitivity of our method to the augmented Lagrangian penalty parameter y on the CIFAR- 100 subclass unlearning task. Table II reports the results for μ {5, 10, 20). Across this range, the forget accuracy remains low (at or below 3% on the test set), and the accuracies on both the adjacent and remote retain subsets vary only slightly. This indicates that our method is fairly robust to the choice of μ within a reasonable range and does not require fine- iine-grained tuning of this parameter.
Table 11: Sensitivity of our method the penalty parameter y on CIFAR-100 subclass unlearning.
Training accuracy Dedi
Test accuracy Dedli
Method
D
Drnm
Df
D
Our method μ 5
0.00+g.00
98. 18,004+0.05
98.33.+0.0
3.00=0.00
77.83
81.08
Our method μ 10
g0.00
98. 98.17 10.31
98.444..00
2.33_
78.17
81.10
Our method μ 20
0..0
98.1740.08
98. 98.4540.12
2.33
78.17
80.97
B.7 ADDITIONAL RESULTS
VIT results on CIFAR-100 superclass unlearning We provide additional experimental results for ViT on the CIFAR- 100 superclass unlearning task in Table [2] These results are generally consistent with our findings from other experiments. Fine-tuning and sparsity-based methods tend preserve performance on the retained set but fail to effectively erase information from the forget set. The gradient ascent method successfully reduces the accuracy on the forget set to zero; however, this comes at the cost of substantial performance drop on the retained set, particularly within the adjacent subset. Notably, the SCRUB method demonstrates competitive performance in this setting, achieving 1.93% accuracy on the training forget set and 3.00% on the test forget set, while maintaining strong performance on the retained set. comparison, our method attains zero accuracy on the training forget set, while simultaneously preserving high accuracy on the retained set.
Robustness to imperfect adjacency. To assess how sensitive our method is to imperfectly specified adjacent retain sets, we conduct a robustness study on the CIFAR-100 superclass unlearning task. Starting from the clean partition of the retain set into adjacent and remote subsets, we consider two noisy variants: (i) Case 1, where 20% of random samples from the remote retain set are mis-identified as adjacent; and (ii) Case 2, where 20% of random samples from the true adjacent retain set
19

---

Published as a conference paper at ICLR 2026
Table 12: ViT results Results for CIFAR-I00 superclass unlearning using ViT-B. The table shows the accuracy of the forget set and retained set for both training and test data. The forget set is the sublass "aquarium fish" "fish" superclass.
Training accuracy Dedi
Test accuracy Dadi
Method
D
Drem
Dj
D
Original
99.93
99.90
100.00
95.00
91.75
98.00
FT
76.67
99.47
99.47:+0.3
62.33
77.83+3.88
83.8940.41
GA
0.00
16.41+1.54
90.6441.38
0.00
15.17+1.20
84.42++.57
sspprse SCRUB
62.00
10.59
10.16
59.33
85.17
89.33 10.26
1.93 10.09
99.98
99.66
3.00 11.41
10.84
94.6910.11
Our method
10.00
98.50
98.8710.16
10.47
89.5011.24
93.2210.36
mis-identified as remote. Table 13 reports the resulting accuracies. Our method remains robust under these perturbations: the forget accuracy stays at 0% on the training data and below 6% on the test data, while the changes in adjacent and remote retain accuracies are modest. This indicates that method does not require perfectly identified adjacency to be effective and can tolerate a reasonable amount of noise in the partition.
Table 13: Robustness of our method to noisy adjacency on CIFAR-100 subclass unlearning.
Training accuracy De
Test accuracy Da
Setting
Dj
De
Dj
D
Clean adjucency
0.00
98. 08.1720.3
98. 10.00
2.33_0.47
78.171031
81. 11.10
+ 20% non-adj - adj (Case I)
0.00g0000
98.8120.00
98 20
3.33
81.37
78 25.9220.95
20% adj + non-adj (Case 2)
0.00
93.93
95. 25.75
5.00=1.00
77.32+0.41
80.08+1.18
We also studies an alternative way to construct the adjacent retain set based on feature-space similarity, instead of task-defined superclasses on CIFAR-100.
We extract output features from the pretrained ResNet-18, compute the k nearest neighbors (k = 20) of each forget sample among all retained samples, and assign every retained sample an adjacency score equal to the number of times it appears in these ANN lists. The top 10% of retained samples by this score are treated as the kNN adjacent retain set, and the remaining retained samples form the kNN remote retain set. Our two-stage unlearning algorithm is then applied using this automatically constructed partition.
The results in Table 14 show that the method continues achieve strong forgetting while maintaining high accuracy on both adjacent and remote retain subsets. Overall performance comparable the setting where adjacency is defined by the superclass structure.
Table 14: Comparison of our method under task-defined adjacency vs. KNN-identified adjacency on CIFAR-100.
Method
Train Dy
Train D
Train D
Test D,
Test D!")
D
Ours (lask-defined)
0.00
98.17
98.44 99.85
2.33+0.47
78.1740.31 77.67
81.10=o.18 83.13
Ours (KNN-identified)
3.00
99.31
6.00
Comparison with Retraining For completeness, we also compare our method with full retraining on CIFAR-100 100, TinylmageNet, and ToxiGen, under the same forget/retain splits as used in the main experiments. In all cases, the retrained model is obtained by training from scratch on the retained data only.
20

---

Published as a conference paper at ICLR 2026
Table 13 summarizes the results. While retraining generally maintains high accuracy the retain sets, it does not always achieve strong crasure on the forget set in our setting: the forget-set accuracy often remains relatively high. In contrast, our method consistently yields substantially lower forget accuracy while preserving competitive performance on both adjacent and remote retain subsets.
Table 15: Retraining vs. our method on CIFAR-100, TinylmageNet, and ToxiGen.
Dataset
Method
Train D
Train D
Train m
Test Dj
Test D
Test D
CIFAR-100
Retrain
38.4013.80
99.9810.0
99.99 10.00
37.00=s.10
83.92
83.37 10.31
CIFAR-100
Ours
0.00
98.1740.31
98.444..05
2.33
78.17
81.1010.18
TinylmageNet TinylmageNet
Retrain
62.82+5.50
99.4620.68
97.9942.01
64.45_4.79
95.71
90.58+0.12
Ours
0.00
98.95
98.4910.00
3.11
91.27
88.8810.54
ToxiGinn ToxiGen
Retrain
8.58 68
93.71 92.71
12.08
91.50
12.13
91.74 45
89.35
Ours
11.95+0.02
88.8840.01
92.7340.01
14.29
85.86
85.23+0.01
C USE OF LLM
In preparing this manuscript, we employed a large language model (LLM) solely to assist with refining and polishing the text. The LLM used to improve clarity, coherence, and readability, as well as to ensure consistent terminology throughout the paper. Importantly, all technical content, experimental design, and results were independently developed and verified by the authors; the LLM did not contribute to any scientific or methodological aspects of the work.
21


Reviews:
[{"review_id": "4rqG3rwdZb", "reviewer": "Reviewer_3FyN", "summary": "This paper addresses retain–forget entanglement in machine unlearning, where removing a forget set unintentionally harms correlated retained data. It introduces a two-stage framework: (1) an augmented Lagrangian step to forget targeted data while preserving unrelated samples, and (2) a Wasserstein-regularized gradient projection to recover performance on correlated retained data. Experiments on CIFAR-100, TinyImageNet, and ToxiGen show that the method achieves strong forgetting while maintaining retention accuracy, outperforming prior baselines.", "questions": "How is the adjacent vs. remote retain set identified in practice for complex or unlabeled datasets?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper pinpoints the practical challenge that forgetting one subset of data (e.g., a specific subclass or biased group) can harm semantically related retained samples, a scenario highly relevant for real-world fairness and bias correction.\n\n2. Ablation studies clarify the importance of the Wasserstein-2 term.\n\n3. The experimental design is well-defined and comprehensive, matching the problem setup.", "weaknesses": "1. The method relies on manual partitioning of the retained set into “adjacent” and “remote” subsets, whether it assumes prior knowledge of semantic relationships is unknown; If so, this decomposition may be impractical or ambiguous in large, unstructured datasets. \n\n2. The experimental evaluation on vision task is limited to small scale datasets.\n\n3. I did not get the meaning of theoretical bounds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pm8by3yK8U", "reviewer": "Reviewer_LxMc", "summary": "The paper tackles retain–forget entanglement in machine unlearning: forgetting a targeted subset often harms performance on semantically adjacent retained examples. The authors formalize the retain set as two parts, adjacent (correlated with the forget set) and remote (less related), and propose a two-stage optimization framework (TMU). Stage 1 uses an augmented Lagrangian to increase loss on the forget set while constraining performance on the remote retain set. Stage 2 applies projected gradient descent on the adjacent retain set but augments the projection with a Wasserstein-2 (W2) loss-distribution regularizer so that improvements on adjacent retains do not \"re-enable\" the forget set (i.e., keep forget accuracy low by maintaining the post-Stage-1 loss distribution). The paper provides propositions showing that Stage 2 decreases adjacent-retain loss while keeping changes to the forget/remote objectives second-order, and derives an upper bound on forget-set accuracy under the W2 constraint. Experiments on CIFAR-100 (ResNet-18), Tiny-ImageNet (ViT), and ToxiGen (RoBERTa-base) show lower forget accuracy with competitive or better retain performance than baselines (Fine-tune, GA, SCRUB, SSD, SalUn, L1-sparse). Ablations indicate W2 regularization is key; membership-inference evaluation reports high \"unlearning privacy\" for the proposed method.", "questions": "1.  How robust is TMU to noisy adjacency (e.g., approximate neighbors via embeddings)? A study where adjacency is inferred (not oracle) would boost practical relevance. \n2. Provide a simple recipe for setting $\\alpha$ and stopping criteria for Stage 2 (e.g., target W2 window + retain gains plateau) to avoid overfitting adjacent retains. \n3. Report wall-clock and memory overhead vs. SalUn/SSD/SCRUB on larger backbones and longer runs; clarify sorting cost for empirical W2 and any minibatch bias. \n\nThe paper identifies an important practical failure mode (retain–forget entanglement), proposes a simple-yet-effective two-stage remedy grounded in standard optimization plus a neat W2 distributional guardrail, and backs it with solid empirical evidence and helpful theory. The main asks are around adjacent-set construction, scalability/sensitivity details, and broader realistic applicable scenarios. Addressing these would elevate the contribution further.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Explicitly decomposing the retain set into adjacent vs. remote is well-motivated and aligns with observed collateral damage patterns in unlearning. \n2. Stage 1’s augmented-Lagrangian constraint is a clean way to preserve remote-retain performance without manual trade-off tuning; Stage 2’s W2-regularized projection addresses the empirical failure mode of plain PGD (mean-loss conservation but accuracy creep on the forget set). \n3. Propositions articulate why adjacent-retain improves while forget/remote are (first-order) preserved; the accuracy bound connects the W2 constraint to forgetting fidelity. \n4. Results span vision and language with multiple architectures; tables consistently show strong forget fidelity with limited adjacent-retain degradation, plus a clear W2 ablation and a privacy (MIA) check.", "weaknesses": "1. The method presumes a way to define and extract the adjacent subset (e.g., via superclasses or group semantics). Practical pipelines may not expose such structure; guidance or automated discovery (e.g., metric-learning neighbors) is not fully developed. \n2. While datasets are standard, scenarios remain largely supervised classification with relatively clean adjacency (superclass/subclass or group semantics). It’s less clear how well the approach extends to open-world or multilabel settings, representation-level forgetting, or generative models. \n3. Two stages (with $\\lambda$, $\\mu$, $\\alpha$, step sizes, and projection subspaces) add complexity. The ablations help, but a deeper resource/efficiency analysis and stability study across wider seeds/tasks would strengthen claims of robustness. \n4. The accuracy bound leverages conditions (e.g., sufficiently large losses post-Stage-1 and batchwise W2) that may not strictly hold in streaming or highly imbalanced forget sets; practical guidance for setting $\\alpha$ and monitoring $\\epsilon$ is limited. \n5. A growing body of work frames unlearning as a multi-objective / multi-task optimization between forget and retain objectives, using gradient projection or surgery ([1], [2]), robustness-oriented or meta-learning variants ([3], [8]), parameter/connection sensitivity ([4]), bargaining/game-theoretic scalarization ([6]), and post-unlearning alignment or re-learning prevention in generative models ([5], [7], [9]). These methods surface well-known issues—gradient interference, oscillatory updates, sensitivity to weights in scalarization, and Pareto inefficiency—that your two-stage constrained-then-projected approach is implicitly designed to mitigate. However, the paper does not explicitly position TMU (proposed method) against this family or analyze when/why its W2-regularized Stage-2 projection outperforms (or fails relative to) gradient-rectified updates, Nash bargaining, or meta-unlearning schedules. A tighter comparative discussion, plus side-by-side experiments under a common protocol (same forget/retain splits, identical budgets), with diagnostics such as gradient-angle statistics, Pareto AUC, and \"re-enablement\" rates, would clarify the contribution boundary and strengthen claims over multi-task baselines.\n\n[1] Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection, WACV 2024. \n\n[2] GDR-GMA: Machine Unlearning via Direction-Rectified and Magnitude-Adjusted Gradients, ACMMM 2024. \n\n[3] Learning to Unlearn for Robust Machine Unlearning, ECCV 2024. \n\n[4] Scissorhands: Scrub Data Influence via Connection Sensitivity in Networks, ECCV 2024. \n\n[5] Boosting alignment for post-unlearning text-to-image generative models, NeurIPS 2024. \n\n[6] MUNBa: Machine Unlearning via Nash Bargaining, ICCV 2025. \n\n[7] Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Concepts, ICCV 2025. \n\n[8] Learning to Unlearn while Retaining: Combating Gradient Conflicts in Machine Unlearning, ICCV 2025. \n\n[9] GRU: Mitigating the Trade-off between Unlearning and Retention for LLMs, ICML 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UTnPGMHLo9", "reviewer": "Reviewer_SCFy", "summary": "The paper proposes a two-stage optimization framework for correlation-aware unlearning. Stage 1 establishes forgetting while preserving performance on remote retain samples. Stage 2 recovers performance on adjacent retain samples while provably constraining the forget-set loss distribution via Wasserstein-distance regularization (W2). Experiments on CIFAR-100, Tiny-ImageNet, and ToxiGen across multiple model backbones demonstrate the effectiveness of the proposed method on both adjacent and remote retain subsets.", "questions": "1. Could you give more explanations about the failure of PGD?\n\n2. What is $m$ in Proposition 4.2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem is well defined, the experimental setup is sound, and the results of the proposed method are promising; the ablation studies are comprehensive.\n\n2. For Stage 2, the analysis of why vanilla PGD fails is interesting and motivates the use of W2 regularization.", "weaknesses": "1. The theoretical components are doubtful, but Proposition 4.1—which relates the update to the learning rate and claims the modified loss remains largely unchanged—does not fully convince; stronger bounds or clearer assumptions would help establish this as evidence.\n\n2. Notation could be clearer: prefer $L\\left(\\theta ; D_r^{\\mathrm{adj}}\\right)$ instead of $L_r^{\\mathrm{adj}}(\\theta)$, and likewise $L\\left(\\theta ; D_r^{\\mathrm{rem}}\\right)$ and $L\\left(\\theta ; D_f\\right)$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Rk5cfUMo2r", "reviewer": "Reviewer_TtJK", "summary": "This paper focuses on machine unlearning, particularly focusing on the performance of the retain set samples that are highly related to the forget set. The forgetting is achieved in two steps. The first stage is very similar to the usual forgetting formulation to balance out retaining and forgetting, except that an adaptive weight is applied. The second stage uses a loss based on Wasserstein-2 to constrain the distributional change with respect to the forget set, which then forms a span for the retain set gradients to perform gradient projections.", "questions": "In the experiments, the compared methods, while representative, seem slightly outdated. For instance, SCRUB, SalUn and SSD were all available online already in 2023, and it has been more than two years. Also, the most recent among them was officially published in AAAI 2024, which is in Feb 2024.\n\nAt the same time, there seems to be an error in the citations for SCRUB:\nTowards Unbounded Machine Unlearning is a NeurIPS 2023 paper, not 2024.\n\n\nThe authors can consider reporting results and/or comparing with more recent works such as:\n\nLearning to Unlearn for Robust Machine Unlearning. ECCV 2024\n\nAdversarial Machine Unlearning. ICLR 2025\n\nDecoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Tasks. CVPR 2025\n\nLoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty. CVPR 2025\n\nMUNBa: Machine Unlearning via Nash Bargaining. ICCV 2025\n\n\nOften, machine unlearning works report the “upper bound” performance, which is the gold model: a complete retraining of the models only on the retain set. This shows us how far we are away from that upper bound. The authors should provide this.\n\nMany works also report the “Unlearning Time”, which is the amount of time used for unlearning. This is important, since a complete retraining of the model would be ideal, but would consume too much resources. The authors should provide this.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is written quite clearly.\n\nTheoretical support for the paper seems to be substantial, and there is a bound on the accuracy of the forget set after PGD. Such theoretical support is good, although I did not check the derivations in too much detail.", "weaknesses": "Overall, while I get the motivation of trying to retain good performance on difficult samples, i.e., the samples in the retain set that are highly related to samples in the forget set, I am not quite convinced of the usefulness of such a setting. For instance, one needs to further define what are the “closely related” samples and pick some out for such further training, which seems somewhat arbitrary. Such picking of samples may be more straightforward in cases with clear classes (e.g., in CIFAR classification) and for class-level unlearning, but may be difficult to transfer to more general settings, e.g., when individual samples need to be forgotten.\n\nAlthough the explanation for the gradient projection with Wasserstein distance regulation is mostly clear, I feel like some intuition is missing. For instance, what does it mean to project the adjacent set gradients onto the orthogonal complement of the space spanned by the forget and retain gradients? Furthermore, why must it be the span of those specific forget and retain gradients? \n\n\nThere are also some weaknesses in the experiments section, which have been placed in the “Questions” section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)