Controllable Unlearning for Image-to-Image Generative Models via $\epsilon$-Constrained Optimization
Authors: XiaoHua Feng
Yuyuan Li
Chaochao Chen
Li Zhang
Longfei Li
JUN ZHOU
Xiaolin Zheng
Venue: ICLR 2025 Poster (2025)
Keywords: Machine unlearning
Generative model
Controllable
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: While generative models have made significant advancements in recent years, they also raise concerns such as privacy breaches and biases. Machine unlearning has emerged as a viable solution, aiming to remove specific training data, e.g., containing private information and bias, from models. In this paper, we study the machine unlearning problem in Image-to-Image (I2I) generative models. Previous studies mainly treat it as a single objective optimization problem, offering a solitary solution, thereby neglecting the varied user expectations towards the trade-off between complete unlearning and model utility. To address this issue, we propose a controllable unlearning framework that uses a control coefficient $\epsilon$ to control the trade-off. We reformulate the I2I generative model unlearning problem into a $\epsilon$-constrained optimization problem and solve it with a gradient-based method to find optimal solutions for unlearning boundaries. These boundaries define the valid range for the control coefficient. Within this range, every yielded solution is theoretically guaranteed with Pareto optimality. We also analyze the convergence rate of our framework under various control functions. Extensive experiments on two benchmark datasets across three mainstream I2I models demonstrate the effectiveness of our controllable unlearning framework.

Published as a conference paper at ICLR 2025
CONTROLLABLE UNLEARNING FOR IMAGE-TO-IMAGE GENERATIVE MODELS VIA E-CONSTRAINED OPTI- MIZATION
Xiaohua Feng"", Yuyuan Lif, Chaochao Chen'1 Li Zhang', Longfei Li,, Jun Zhou², Xiaolin Zheng! 'Zeejiang University, PHangzhou Dianzi University, 3 Ant Group
ABSTRACT
While generative models have made significant advancements in recent years, they also raise concerns such as privacy breaches and biases. Machine unlearning has emerged as viable solution, aiming to remove specific training data, e.g., con- taining private information and bias, from models. In this paper, we study the ma- chine unlearning problem in Image-to tosttto--mm to-Image (I2I) generative models. Previous studies mainly treat it as single objective optimization problem, offering soli- tary solution, thereby neglecting the varied user expectations towards the trade-off between complete unlearning and model utility. To address this issue, we propose controllable unlcarning framework that uses control coefficient to control the trade-off. We reformulate the 12I generative model unlearning problem into e-constrained optimization problem and solve it with gradient-based method to find optimal solutions for unlearning boundaries. These boundaries define the valid range for the control coefficient. Within this range, every yielded solution is theoretically guaranteed with Pareto optimality, We also analyze the convergence rate of our framework under various control functions. Extensive experiments on two benchmark datasets across three mainstream 12I models demonstrate the effectiveness of our controllable unlearning framework.
INTRODUCTION
Generative models have recently made significant progress in fields such as image recognition (Ho let al.
2020 Dhariwal & Nichol (202)) and natural language processing (OpenAI] 22033 Touvron
al. 2023), capturing significant academic interest due their boundless generative potential. Typ-
ically trained on vast datasets from the Internet, generative models inevitably assimilate latent biases
and expose private information (Schwarz et al. 2021 Yang et al. (025). Existing studies (Kuppa
eet al., 2021 Tirumala et al., 2022; Carlini et al. 2023 Xu et al. 2024) have revealed that genera- tive models have a strong tendency to recall specific instances encountered during training, raising concerns that the models might output biases and leak private information when put into practical situations. Machine unlearning (Nguyen et al. 2022 Feng et al. (024) presents a viable solution to address this issue. It aims to eliminate the knowledge learned from specific training data (forget set) while preserving the knowledge learned from the remaining data (retain set).
Implementing unlearning for generative models serves dual objectives. i.c., fulfilling privacy quirements and enhancing model reliability. On the one hand, legislation such as the General Data Protection Regulation (Voigt & Von dem Bussche, 2017) grants individuals the right be forgot- ten. Consequently, service providers must unleam specific private information from the model in response to individual's 's request. On the other hand, the data available the Internet is rife with biases and inaccuracies, which compromises model performance when used for training. By proac- tively unlearning the biased and inaccurate data, the service providers can improve the liability of their models.
"Equal contributions. Corresponning author (rjuccc@zzu.edu..co

---

Published as a conference paper at ICLR 2025
Farible Degior
Forget See
Model Liiityy
Poitts Optiral Poits
Non Parcs Optand
Recain See
Ground Truth
Input
Original
Model
Boundary Fass Hightr Degines
25%
Bulan-ds soN
Relox-
75%
Boundary--oou Lownet Degris
Degres of Completeness
An overview of controllable unlearning. On the left, the first and second rows represent the forget set and the retain set, respectively. We first present the effect of unlearning in I2I generative models, followed by collection of controllable solutions, where is the control coefficient. On the right, we demonstrate that for each E, our solution guaranteed with the Pareto optimality.
Figure 1:
In this paper, we focus on the unlearning problem in Image-to- Imagee (I2I) generative models (Yang jet al., 2023), where unlearning is defined by the model's incapacity to reconstruct the full image from partially cropped one (Li et al. 2024a), shown in Figure [ Previous study (Li et al. 2024a) frames machine unlearning in generative models as a single-objective optimization problem, with the loss defined combination of performance
both the forget and retain sets. However, this approach faces three main challenges: i) First and foremost, this approach offers a fixed result, ignoring the real-world need for flexible trade-off's between model utility and unlearning complete- ness aligned with varying user expectations. Regrettably, this challenge remains overlooked in the majority of current research on unlearning. ii) This approach relies wholly on fine-tuning with manual terminating conditions, lacking a theoretical guarantee for convergence. iii) This approach integrates two optimization objectives into a single loss function, which compromises unlearning efficiency due the competition or conffict between different objectives.
To address these challenges, we propose a controllable unlearning approach that provides a set of Pareto optimal solutions to cater to varied user expectations. Users can select a solution based on the degree of unlearning completeness through a simple control coefficient s. Specifically, we reframe machine unlearning of 12I generative models into bi-objective optimization problem (Kim) & De Week 2005). i.e., unlearning the forget set (Ist objective, unlearning completeness) while preserving the retain set (2nd objective, model utility). Due to legislation requirements, the first objective prioritizes the second objective, meaning that minimizing the negative impact on the retain only arises once the unlearning objective sufficiently optimized. Therefore, reformulate the bi-objective optimization problem into a e-constrained optimization problem, where the unlearning objective is treated as a constraint (primary satisfy) and & is the control coefficient. Utilizing gradient-based methods to solve this s-constrained optimization, we can obtain two Pareto optimal solutions for the boundaries of unlearning with theoretical guarantee, which can be used deter- mine the valid range of values for e. Subsequently, we select the value of E within its valid range and relax the constraints on the unlearning objective by increasing 8. As result, we obtain a set of solutions that dynamically fulfill user's varied expectations regarding the trade-off between unlearn- ing completeness and model utility. Finally, to enhance the efficiency of unlearning, we analyze the convergence rates of unlearning framework under various settings of the control function which utilized to govern the direction of parameter updates. The main contributions of this paper are summarized as follows:
We focus 121 generative models, and propose controllable unlearning approach that balances unlearning completeness and model utility, providing set of solutions to fulfill varied user ex- pectations. To the best of our knowledge, we are the first to study controllable unlearning.
We reformulate the machine unlearning of generative models E-constrained optimization prob- lem with unlearning the forget set as the constraint, guaranteeing optimal theoretical solutions for the boundaries of unlearning. By progressively relaxing the unlearning constraint, obtain the Pareto set and plot the corresponding Pareto front.
We utilize gradient- based methods solve the e-constrained optimization problem. To enhance the efficiency of unlcarning, we analyze our framework's 's performance across different settings of the control function and validate with multiple combinations.
2

---

Published as a conference paper at ICLR 2025
We conduct extensive experiments to evaluate our proposed method over diverse 12I generative models. The results from two large datasets demonstrate that the Pareto optimal solutions yielded by method significantly outperform baseline methods. Additionally, the solution set achieves controllable unlearning to fulfill varied expectations regarding the trade-off between unlearning completeness and model utility.
2 RELATED WORK
2.1 121 GENERATIVE MODELS
Many computer vision tasks can formulated as 121 generation processes, e.g., style transfer (Zuu et al. 2017). image extension (Chang et al.) 2022), restoration (Teterwak et al. 2019), and image synthesis Yu al. 2020). There are mainly three architectures for 121 generative models, i.e., Auto- Encoders (AEs) (Alain & Bengio 2014). Generative Adversarial Networks (GANs) (Goodfellow) ct al., 2014). and diffusion models (Ho et al. 2020). AES mainly aim to reduce the mean squared error between generated and ground truth images but often produce lower-quality outputs (Doso- vitskiy et al., 2021; Esser et al. 202T). GANs, through adversarial training, significantly improve generation quality, despite their unstable training (Arjovsky et al. 2017 Gulrajani et al. [0077] Brock (ct al., 2019). Diffusion models, which use a diffusion wewesioon-ten--eeeeee then-denoising approach, aim for stable train- ing and high-quality generation by minimizing the distributional distance between generated images and ground truth images (Ho et al. 2020; Song & Ermon] 2020; [Salimans & Hoj (022). How- ever, diffusion models require greater amount of data and computational resources (Saharia al., 2022b, |Rombach et al., (0022). In this paper, we aim to design a universal unlearning method that can be applied across different 12I models.
2.2 MACHINE UNLEARNING
Machine unlearning aims to climinate the influence of specific training data (unlcarning target) from trained model. A naive approach is retrain the model from scratch using a modified dataset that excludes the unlearning target. However, this approach can be computationally prohibitive in practice. Based on the degree of unlearning completeness, machine unlearning can be categorized into exact unlearning and approximate unlearning (Xu et al. 2023).
Exact unlearning aims to ensure that the unlearning target is fully unlearned, i.e., as complete as retraining from scratch (Bourtoule et al.) 2021 Yan et al.) 2022, Li et al. (2024b). This ap- proach, which typically relies on retraining, is limited to unlearning specific instances and cannot be readily extended to generative models with strong feature generalizations. Approximate un- learning aims to obtain an approximate model, whose performance closely aligns with a retrained model (Golatkar al. 2020 Sekhari et al. 2021). This approach estimates the influence of unlearn- ing targets, and updates the model accordingly gly, usually through gradient- based updates, avoiding full retraining (Basu et al.
2021 Li al. 2023b). However, accurate influence estimation still challenging (Graves et al.
2021 reducing the applicability of this approach to generative models.
In text-to-image generative models, unlearning typically refers concept erasure Gandikota et al. 2003);; Pham et al. 2; Lu et al. (2024), such as removing an abstract concept like the artistic style represented by "Van Gogh" or the entity concept represented by "Musk". : In contrast, in I2I generative models, the objective of unlearning is to remove the knowledge the model has acquired from specific set of samples. In this paper, We focus on 12I generative models. The exploration of unlearning is accomplished by minimizing a composite loss, which is a combination of training loss on the retain and the forget sets (Li et al. 2024a). This approach is highly dependent on manual parameter tuning and cannot guarantee unlearning completeness. As for comparison, the solutions yiclded by our proposed framework are theoretically guaranteed with Pareto optimality.
PRELIMINARY
3.1 UNLEARNING PRINCIPLESS
As outlined in Chen et al. 2022 Li et al. 2024c), an unlearning task typically has three main principles: i) unlearning completeness, which involves eliminating the influence of specific data
3

---

Published as a conference paper at ICLR 2025
from already trained model; ti) unlearning efficiency, which focuses enhancing the speed of the unlearning process, and iii) model utility, which aims to ensure that the performance of the unlearned model remains comparable to that of model retrained from scratch.
3.2 PARETO OPTIMALITY
Given multi-objective optimization problem, where f((0) represents the i-th objective, the problem can be formalized as: ming f(0) (f) fa( fm)
Pareto dominance. Let fa, 8 be two points in feasible set Ω2 6 is said to dominate 6¹ (f x 8%) if and only if f: (8*) S (0") Vi E {1...., m) and (0r) fy (0*) jj E {1...., m}.
Pareto optimality (Lin et al. 2019 A point 0* is Pareto optimal if there is no Ô for which < 8" The collection of all such Pareto optimal points forms the Pareto set, and the surface of this set in the loss space is called the Pareto front.
3.3 121 GENERATIVE MODEL UNLEARNING
Model architecture. Encoder-decoder structures are widely used in 12I models, with: i) an encoder E, reducing images the latent space, and ii) decoder Dø reconstructing images from the latent space. For model Ia with input image x, the output is:
(z) D& (T(
where Txx) denotes the cropping operation (such as center cropping or random cropping), and 0 {y, } denotes the full parameter set.
Unlearning objective. Define the unlearning task for an 121 generative model Ia, involving data partitions D2 (forget set) and D, (retain set). Consider Teg i.e., the original model, with training data D = Dy U D,- Assume that Top is proficiently trained to generate satisfactory results on both Dy and D,.. The objective of unlearning to obtain an unlearned model that cannot generate satisfactory results on Dy (Ist objective, unlearning completeness) while maintaining comparable performance on DR (2nd objective, model utility). Formally,
max (PIX
and min (PI
where X, and X, are the variables for ground truth images in D, and D,, (X)) is the model output distribution for input variable X, and Die(-Il-) represents distributional distance, measured by Ballback-Leibber (KI.) divergence in this paper.
Following prior work (Kingma al. 2019 Xia et al., 2022 Wallace et al.) 2023). as the model is proficiently trained, hypothesize that Lg, can approximately replicate the distributions over both forget and retain sets (Kingma et al.) 2019 Xia et al. 2022 Wallace et al. 2023), i.e., P(X,) P(Ie, (T(X)))), and P(X,) Px = (())() Then, Eq. can be simplified to:
P(Tg, (T(X,))). Let Px Pllo. (X))) and
max Diu ||P x and min Div 1Px)))
(3)
where Px, and P represent the output distributions of the forget set before and after unlearning respectively. Similarly, Pxx and xx, represent those for the retain set.
4 METHODOLOGY
In this section, we first introduce a controllable unlearning framework for 121 generative models, which formulates unlearning as a constrained optimization with the unlearning objective as a con- straint. We utilize baadiettt-baedd method to obtain the boundaries of unlearning. Then we relax the constraint within the boundaries to derive a set of Pareto optimal solutions to fulfill varied user expectations.
4

---

Published as a conference paper at ICLR 2025
4.1 E-CONSTRAINED OPTIMIZATION FORMULATION
The unlearning task for 12I models is reformulated as bi-objective optimization (Eq. 3 with the first objective to maximize Die(Pxx x, || xy)) Nonetheless, the value of Div(- can theoretically be maximized to infinity, yielding an infinite number of possible 1" , (Li et al. 2024a), consequently resulting in extremely diminished model utility. To balance unlearning completeness and model utility, we bound Div([" x, ||P x,) by Lemma
Lemma (Divergence Upper Bound (Cover Thomas 2012).. Assuming the forget set with dis- tribution Px, characterized by zero-mean and covariance matrix E, and signal P Px, with the same statistical properties, the maximal KL divergence is realized when P. X, = (0, E).
Dio(Px,1I1x x,) S Din(Px1II((0,))
As image normalization typically involves mean subtraction (Elasri et al. 2022), we can assume Fx, and x follow zero-mean distributions for conciseness without sacrificing generality. Lemma reveals that the upper bound of Dio(IPx, xx, ||px
x ) is achieved when P Px, , (0, E). This suggests that maximizing Div(P x, )I equates minimizing Div(P x, (O, E)). Consequently, we
rewrite Eq. 33 as:
min (i,(A(O,, x,,) and min Div(IP x, xx,))
(5)
As both terms in Eq. (5) depend on 0, define (i(0) = Div(A((0, Σ)|| x and f2(0)
(ie x, x for conciseness. However, unlike classification models where their outputs are precisely univariate discrete distributions (Kurmanji al. 2024| Zhang et al. 2024a| Shaik al.) 2022 Wu et al., 2024a)), high-dimensional KL divergence calculations in 12I generative models are intractable. Thus, following (Li et al. 2024a), we adopt the L2 loss as a surrogate. Due to privacy legal requirements, unlearning objectives typically takes precedence. Thus, we set fi() as the primary constraint and treat Eq. (5) as a e-constrained optimization problem:
min f2( #CRd
s.t. (1 S
(6)
where e is a parameter to control the completeness of unlearning. We minimize f2(0) inside the feasible set Ω (0: (0) < $). which implies that our priority lies in unlearning the forget set rather than mitigating performance degradation on the retain set.
4.2 SOLVING THE E- C-CONSTRAINT OPTIMIZATION
To solve the e-constrained optimization problem in Eq. 66) approaches such as Sequential Quadratic Programming (SQP) (Nocedal & Wright Bonnans al. 2006). penalty function method (Yeniay) 2005), and interior point method (Renegar) (001) are commonly employed. Given the extensive parameter set of the 12I generative model, select special variant of the SQP algorithm for its lower complexity and comparable convergence guarantee (Nocedal & Wright Gill & Wong 2011 Gong al. 202T)
Specifically, we employ gradient-based method to solve Eq. 8.- Here, f4 denotes the step size, and 9e represents the direction of the parameter update, which is determined by solving a convex quadratic programming problem w.r.t. g (for detailed derivation, please refer to the Appendix C.I.:
(6). updating the parameter by 0z+1 +
ft = min [V - gll S.t. V.r 2 (1,)) - gERd
(7)
Due to the inability to obtain the effective range of & in the early stages of unlearning, direct com- putation of fi,)))
is not feasible. Consequently, we adjust the constraint of Eq. 7 by em- ploying a control function (8,) (i.e., ff.())) 2 <b.))), which should satisfy (1)) = sign(fi (0,) - E), where sign(x) z/ |x| for x 0 and sign(0) = 0. This ensures that the di- rection of updates remains as consistent as possible before and after the substitution. Further, we provide summary of our proposed unlearning algorithm in Algorithm
5

---

Published as a conference paper at ICLR 2025
Algorithm 1 e-Constraint Optimization Algorithm
Require:
Original model Ie., forget set D,, retain set D,, control function (0)), step size μ. co-
variance matrix Σ, numerical stability variable v le
1: Initial: Initialize
0, To. Top do Sample {z ). } and { from D,, D, and (N respectively, ensuring that [
2: for to T
Har,-|l = |
4:
Compute loss:
5:
(1,) = To. (T(D))) Io, (T()))
6:
(2) = La, (T(D.) D) Is. (T(
7:
Compute gradient: Víi (O.). (2..)));
Compute the solution to the dual problem of Eq. 7 : = max (((,) V
.0);
8:
[VS(
9: Compute parameter update direction: 9: = f2 (t.) + 7 ff (8t);
Update the parameter of the target model lo, 0++1 + 02 Mr9t
11:
end for Return Unlearned model Teri
Assumption 1. Assume (0) and fz(0) are continuously differentiable, and the trajectory {0: t E [0, +oo) follows the continuous-time dynamics 0. -9. where 9t is defined in Eq. and max(g(u,+oc) T < +00.
The convergence analysis of Algorithm regardiin Eq. (6) utilizes the continuous-time framework given by f -9t. mentioned in Assumption | further details of convergence.
Please refer Lemma in Appendix C.2 for
4.3 CONTROLLABLE UNLEARNING FRAMEWORK
Our controllable unlearning framework consists of phases. In Phase reformulate Eq. (6) into a special form to obtain the solution for the boundaries of unlearning. In Phase II, we adjust the value within its valid range to relax the unlearning constraint and obtain the Pareto optimal solutions for controllable unlearming. This relaxation of unlearning completeness allows for a controllable trade-off between completeness and model utility, therehy catering to varied user expectations.
Phase Boundaries of unlearning. The boundaries of unlearning refer two Pareto optimal solutions with the highest and lowest degrees of unlearning completeness.
To obtain the Pareto optimal solutions with the highest degrees of unlearning completeness, we reformulate Eq. (6)into the following special form:
min f2( s.t. f1( S. BERd
where = fi. and fi = inf (i).
(8)
BERd
The solution of this optimization problem we need to ensure that (+() 2 0 in Eq. g to guarantee the same sign with fi()) -. this
be obtained by Algorithm 1 According Assump-
tion I
paper, we simply define (0)) = a:l| (i)( with a 0 and 8 2 1.
Proposition 1 (Boundary Pareto Set). Under Assumption 7 let fi > -0 and f > -c be the infimum of (0) f2( respectively. Further, let (b) be continuous and V ((() (0) be continuously differentiable. ff + 4" and 9t + 0 as t - +00, with f( of constant rank near 6* and (i))), f2 being convex near O,, then O" is Pareto optimal solution and fi(0") ft.
Proof. The proof can be found in Appendix
Proposition I ensures that the solution O obtained by Algorithm | for solving Eq. 8 is the boundary of the Pareto set, specifically refer to the highest degree of unlearning completeness. Meanwhile, fa (0T) achieve the infimum of (0).

---

Published as a conference paper at ICLR 2025
Forget set
((0,5)
Retain set
T(O)
Random Sample
-9:
-77,10)
-7)2(0))
(2)
Fi
((0)
Casel: (1
Xy
Zune
Paramitar ipione direction
Case2
11
8
Gonr
Target
Encoder
Encoder
Highest Unteerning Completenee
Rebi
12001
Decoder
model
((0)
Stict
Fossitic Bossen
(2
ft
F2(0)
Paramater update dirsction
(1(0)
Min (2
#1 Control Contlident Unleariing Compleneness
Complstiones
Phase
Phase II
c-oosttraieed Optimization
(b) Controllable Unlearning
Figure 2: Pipeline of the controllable unlearning framework. (a) shows the unlcarning task of the 121 generative model which is framed e-constrained optimization problem. (b) shows that the implementation of controllable unlearning unfolds in two phases: i) initially identifying two boundary points of unlearning, necessitating strict reduction in (0) (or f2()) for optimality; and ii) then locating the given e's Pareto optimal point, with strict reduction in fi (1) when fi(0,) > and permitting increase when (0,)
Obtaining the Pareto optimal solution with the lowest unlearning completeness is similar to the process mentioned above, with the difference of exchanging the positions of fi (i) and f2(0) in Eq. This new problem is formulated as mingeRs fi s.t. f2(0) < where = f2. and f2 == infaERd (0). The solution 05 obtained by solving this problem is another boundary the Pareto set, i.e., the Pareto optimal solution with the lowest unlearning completeness, with f2 achieving the infimum of (0).
Phase I:: Controllable unlearning. To adjust the trade-off between unlearning completeness and model utility, we relax the unlearning constraint by defining fi (0i) < f1(0)) in Eq. (6). where 07 and 05 have already been obtained in Phase I. Then rewrite Eq. (8) for controllable unlearning:
min f2( S.t. fi( S BERd
where E ff, and fi == inf (0), BERd
where e R is used adjust the completeness of unlearning. In Phase II, according the sign condition in Assumption [ we simply set (0) = (0) ee) with 8 > 0, 8 = 2n +1++1 and n E N. Proposition (Interior of Paret Set). Under Assumption ! let fž = inføez f2(0) > -00 and suPre(o.+2o) T) Tmaz < +00. If 0e is stationary point with g. 0 and mh < toc, and both f1( and f2(0) are convex at O,, then Or is Pareto optimal solution w.E.t
Proof. The proof can be found in Appendix C.4
From Proposition] 2 Eq. 9 provides Pareto optimal solution w.r.t. By progressively increasing e from fr. which is estimated by fi (07) Phase I, we can trace path of Pareto optimal solutions for different completeness of unlearning. As result, this path offers controllable unlearning for varied user expectations. In addition, our framework demonstrates high unlearning efficiency, with detailed efficiency analysis provided in Appendix D
EXPERIMENTS
5.1 EXPERIMENTAL. SETTINGS
We evaluate our proposed method on three mainstream 12I generative models, i.c., Masked Autoen- coder (MAE) (He al. 2022). Vector Quantized Generative Adversarial Networks (VQ-GAN) (Li
7

---

Published as a conference paper at ICLR 2025
et al. 2023a), and diffusion probabilistic models (Saharia et al. 2022a). Please refer to Appendix E.1 for the settings of hyperparameters.
Datasets: Following (Li et al., (024a), we conduct experiments the following two large-scale datasets: i) ImageNet- I (Deng et 2009). from which we randomly select 200 classes, designat- ing 100 of these as the forget images, with 100 allocated for training and the remaining for validation; and ii) Places-365 (Zhou al. 2017), from which we randomly select 100 classes, designating 50 of these as the forget set and the remaining 50 as the retain set. Each class contains 5500 images, with 5000 allocated for training and the remaining 500 for validation.
and the remaining 100 as the retain set. Each class contains 150
Baselines: We first report the performance of the original model (i.c., before unlearning) as a ref- erence. Following (Li et al. 2024a), we set the following baselines: i) Max Loss (Warnecke al.) 2023 Gandikota et al. 2023), ) which maximizes the training loss on the forget set; ii) Retain La- bel (Kong& & Chaudhuri 2023). which minimizes training loss by setting the true values of the retain samples as those of the forget set; iii) Noisy Label (Graves et al., 2021; Gandikota et al., [002)) which minimizes the training loss by introducing Gaussian noise to the ground truth images of the forget set; and iv) Composite Loss (Li et al.) (024a), the State-Of-The-A builds upon Noisy Label by calculating the loss on the retain set and obtaining their weighted sum, thereby minimizing this weighted training loss.
(SOTA) method, which
Evaluation metrics. We adopt three different types of metrics to comprehensively compare our method with other baselines: i) Inception Score (IS) of the generated images (Salimans al. 2016); ii) the Frechét Inception Distance (FID) between the generated images and the ground truth im- ages (Heusel al. 2017); and iii) the cosine similarity between the CLIP embeddings of the gen- erated images and the ground truth images (Radford et al. 2021). IS evaluates the quality of the generated images independently. while the FID further measures the similarity between the gener- ated and ground truth images. On the other hand, the distance of CLIP embeddings assesses whether the generated images still capture similar semantics. Please refer Appendix |..2 for more infor- mation of evaluation metrics.
5.2 UNLEARNING PERFORMANCE
We test our method on image extension, inpainting, and reconstruction tasks. We report the results for center uncropping (i.e., inpainting) in Tabel [ and the others in Appendix
Baseline comparison: As shown in Table I compared to the original model, our method retains almost the same performance the retain set only exhibits minor degradation. Meanwhile, there is a significant reduction in the three metrics on the forget set. In contrast, these baselines generally cannot perform well simultaneously Composite Loss has the least performance degradation on the retain set, but its performance on the forget set is also the worst. We also observe similar findings for Max Loss in VQ-GAN. Furthermore, we provide some examples of generated images in Figure 3 and more images in Appendix G
both the forget set and the retain set. For instance, in MAE,
T-SNE analysis:
Following Li et al. 2024a), we conduct T-SNE analysis (Van der Maaten & Hinton 2008) to further analyze our method's effectiveness. Using our unlearned model, we gen- erate 50 images for both the retain set and the forget set. We then calculate the CLIP embedding vectors for these images and their corresponding ground truth images. As illustrated in Figure 4 after unlearning, the embeddings of retain set are close to that of the ground truth images, while most of the generated images on the forget set diverge significantly from the ground truth one.
Unlearning robustness: We validate the performance of our controllable unlearning framework in different image generation tasks by changing the cropping patterns. The results indicate that our framework is robust to various image generation tasks and generally outperforms baselines, with detailed results provided in Appendix I.I Morcover, we examine the unlearning effects of our controllable unlcarning framework under different crop ratios. The results in Appendix [.3] demonstrate that framework is robust to different crop ratios. Furthermore, we find that the visual effects of unlearning control are more prominent with larger crop ratios.
Summary: These results validate the effectiveness of our proposed method, which is universally applicable to mainstream 121 generative models as well as a variety of image generation tasks, con- sistently achieving favorable outcomes across all these tasks.
8

---

Published as a conference paper at ICLR 2025
Table Results of center cropping 50% of the images. 'F' and 'R' stand for the forget and retain set, respectively. Here, "Ours" refers to the boundary points of unlearning obtained in Phase I, that is, the solution with the highest degree of unlearning completeness. The best results are highlighted in bold, and secondary results are highlighted with underline.
MAC
VQ GAN
Diffusion Models
F4
P.
F-
F4
R.
F1
R.1
F1
R.
F4
PT
FJ
Original
Noiny
Compesile Ours
26.39
Forget Set
Set
Ground Truth
Input
Original Model
Max Loss
Retein Label
Noisy Label
Composite Loss
Ours
Figure 3: Generated images of cropping 50% at the center the image on VQ-GAN. From left to right, the images generated by baselines are presented. Our method results in the highest degree of unlearning completeness while maintaining a minimal reduction in model utility.
5.3 CONTROLLABLE UNLFARNING
We also evaluate the controllability of our method which provides set of solutions for varied user expectations. First, obtain two boundary points of unlearning, thereby establishing the valid range of values for E. We linearly increase the value E within this range, adding 25% the range interval each time, obtain optimum solutions corresponding some generated images corresponding to these solutions in Figure Due to the space limit, please refer to Appendix H for increments of the linear increase of E), please refer to Appendix 1.2
different values. We provide
examples. For results of more fine-grained control (i.e., smaller
We verify the unlearned models at different values, and report results in Table 2 As e increases, we observe trade-off: the unlearning completeness decreases, while the generated images' perfor- mance on the forget set progressively improves, and, simultaneously, the performance on the retain set also improves. This observation clearly demonstrates the controllability of our proposed method, which can cater to varied user expectations. Please refer Appendix [ for additional results of the generated images and T-SNE analysis, which corroborates the above numerical results.
Before Unlearning Forget Set
Before Unlearning: Retain Set
x Unlearning Forget Set
Unlearning: Retain Set
Figure 4: T-SNE analysis between images generated by method and ground truth images.

---

Published as a conference paper at ICLR 2025
Table 2: Results of center cropping 50% of the images under different unlearning completeness. "Highest" and "Lowest" respectively represent the two boundary points of unlearning identified in Phase is coefficient used to control the unlearning completeness in Phase II.
VQ-GAN
Diffusion Models
IP
FID
CLIP
CLJP
F4
F4
FT
Original
21.59
16.28
23.74
21.80
Highos
12:33
17:47
154,60
68:453
0.69
0.75
13:23
22.55
139.21
26.
11.84
18.47
165.05
95.42
0.55
0.81
2-25%
17.93
20.55
85.35
59.09
0.14
0.77
14.14
22.63
130.71
24.57
0.46
082
13.12
19.27
137.95
84.21
0.50
0.81
6-508
19.47
21.42
57.81
50.99
0.77
0.79
14.60
22.25
123.32
22.65
0.47
0.83
15.92
18.70
118.76
71.43
0.66
0.83
6-75%
20.8
22.87
42.51
$1.80
0.80
0,82
15.20
22.53
116.59
20.43
0.47
0.84
16.35
19.53
104.21
63.62
0.77
0.83
Lowest
21.23
22.92
31.28
25.83
0.82
0.84
15.77
22.75
109.28
20.36
0.48
0.84
16.36
20.78
90.03
52.96
0.77
0.84
5.4
UNLEARNING EFFICIENCY
To enhance the efficiency of our controllable unlearning framework, modify the selections of control function (0) during various phases. Specifically, we empirically examine the convergence under these conditions assess the framework's unlearning performance of efficiency. Phase with the control function satisfying (b) a V/VV (O)] we manipulate the value of the exponent ó to change the control function. Additionally, we verify the changes in the convergence rates of f (0) and f2( under four different values across three models, with results shown Appendix K It is evident that (0) and f2) achieve optimal balance in convergence rates when 5 2, and the overall rate of convergence fastest. In Phase II, where the control function satisfies (8) B(f (0) z)², we test the changes in the convergence rates of fi (0) and f2(0) for two different 5 values on three models. To stabilize the optimization process, we scale the form of the control function (i.e., (b( = (() 5) (0)lI, selecting two different values, with results presented in Appendix K can be observed that the overall rate of convergence was optimized.
6 CONCLUSION
In this paper, we propose controllable unlearning framework for 12I generative models to over- come the limitation of the existing method's incapability to fulfill varied user expectations. Our approach allows for controllable trade-off between unlearning completeness and model utility by introducing a control coefficient E to control the degrees unlearning completeness. We reformu- late unlcarning a E-constrained optimization problem and solve it with gradient-based method to find two boundary points that guide the valid range for e. Within this range, every chosen value of will lead to Pareto optimal solution, addressing the existing method's issue of lacking theoretical guarantee. Extensive experiments on two large datasets (i.e., ImageNet-IK and Places-365) across three mainstream 121 models (i.e., MAE, VQ-GAN, diffusion model) demonstrate significant advan- tages of our method over the SOTA methods with higher unlearning efficiency, and controllable balance between the unlearning completeness and model utility.
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China (No. 62402148) and Ant Group.
REFERENCES
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data- gentragnnn distribution. The Journal of Machine Learning Research, 15(1):356333593, 2014.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223 PMLR, 2017.
Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. In International Conference Learning Representations (ICLR), 2021.
10

---

Published as a conference paper at ICLR 2025
Joseph-Frédéric Bonnans, Jean Charles Gilbert, Claude Lemaréchal, and Claudia A Sagastizábal. Numerical optimization: theoretical and practical aspects. Springer Science & Business Media, 2006.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 202/ IEEE Symposium on Security and Privacy (SP), pp. 141-159. IEEE, 2021.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Representations (ICLR), 2019.
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. In 32nd USENIX Security Symposium (USENIX Security 23), pp. 5253-5270, 2023.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315 1325, 2022.
Vira Chankong and Yacov Y Haimes. On the characterization of noninferior solutions of the vector optimization problem. Automatica, 18(6):697-707, 1982.
Chong Chen, Fei Sun, Min Zhang, and Bolin Ding. Recommendation unlearning. In Proceedings of the ACM Web Conference 2022, pp. 2768-2777, 2022.
Lisha Chen, Heshan Fernando, Yiming Ying, and Tianyi Chen. Three-way trade-off in multi- objective learning: Optimization, generalization and conflict-avoidance. Advances in Neural Information Processing Systems, 36, 2024.
Dasol Choi and Dongbin Na. Towards machine unlearning benchmarks: Forgetting the personal identities facial recognition systems. arXiv preprint arXiv:2311.02240. 2023.
T.M. Cover and J.A. Thomas. Elements of Information Theory. Number 12. Wiley, 2012. ISBN
Stephen Dempe, Nguyen Dinh, and Joydeep Dutta. Optimality conditions for simple convex bilevel programming problem. Variational Analysis and Generalized Difserentiation in Optimiza- fion and Control: Honor of Boris S. Mordukhovich, pp. 149-161, 2010.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition (CVPR), pp. 248-255. Iece, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations (ICLR), 2021.
Mohamed Elasri, Omar Elharrouss, Somaya Al-Maadeed, and Hamid Tairi. Image generation: A review. Neural Processing Letters, 5465))4609-4665 4646, 2022.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings the IEEF/CVF conference on computer vision and pattern recogni- tion (CVPR), pp. 12873-12883, 2021.
XiaoHua Feng, Chaochao Chen, Yuyuan Li, and Zibin Lin. Fine-grained pluggable gradient cent for knowledge unlearning in language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 10141-10155. 2024.
11

---

Published as a conference paper at ICLR 2025
Rohit Gandikota, Joanna Materzynska, Jaden Potto-Kauffann and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 24222222224222 2436, 2023.
Philip E Gill and Elizabeth Wong. Sequential quadratic programming methods. In Mixed integer nonlinear programming, pp. 147-224. Springer, 2011.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9304-9312, 2020.
Chengyue Gong, Xingehao Liu, and Qiang Liu. Automatic and harmless regularization with con- strained and lexicographic optimization: dynamic barrier approach. Advances in Neural Infor- mation Processing Systems, 34:29630-29641 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1 1516-11524, 2021.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition (CVPR), pp. 16000-16009, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-653 68511, 2020.
Robert Janin. Directional derivative of the marginal function nonlinear programming. Springer, 1984.
Yong Kim and Oliver L De Weck. Adaptive weighted -um method for bi-objective optimization: Pareto front generation. Structural and multidisciplinary optimization, 29:149 15-1195-8, 2005.
Diederik Kingma, Max Welling, et al. An introduction to variational autoencoders. Foundations and Trends Machine Learning, 12(4):307-392, 2019.
Zhifeng Kong and Kamalika Chaudhuri. Data redaction from conditional generative models, arXiv preprint Xrvv:22305.1 1351, 2023.
Nupur Kumari, Bingliang Zhang, Sheng-Yu Yu Wang, Eli Shechtman, Richard Zhang, and Jun- Yan Zhu. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22691-22702, 2023.
Aditya Kuppa, Lamine Aouad, and Nhien-An Le-Khac. Towards improving privacy of synthetic datasets. In Annual Privacy Forum, pp. 106-119. Springer, 2021.
Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. Towards unbounded machine unlearning. Advances in Neural Information Processing Systems, 36, 2024.
Guihong Li. Hsiang Hsu, Radu Marculescu, et al. Machine unlearning for image-to-ing tene-too-nia to--togiiiieeee genera- tive models. In International Conference Learning Representations (ICLR), 2024a.
12

---

Published as a conference paper at ICLR 2025
Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 2142- 2152, 2023a.
Yuyuan Li, Chaochao Chen, Xiaolin Zheng, Yizhao Zhang, Biao Gong, Jun Wang, and Linxun Chen, Selective and collaborative influence function for efficient recommendation unlearning. Expert Systems with Applications, 234:121025, 2023b.
Yuyuan Li, Chaochao Chen, Yizhao Zhang, Weiming Liu, Lingjuan Lyu, Xiaolin Zheng, Dan Meng, and Jun Wang. Ultrare: Enhancing receraser for recommendation unlearning via error decompo- sition. Advances in Neural Information Processing Systems, 36, 2024b.
Yuyuan Li, Chaochao Chen, Xiaolin Zheng, Junlin Liu, and Jun Wang. Making recommender sys- tems forget: Learning and unlearning for erasable recommendation. Showledge-Basel Systems, 283:1111244 2024c.
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. Advances in neural information processing systems, 32, 2019.
Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept erasure in diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6430-6440, 2024.
James Martens. Second-order optimization for neural networks. University of Toronto (Canada), 2016.
Kaisa Miettinen. Nonlinear multiobjectiive optimization, volume 12. Springer Science & Business Media, 1999.
Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoe Viet Hung Nguyen. A survey of machine unlearning. arXiv preprint 20X0222990029 2022.
Jorge Nocedal and Stephen J Wright. Numerical optimization. Springer.
OpenAl. GPT-4 technical report. CoRR, abs/2303,087744 2023. doi: 10,4850W/rXix.2233007744 URL
Panos M Pardalos, Antanas Zilinskas, Julius Žilinskas, et al. Non-convex multi-objective optimiza- tion. Springer, 2017.
Vitali Petsiuk and Kate Saenko. Concept arithmetics for circumventing concept inhibition in diffu- sion models. European Conference on Computer Vision, pp. 309-325. Springer, 2025.
Minh Pham, Kelly O Marshall, Niv Cohen, Govind Mittal, and Chinmay Hegde. Circumventing concept erasure methods for text-to-image generative models. In The Twelfth International Con- ferencee Leurning Representations, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR. 2021.
James Renegar. A mathematical view of interior-potie point methods convex optimization. SIAM, 2001.
Stephen M Robinson. Perturbed kuhn-tucker points and rates of convergence for class of nonlinear- programming algorithms. Mathematical programming. 7:1-16, 1974.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
13

---

Published as a conference paper at ICLR 2025
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- fion processing systems, 35:36479-36494, 2022b.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference Learning Representations (ICLR), 2022.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Katja Schwarz, Yiyi Liao, and Andreas Geiger. On the frequency bias of generative models. Ad- vances in Neural Information Processing Systems,
2021.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Pro- cessing Systems, 34: 18755088-88 2021.
Abdul Lateef Haroon Phulara Shaik, Monica Komala Manoharan, Alok Kumar Pani, Raji Reddy Avala, and Chien-Ming Chen. Gaussian mutation-spider monkey optimization (gm-smo) model for remote sensing scene classification. Remote Sensing, 14(24):6279, 2022.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems,
2020.
Gilbert W Stewart. On the perturbation of pseudo-inverses, projections and linear least squares problems. SIAM review, 19(4):634-66) 1977.
Hailin Tao, Jinjiang Li, Zhen Hua, and Fan Zhang. Dudb: deep unfolding-based dual-branch feature fusion network for pan-sharpening remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 62:1-17. 2023.
Piotr Teterwak, Aaron Sarna, Dilip Krishnan, Aaron Maschinot, David Belanger, Ce Liu, and William T Freeman. Boundless: Generative adversarial networks for image extension. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10521-10530. 2019.
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 01338274-382900 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.009286 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Paul Voigt and Axel Von dem Bussche. The general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing. 1003152676):0-55555 2017.
Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled trans- formations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 22532- 22541, 2023.
Yiting Wang. Jiachen Zhong, and Rohan Kumar. systematic review of machine learning applica- tions in infectious disease prediction, diagnosis, and outbreak forecasting. 2025.
14

---

Published as a conference paper at ICLR 2025
Alexander Warnecke, Lukas Pirch, Christian Wressnegger. and Konrad Rieck. Machine unlearning of features and labels. In 30th Annual Network and Distributed System Security Symposium NDSS, 2023.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-II), pp. 68 1-88. Citeseer, 2011.
Tsu-Yang Wu, Haonan Li, Chien-Ming Chen, al. A spectral convolutional neural network model based on adaptive fick's law for hyperspectral image classification. Computers, Materials & Continua, 79(1), 2024a.
Yongliang Wu, Shiji Zhou, Mingzhuo Yang. Lianzhe Wang, Wenbo Zhu, Heng Chang, Xiao Zhou, and Xu Yang. Unlearning concepts in diffusion model via concept domain correction and concept preserving gradient. arXiv preprint arXiv:2405.15304 2024b.
Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: A survey. IEEE transactions pattern analysis and machine intelligence, 45(3): 3121-3138, 2022.
Hanguang Xiao, Shihong Liu, Kun Zuo, Haipeng Xu, Yuyang Cai, Tianqi Liu, and Zhiying Yang. Multiple adverse weather image restoration: A review. Neurocomputingg pp. 129044, 2024.
Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Philip S. Yu. Machine unlearning A survey, Association for Computing Machinery, 56:36, 2023.
Naen Xu, Changjiang Li, Tianyu Du, Minxi Li, Wenjie Luo, Jiacheng Liang, Yuyuan Li, Xuhong Zhang, Meng Han, Jianwei Yin, al. Copyrightmeter: Revisiting copyright protection in text-to- image models. arXiv preprint arXiv:2411.13144 2024.
Haonan Yan, Xiaoguang Li, Ziyao Guo, Hui Li, Fenghua Li, and Xiaodong Lin. Arcane: An efficient architecture for exact machine unlearning. IJCAI, volume pp. 19, 2022.
Jing Yang, Yuangui Wu, Yuping Yuan, Haozhong Xue, Sami Bourouis, Mahmoud Abdel-Salam, Sunil Prajapat, and Lip Yee Por. LIm-ae-mp: Web attack detection using large language model with autoencoder and multilayer perceptron. Expert Systems with Applications, 274:126982, 2025.
Ling Yang. Zhilong Zhang, Yang Song, Shenda Hong, Runsheng Xu, Yue Zhao, Wentao Zhang, Bin Cui, and Ming-Hsuan Yang. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 56(4):1-39, 2023.
Özgür Yeniay. Penalty function methods for constrained optimization with genetic algorithms. Mathematical and computational Applications, 10(1):45-56, 2005.
Lu Yu, Joost van de Weijer, et al. Deepi2i: Enabling deep hierarchical image-to-image translation by transferring from gans. Advances in Neural Information Processing Systems, 33:1 33:11803-11815, 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
Fan Zhang, Gongguan Chen, Hua Wang, Jinjiang Li, and Caiming Zhang. Multi-scale video super- resolution transformer with polynomial approximation. IEEE Transactions on Circuits and Sys- tems for Video Technology, 33(9):449-4455 2023.
Fan Zhang, Gongguan Chen, Hua Wang, and Caiming Zhang. Cf-dan: Facial-expression recogni- tion based on cross-fusion dual-attention network. Computational Visual Media, 10(3):593 608,
2024a.
Yihua Zhang. Yuguang Yao, Parikshit Ram, Pu Zhao, Tianlong Chen, Mingyi Hong, Yanzhi Wang, and Sijia Liu. Advancing model pruning via bi-level optimization. Advances in Neural Informa- fion Processing Systems, 35: 15319-1888866 2022.
15

---

Published as a conference paper at ICLR 2025
Yihua Zhang. Prashant Khanduri, loannis Tsaknakis, Yuguang Yao, Mingyi Hong, and Sijia Liu. An introduction to bilevel optimization: Foundations and applications in signal processing and machine learning. IEEE Signal Processing Magazine, 41(1):38- 59, 2024b.
Jiachen Zhong and Yiting Wang. Enhancing thyroid disease prediction using machine learning: A comparative study of ensemble models and class balancing techniques. 2025.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 4056):1452-1464, 2017.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference 2017.
computer vision, pp.
16

---

Published as a conference paper at ICLR 2025
A BROADER IMPACTS AND LIMITATIONS
The abundance of training data not only enhances the performance of generative models but also introduces issues with privacy, unfairness, and bias. Our proposed controllable unlearning frame- work offers a viable solution to these issues. Our proposed framework is not limited to unlearning 12I generation models but can be easily extended to other types of generative models, including text-to-image and text-to-text models. However, the unlearning framework presented herein has cer- tain limitations. Note that Propositions [ and 2 in Section A assume the convexity of the objective function and the feasible set. This assumption is essential to guarantee that the yielded solutions Pareto optimal. In cases where the objective function and the feasible set are non-convex, the solu- tions obtained from solving Eq. (6) can only be guaranteed to be weakly Pareto optimal (Miettinen 1999) or Pareto stability (Chen et al. 2024).
B DISCUSSION ON THE OBJECTIVE OF UNLEARNING
Describing the unlearning target as inpainting image using only background content is feasible to some extent, such as concept unlearning (Wu et al. 2024b). For instance, we aim to protect privacy by unlearning parts of image generation model that contain personal information (i.e., an abstract concept), we can first identify the region of the image containing such information, then simply mask this region, and subsequently generate the model's output aligns with the inpainted new image. However, this approach has two issues:
image through inpainting, ensuring that
Firstly, it must be ensured that the new image generated through inpainting does not contain the information that needs be forgotten. We believe this can be accomplished by incorporating an additional adversarial discriminator using GAN training strategies or by employing reinforcement strategies.
Secondly, aligning the model's output with the inpainted new image merely confuses the knowl- edge learned by the model, increasing uncertainty during generation, which constitutes a super- ficial form of unlearning. However, based to erase the influence of certain samples on the model, directly aligning with Gaussian noise may yield more pronounced unlearning effect.
experimental experience, if the goal is merely
C THEORETICAL VALIDATION
C.I PROOF OF EQUIVALENCE
Given the original problem
min (2( s.t. f1(0) Se, BER
(10)
which is a constrained nonlinear programming problem. To solve it, we formulate its Lagrangian equation:
C(O, A) fz(0) + X(f =).
(1I)
Further, derive the KKT conditions for Eq. 11
VoL(0r .²) f22(0)) + (' -c
(1())) - 5
(12)
2
(f - -) = 0.
The standard Newton's Method searches for the solution (o X) 0 by iterating the following equation:
Dc+1
0,
VGC (l.)
((J((0)) s]
VeL(Or, Ar)
(13)
Ae+1
A
-|'T
((,) -
126-1
17

---

Published as a conference paper at ICLR 2025
where V denotes the Hessian matrix. However, the Newton step gt (V) cannot be calculated directly and we also have other optimal condition in Eq. (12 introduced by the inequality constraints. Instead, the basic sequential quadratic programming algorithm defines appropriate search direction St at an iterate (0t, Ax), as solution to the quadratic programming subproblem.
Denoting by St (of, the change in the variables at the current point (02, Ae), where (g g²))
solve the Newton- KKT system (Nocedal Wright):
VeL(O,, A + ((.i(())
(0:))0))))) - =lgg; - -V -VoL(O,,
f1())) - + ([f((()) -5 <0
(14)
d o 20 (he + 9;¹) (fi(0)) - ((f((()) - =llof O.
Denoting by Ae+1 d + ot. we have
VoL(O,, Adgs + (0.) -6//411 V-2(2()))
(1(),) - ((.) -elg? 5O
(15)
d-1 20
A-1 (fi(0*) - (f,) - =egg? = =0
is easy to check that Eq.
15) is the optimality system of the following quadratic problem (QP)
min (0,) + V (0.) T
29 (B,,
(16)
- V(
S
Setting -f = 9. the KKT conditions for Eq. (6) are consistent with the constraints specified in Eq. (1).. Further, according to Lemma 2 the optimal solution for Eq. (16), when approaching the
optimal solution of the original Problem (i.c., Eq. (IO). satisfies the KKT conditions Eq. (0O). Considering that the models discussed in this paper are all deep neural networks, based on previous
studies WWelingg & Teh 2011 Martens 2016 Zhang et al. 2021 2022 2023 2024b), the initial guess Hessian matrix can be approximated
identity matrix. Additionally, for consistency with the main text (i.e., 0,41 < 0, (.9t). setting = 9t. yields the following form:
min f2 (0.) V22 (6.) 22 (0.) 91 1 9t
(17)
(l 2 (1.0)
Lemma 2. Theorem of Robinson 1974). Suppose that 0* is local solution of Eq. 10p which the KKT conditions are satisfied for some X² Suppose, too, that the linear independence constraint qualification, the strict complementaring condition, and the second-order sufficient conditions hold at (6*, x²). Then if (ft, Xe) is sufficiently close to (8", ))), there a local solution of the subproblem Eq. 16 whose active set Ar is the same as the active set A (0*) of the nonlinear program Eq. (0))
at 8
C.2 BASIC COMPONENTS
Before exploring the proofs of Propositions [ and 2] essential define some fundamental concepts and lemmas. This references some works (Boyd & Vandenberghe, [204]] Pardalos et al.,
2017
Gong al. 202T) mentioned earlier; for the sake of readability, we will reiterate them here.
Penalty Function. An alternative method to evaluate the optimality of Algorithm involves the L penalty function given by:
Po( (2(0) + (0) -e)++
(18)
where & > 0 is scaling coefficient. The minima Eq. 18 align with the solutions to Eq. 6 for
sufficiently large values of & (Nocedal & Wright).
First-order KKT Condition and KKT Function. We revisit the first-order KKT condition (No-
cedal & Wright) for the constrained optimization described in Eq. (9.. Assume 0* is local optimum
18

---

Published as a conference paper at ICLR 2025
with continuously differentiable fi(0) and fz(0), and ( + 0. There exists a Lagrange mul- tiplier w" E [0, +00) such that:
f2(00) + (') 0, fi(00) Se. w ('(((0))- -
(19)
This setup highlights the importance of /V (0*) 7 0 as constraint qualification condition.
Utilizing Algorithm T for Eq. (9) and for vy 2 0, the KKT function (Gong et al. 2021) to verify the first-order KKT condition is defined as:
K (0,, me) = (2v)) + ( + + (0))
(20)
where 7 0, and [x]+ max( 0). It is clear that (T n1) > 0 for all 0r E and T 0,
achieving (f,, m7.) iff (fz, nt) satisfies the first-order KKT condition.
Second-order KKT Condition and KKT Function
In the context of Algorithm applied to Eq. (8 we expect that [/ (0:)l approaches zero, leading
to πt potentially diverging infinity. This scenario indicates violation of the first-order KKT
condition, potentially interpreted as 17" 00.
While the first-order condition (Eq. (19) is inadequate, the second-order KKT conditions involving the Hessian 2 (8) are applicable Dempe et al. 2010). Consider the relaxed form of Eq. 8 as:
min (0) VER
s.t. (f) =
(21)
If 6* is a local minimum of Eq. (8) it coincides with a local minimum of Eq. 21 Assuming f2()) and ( are continuously differentiable, with the Hessian (0) maintaining constant rank near 0* (Janin 1984), the first-order KKT condition for Eq. 21 can be formulated. There exists vector w E R' such that:
V f2 (0*) + 2 fi (0 w 0.
(22)
This condition implies that V f2 is orthogonal to the null space of V fi()), defining the tangent space of the stationary manifold {0 (f) 0} for (0).
For verifying local optimality under the constraints of Eq. S where (0) 2 0, the KKT function is proposed as:
K- (0,, m/.) = V2 (0.) + 1/. VSi (0.) + Tub (0.),
(23)
where (0,) 0 asserts that 0, is stationary for (0), and /V)2 + T/ ((.)))) 0 signifies local optimality with respect to f2))). aligning with the KKT condition for the relaxed problem ming (f) s.t. (0) S }r). with e; (0.).
In the analysis of Algorithm I fundamental theorem concerning the behavior of the penalty func- tion (0) and the KKT function (+ 17). given in Eqs. (20) and 23 is essential for understanding the algorithm's 's convergence and feasibility characteristics. This lemma is stated as follows:
Lemma 3. Theorem 3.2 of Gong et al. (202)) Assume Assumption holds, for any E 2 0, we have
d (o)) - -K (0t, (O,, Vt +00).
(24)
de
This equation indicates that (o.):) is non-increasing w.r.f time provided that Ke-m (0z, Th.) 2 O. This condition is satisfied if is sufficiently large such that Me O, or when the constraint is met, iee. (1(),) S ensuring (g))) =
C.3 PROOF OF PROPOSITION 1
Proof of Proposition 7 As O, converges to 0 for ( + too and given the continuity of (0) and V (O). it follows that din vi (0.) v (0*), and lim [[V (0.) = -IV (0*)].
Let J infoere fi (0) and f = inf f2()). Since (0)) 2 0, by substituting Eq. 23 into
Eq.
24 we have for any 0,
(0.) El+) E V[ (0.) + V (f))1 (5 -
( + (B)) Vt E +00).
dt
19

---

Published as a conference paper at ICLR 2025
Integrating both sides from 0 to yields:
L. [ (1,)
n. V. (0.)) + (5 na) v (0+)) ds - (f2 12) + 5 (06) -fi).
(25)
Given vi(d) 2 0 and = ft. Eq. 25 establishes that f0²² ab (b,) dt I fi (06) - fî < to0. Consequently, lim v (0.) = +
0.
Given 0 as limit point of {0.). there exists an increasing sequence {la n 1,2..... such
that bx + +o0 and Oz.. + 0 as n + too. The continuity of v0)) and V (f) ensures
lim v (0c..) = ف (0*) 0, and +im f (ft..) f/ (f*)l.
Since ab (8)) = 0 and the sign condition of (0),, it implies sign (fi (0*) - -f)) sign (bb (8*)) = 0.
Therefore f (0*) fi and H minimum point of f0). This gives + V (0,,, )|
fi (0*) 0.
Given lim Sr = 0, we deduce that lim f2 (0.) + 7/ Vi (0.) li [ = 0. Additionally, employing (b 2 0, Eq. (23) implies lim (-(0,, m1) = 0 for some T > 0.
Combining lim V f2 (8:) + me f (6+) = 0 and fi (6*) = lim fii (0t.)) we can derive
[2 (0.) T S (0+) [V (0.) v (v (0,) V (0)) V.22 (0.) v S1 (0) will
where fg is a convex combination of 0r and 0*, and we defined w/ = V/ (0.- f¹))
f2 (0.), where (v2 (8))) + denotes the Moore-Penroee pseudo- inverse of matrix &i (8)), which satisfies that
Define Wi (V (0))
W/ = arg min WER
|w||
s.t.
w E arg mig
Vi22 (0,) + V (0)
follows that
V (6.) + V2 (8)) S VZ2 v2h w/ll = V. (0,) + 7o Vf (0+)ll
Given V (0...) + 7/t.. Vfi (l:.)]l - 0 as n + +00, we have [ (0...) ) + V (8%.) wi..|| -
0 as n + +00, and by the constant rank condition and relevant corollary of Stewart 1977) (rephrased in Lemma +. we deduce
Assuming O,,, + 0" and 0g
and hence Wt. + ~² as n + +00, where w" f2 (0*). Thus, f (0.) + & (0)) will f (0*) + v (0*) w0 lead- ing f2 (8*) V fi (0*)
(V (())) +
(V fi (0))
w0| 0. which implies that H* satisfies the second-order KKT con-
ditions for Eq.
22
Given the convexity of (0) and f2( with respect to 0 then fz f2(0)) ) is the minimum in the feasible set Ω = {0 (0) < E). without any Å E Q such that fz (8) < fz(0"). Consequently, 8" is solution to Eq. (8). According to [Chnnkog & Haimes (1982). this solution is unique without
further checking, as affirmed by theorem of Miettinen (999) (rephrased in Lemma (5), 6 is Pareto optimal.
Therefore, combining the conclusions, 8* is established as both the minimum of fi) and Pareto optimal, confirming its status as Pareto optimal for complete unlearning.
Lemma 4. Corollary 3.5 of Stewart (977). Let (At} be a sequence of matrices converging to A,
as t +00. The condition lim A A is equivalent the condition that rank(A,) rank(A.) for all t sufficiently large.
Lemma 5. Theorem 3.2.4 of Miettinen (199)). A point 0" E Pareto optimal if it is unique solution of e-constraint problem (Eg.
6 for any given upper bound vector
(E), (et,-..,8t-1, (.et,,.,,,..............
20

---

Published as a conference paper at ICLR 2025
C.4 PROOF OF PROPOSITION 2
Proof of Proposition 2 Since 0, is stationary, Of - O. implying 3े Pe(O)) - 0 for all g 2 0. for all & 2 Th- Setting & = TJ. where 2 0, it follows that K, (0,,,, O. This implies that 0* satisfies the first-order KKT conditions for Eq. 19). i.e., there exists a Lagrange multiplier 17" [0, +00) such that
From Eq. 24 have 4 (0,) < -Kg- (0,, me). Consequently, Kg-m, (0,, me)
V22 (0)) (0*) 0, fi (1)))) s, "/ (fi -) 0.
As affirmed by theorem of Miettinen 1999) (rephrased in Lemma 6). 0g is Pareto optimal solution.
Lemma 6. Theorem 3.1.8 of Miellinen 1999). (Karush-Katle Kacknsh sufficient condition for Pareto optimaldy) Let the objective and the constraint functions of problem Eq. (9) be convex and continu- ousty differentiable at a decision vector 0* E Q. A sufficient condition r 0" be Pareto optimal
that there exist multipliers μ > 0 and 17" > 0 such that
(1) V f2 (0") + 1i "7 Vi (0*)
(2) ni (Ji (0") - 00
D ENHANCING THE EFFICIENCY OF UNLEARNING
To enhance the efficiency of unlearning, we investigate the influence of the control function (0) on convergence rates across different phases, as outlined in the lemma below:
Lemma 7. An extension based on Proposition 3.7 of Gong el al. 20027.
Under Assumption 4 with with a >> and 5 2 1, the respectively. For Phase II,
fz = infeeR* f2(0) > -oc, then: For Phase I, if (0) = all V fi [I)
convergence rates of (0) and f2( are O (1/th) and O (1/t² 1
(() = (0) - =) with 3 > 0, 8 = 2n + 1, n E N, and suprejo. Te Umax < +oc, the convergence rate of [fi =] is O (1/t2).
Lemma 7 demonstrates that the convergence rate depends on the exponent in (0),, where higher values of 8 result in faster convergence rate of fi (0). However, excessively large 5 can also lead slower convergence rate of (0) and instabilities in training. To balance convergence rate and training stability, we explore various 8 in (0) in both phases with extensive empirical studies. The results can found in Section 5.4
E MORE DETAILS OF EXPERIMENTS
E. HYPER-PARAMETE PARAMETER OF EXPERIMENTS
MAE. We set the learning rate to 10-4 1 with no weight decay. Both baselines and our method employ AdamW the foundational optimizer with 8 (0.90, 0.95), with the distinction being that our method necessitates some improvements on the basic optimizer. We set the input image resolution 224x224 and batch size to 32. Simultancously, I to a = 5, and the coefficient of v in Phase II to 3 = 5, followed by training for 8 epochs. Overall, it takes an hour NVIDIA A40 (48G) server.
set the coefficient of (0)) in Phase
We set the learning rate to 10-4 with no weight decay. Both baselines and our method employ AdamW as the foundational optimizer with 3 (0.90, 0.95). Our method necessitates some improvements on the basic optimizer. We set the input image resolution to 256x256 and batch set the coefficient of (b) in Phase I to 10, and the coefficient
VQ-GAN.
size to 16. Simultaneously, of (8) in Phase II to 3 = 10, followed by training for epochs. Overall, takes two hours on an NVIDIA A40 (48G) server.
21

---

Published as a conference paper at ICLR 2025
Diffusion model. We set the learning rate to 10 with weight decay. Both baselines and our method employ Adam as the foundational optimizer. Our method necessitates some improvements on the basic optimizer. We set the input image resolution to 256x256 and batch size to 16. Simulta- neously, we set the coefficient of (0)) in Phase to a 1, and the coefficient of (b()) ) in Phase II to 1, followed by training for 4 epochs. Overall, takes twelve hours NVIDIA A40 (48G) server.
E.2
EVALUATION METRICS
IS. Following (Li et al. 2024a). for ImageNet-IK, we directly use the Inception-v3 model check- point to calculate the IS score. For Places-365, use the Resnet-50 model checkpoint to calculate IS scores (Zhou et al. 2017).
FID. Regardless of whether ImageNet-1K K or Places-365, we directly use the Inception-v3 model checkpoint calculate the FID score.
Following (Li et al. 2024a), whether it is for ImageNet-Ik or Places-365, we use the VIT- H-14 model checkpoint to calculate the clip embedding vectors of the generated images and the ground truth images (Radford et al. 2021). Afterward, we calculate the cosine similarity between the vectors the clip score.
CLIP.
F ROBUSTNESS TO RETAIN SAMPLES AVAILABILITY
In machine unlearning, sometimes the real retain samples are not available due to data retention policies. To tackle this challenge, following (Li al. 2024a), we assess our method using images from other classes as substitutes for real retain samples. For instance, on ImageNet-IK, since we have already selected 200 classes, we randomly chose some images from the remaining 800 classes to act "proxy retain set" during the unlearning process. We incrementally reduce the proportion of real retain samples in the retain and increased the proportion of proxy retain samples, with the experimental results presented in Table 3 As demonstrated, our method is largely unaffected by the reduced availability of retain samples, indicating robust performance.
Table Results of center cropping 50% of the images under different retain set usage proportions. I indicates higher is better, and indicates lower is better. "F" and 'R' stand for the forget set and retain set, respectively. Here, results are based on the solution with the highest degree of unlearning completeness in Phase I.
MAE
VQ-GAN
Diffusion Models
IS
FTD
CLIP
IS
FID
CLIP
IS
FID
CLIP
F1
R
F1
R
F1
R1
F1
R.
F
R.
F1
1
FL
R.1
F1
R.
F
R.
Original
21.59
21.83
16.28
14.87
0.88
0.38
23.74
24.06
21.80
15.17
0.78
0.85
16.90
19.65
$2.12
81.51
0.89
0.91
100%
12.33
17:47
154.60
68.453
0.69
0.75
13.23
22.55
139.21
36.39
B-46
0.82
11.84
18.47
165.05
95.42
0.55
0.81
30%
12.32
17.46
150.05
73.14
0.70
0.53
13.27
22.30
138.69
24.83
0.46
0.81
11.91
18.10
167.32
98.52
0.55
0.80
22.54
18.53
165.24
0.80
16.64
30%
70.75
0.70
12.77
18.67
168.26
0.79
12.21
17.68
147.31
68.09
0.70
0.74
12.39
22.35
147.17
29.79
0.62
0.50
13.24
18.76
168.43
96.63
0.60
0.79
G MORE GENERATED IMAGES: BASELINES VS OURS
We conduct various generative tasks on three mainstream 121 generative models (i.e., MAE, VQ- GAN, and the diffusion model), including image expansion, inpainting, and reconstruction (Zhong & Wang 2025 Wang et al. 2025 Xiao al. 2024 Tao et al. 2023), to assess both baselines and our proposed method. Specifically. we conduct evaluations of image inpainting and expansion tasks VQ-GAN, image reconstruction tasks MAE, and image inpainting tasks the diffusion model. The results indicate that our method can adapt to mainstream 12I generative models and various image generation tasks.
22

---

Published as a conference paper at ICLR 2025
We conduct experiments image inpainting and expansion task unlearning VQ- GAN, where examples of the image inpainting tasks are illustrated in Figure 3 and examples of image expansion can be referred in Appendix !I Our unlearning method is effective for both image inpainting and image expansion tasks, and it signfficantyy surpasses baselines.
VQ-GAN.
Input
Original
Noisy
Composite
Qurs
(a) Forget Set
Ground Truth
Input
Original Model
Max Loss
Retain Label
Noiey Labal
Composite Loss
Clurs
(b) Retain
generated images of cropping 50% the center of the image. The upper part (a) represents the forget set, while the lower part (b) represents the retain set. "Ours" denotes the boundary condition of unlearning obtained in Phase which represents the point of the highest degree of unlearning completeness. It evident that our method significantly outperforms baselines in terms of the unlearning effect the forget set, most closely approximating Gaussian noise, and exhibits the least performance degradation on the retain set.
Figure 5: VQ-GAN:
MAE. We conduct experiments unlearning image reconstruction tasks the MAE. As shown in figure 6 our unlcarning method also effective the task of image reconstruction, with the effects of unlearning showing significant advantage over baselines.
23

---

Published as a conference paper at ICLR 2025
Input
Original
Noisy
Composite
(a) Forget Set
Ground Truth
Input
Original Model
Max
Retain Label
Label
Composite
(b) Retain Set
Figure 6: MAE: reconstruction of random masked images. We set the proportion of the random mask to 50%. The upper part (a) represents the forget set, while the lower part (b) represents the retain set. "Ours" denotes the boundary condition of unlearning obtained in Phase I, which represents the point of the highest degree of unlearning completeness.
24

---

Published as a conference paper at ICLR 2025
Diffusion model. We validate unlearning framework the diffusion model task for image in- painting. As shown in figure the results indicate that our method is equally applicable to diffusion models, and the effectiveness of unlearning surpasses that of baselines.
Input
Original
Noisy
Composite
(a) Forget Set
Ground Truth
Input
Original Model
Max Loss
Retain Label
Noisy Label
Compasite Loss
Qurs
(b) Retain Set
Figure 7: Diffusion model: generated images of cropping 50% the center of the image. The upper part (a) represents the forget set, while the lower part (b) represents the retain set. "Ours" denotes the boundary condition of unlearning obtained in Phase I, which represents the point of the highest degree of unlearning completeness.
H MORE GENERATED IMAGES: DIFFERENT DEGREES OF COMPLETENESS
We validate the control effect our controllable unlearning framework across multiple genera- tive tasks in three mainstream 121 generative models. The results demonstrate that controllable unlearning framework can effectively control unlearning across various image generation tasks of mainstream 121 generative models.
25

---

Published as a conference paper at ICLR 2025
VQ-GAN. We center-cropp the image by 50% and utilize the VQ-GAN for image inpainting. Subsequently, we applied our unlearning framework to enforce unlearning. The results in Figure demonstrate the effectiveness of our method, with the control effect being very pronounced.
Ground Truth
Input
Original
Boundary Eaits Highest Degree
Relax- Et
Relax- d2
Rellax ita 1
Boundary Lowest Degree
(a) Forget Set
Input
Original Model
Boundary-Fate Highest Degree
Relax-ey 25%
Baundary-Faas Lowest Degree
50%
75%
(b) Retain Set
Figure 8: VQ-GAN: generated images of cropping 50% at the center of the image under different degrees unlearning completeness requirements. The upper half (a) represents the forget set, and the lower half (b) represents the retain set. Our method first determines the two boundary conditions of unlearning, and then linearly increases the value within its range (here, we increase by 25% each time) to adjust the balance between unlearning completeness and model utility.
We verify the control effect of our controllable unlearning framework within the reconstruc- tion task using the MAE. The results in Figure Dareeeeiidiie Inddicate that our method can effectively control the completeness of unlearning in image reconstruction tasks well.
MAE.
26

---

Published as a conference paper at ICLR 2025
Ground Truth
Input
Original
Boundary Eain Highest Degree
Relak-z,
Relax
Relax-23
Boundary Lowest Degree
(a) Forget Set
Ground Truth
Input
Original Model
Boundary-Fyen
Relax-Er 25%
Relax-cy 50%
Relex-Fy 75% 1
Boundary Sanndayy-Cueer Cesex Degree
(b) Retain Set
Figure 9: MAE: construction of random masked images under different degrees of unlearning com- pleteness requirements. We set the proportion of the random mask to 50%. The upper half (a) represents the forget set, and the lower half (b) represents the retain set. Our method first determines the two boundary conditions of unlearning, and then linearly increases the value of E within its range (here, we increase by 25% each time) to adjust the balance between unlearning completeness and model utility.
27

---

Published as a conference paper at ICLR 2025
Diffusion model. We validate the control effect of our controllable unlearning framework within the inpainting task a diffusion model. As shown Figure I0 the findings illustrate that our method can effectively adjust the balance between the completeness of unlearning and the utility of the model in the context of diffusion model.
Ground Truth
Original Model
Boundary- Highest Degree
Relax-Et
Boundary
25%
50%
Degree
(a) Forget Set
Ground Truth
Input
Original
Boundary-fawe Highest Degree
Relex-6, 25%
Relax- di 50%
Relax-Ey 75%
Boundary Lowest Degree
Model
(b) Retain
Figure 10: Diffusion model: generated images of cropping 50% the center of the image under different degrees of unlearning completeness requirements. The upper half (a) represents the forget set, and the lower half (b) represents the retain set. Our method is also effective when applied to the diffusion model.
I ABLATION STUDY
To verify the robustness of our method on mainstream 12I generative models and various image generation tasks, conducted the following ablation studies: i) we vary the cropping patterns to
28

---

Published as a conference paper at ICLR 2025
demonstrate robustness across multiple image generation tasks; ii) we decrease the linear increment size of to validate that our method allows for more fine-grained control; and iii) we alter the cropping ratios to confirm the robustness of our method to changes in crop ratio.
1.1 MORE GENERATIVE TASKS
Similar to validating unlearning in classification models through Membership Inference At- tacks Choi & Na (2023), generative models can also be assessed for unlearning robustness by em- ploying attack methods to reconstruct the forget set. Although there substantial research in this area Kumari et al. 2023); Petsiuk & Saenko (025)) it typically focuses concept unlearning in text-lo-image generative models. In contrast, our focus on unlearning image-to-image gener- ative models. Unlike unlearning a single concept. goal is to unlearn the influence of set of samples or their distribution on the model. This makes challenging to validate the effectiveness and robustness of our method through attacks. Specifically, ness of our controllable unlearning framework for image extension tasks on VQ-GAN by varying the patterns of cropping. The results indicate that our controllable unlearning framework is robust to different cropping patterns.
validate the effectiveness and robust-
1.1.1 OUTPAINTING TASK
We retain 25% of the image center and utilize VQ-GAN for image outpainting. As shown in Figure 11 our method produces outpainting on the forget set that is most similar to Gaussian noise, and the outpainting performance the retain set shows the least decline compared the original model.
1.1.2 2
UPWARD EXTENSION TASK
We crop the upper half of the image, retain the lower half, and employ VQ-GAN for image exten- sion. The results in Figure 12 indicate that method produces extension the unlearning set that closely resembles Gaussian noise, and on the retain set, the extension performance decreases the least compared the original model.
1.1.3 LEFTWARD EXTENSION TASK
We crop the right half of the image, retain the left half, and use VQ-GAN for image extension. As shown in Figure 13 method produces leftward extension the forget set that closest resem- bles Gaussian noise and, on the retain set, the leftward extension performance exhibits the minimal decrease compared to the original model.
1.2 MORE FINE-GRAINED CONTROL OF UNLEARNING COMPLETENESS
After obtaining two boundary points of unlearning, our controllable unlearning framework linearly increases within its valid range to balance the completeness of unlearning and the utility of the model. However, the main paper, the increase of E is by 25% each time. For example, if the range of is [1,9], then the sequence of values would be {3,5,7). It is evident that the increments of are quite substantial, which results in a coarser granularity of control. Here, we reduce the linear increment of to extend the effectiveness of our controllable unlearning framework across various image generation tasks in VQ-GAN. The results show that our framework can achieve fine- grained control.
1.2.1 OUTPAINTING TASK
We retain the central 25% the image and utilize VQ-GAN for image outpainting. The results in Figure 14 show that the performance of controllable unlearning framework on the forget set gradually improves with the increase of E, and the extent of decline in outpainting performance on the retain compared to the original model, is also reducing.
29

---

Published as a conference paper at ICLR 2025
Input
Original Model
Retain Label
Noley
Composite
(a) Forget Set
Ground Truth
Input
Original Model
Max Loss
Retain Label
Noisy Label
Composite Loss
Ours
(b) Retain Set
Figure 11: Outpainting by VQ-GAN. We retain 25% of the image center. The upper half (a) desig- nated as the unlearning set and the lower half (b) as the retain set. For each subset, we compared the performance of both the baselines and our method on the outpainting task, where "Ours" represents the boundary condition of unlearning in Phase I, indicating the point of highest degree of unlearn- ing completeness. The results show that our method significantly outperforms the baselines on outpainting task.
30

---

Published as a conference paper at ICLR 2025
Ground Truth
Input
Original Model
Max Loss
Retain Label
Noiey Label
Composite Loss
Ours
(a) Forget
Truth
Input
Original Modal
Loss
Retain
Noisy
Composite
Curs
(b) Retain Set
Upward extension by VQ-GAN. We retain 50% of the lower half of the image. The upper half (a) is the forget set, and the lower half (b) is the retain set. For each set, compare the performance of the baselines and our method on the upward extension task, where "Ours" represents the unlearning boundary condition in Phase I, which the point of the highest degree of unlearning completeness. The results suggest that our method also significantly outperforms the baselines on the upward extension task.
Figure 12:
31

---

Published as a conference paper at ICLR 2025
Ground Truth
Input
Original Model
Max Loss
Retain Label
Noisy Label
Composite Loss
Quts
(a) Forget Set
Ground Truth
Input
Original Model
Max Loss
Retain Label
Noisy Label
Composite Loss
Qurs
(b) Retain Set
Figure 13: Leftward extension VQ-GAN. We retain 50% of the right half of the image. The upper half (a) is the forget set, and the lower half (b) is the retain set. For each set, we compare the performance of the baselines and our method on the upward extension task, where "Ours" represents the unlearning boundary condition in Phase I, which is the point of highest degree of unlearning completeness. The results suggest that our method also significantly outperforms the baselines the upward extension task.
32

---

Published as a conference paper at ICLR 2025
Ground Truth
Input
Original Model
Boundary-oe Cass Highest Degree
Boundaryy- -onas Loweet Degree
(a) Forget Set
Ground Truth
Input
Original
Boundary Feats
Boundary
(b) Retain Set
Figure 14: Outpainting by VQ-GAN under different degrees of unlearning completeness, We retain 25% of the image center. The upper half (a) is the forget set, while the lower half (b) is the retain set. For each part, we compare the unlearning effects of our method different values of a. "Highest" and "Lowest" represent the conditions of the highest and lowest degree of unlearning completeness, respectively. We increase 16% each time.
33

---

Published as a conference paper at ICLR 2025
1.2.2 UPWARD EXTENSION TASK
We retain the lower half of the image center and crop the upper half, employing VQ-GAN for image extension. As shown Figure 13] results indicate that, with an increase in the value of e, the upward extension effectiveness improves. Concurrently, the degree of decrease in upward extension effectiveness on the retain set, in comparison to the original model, also diminishes.
the forget set of our controllable unlearning framework gradually
Input
Original
Boundary-Fate
Boundary-
(a) Forget Set
Ground Truth
Imput
Original Modal
Feus
Boundary-fau Lowest Degree
Highest Degree
(b) Retain Set
Figure 15: Upward extension by VQ-GAN under different degrees of unlearning completeness. We retain 50% of the lower half of the image. The upper half (a) is the forget set, while the lower half (b) is the retain set. For each part, we compare the unlearning effects of our method at different values of "Highest" and "Lowest" represent the conditions of the highest and lowest degree of unlearning completeness, respectively. We increase & 16% each time.
1.2.3 LEFTWARD EXTENSION TASK
We retain the right half of the image and utilize VQ-GAN to extend the image from the left. The results in Figure 16 demonstrate that the leftward extension performance on the forget set of our controllable unlearning framework progressively improves with the increase of e, and the reduction in leftward extension performance on the retain set also diminishing compared to the original model.
34

---

Published as a conference paper at ICLR 2025
Ground
Input
Original Model
Boundary Highest Degree
Lowest Degroe
(a) Forget
Ground Truth
Imput
Original Model
Boundany-faae
Boundary Hhww Lowest Degice
Highest Degice
(b) Retain Set
Figure 16: Leftward extension by VQ-GAN under different degrees of unlearning completeness. We retain 50% of the right half of the image. The upper half (a) is the forget set, while the lower half (b) is the retain set. For each part, we compare the unlearning effects of our method at different values of e. "Highest" and "Lowest" represent the conditions of the highest and lowest degree of unlearning completeness, respectively. We increase 5 16% each time.
35

---

Published as a conference paper at ICLR 2025
1.3 VARYING CROPPING PATTERNS AND RATIOS
In the preceding sections, we have demonstrated the performance of our controllable unlearning framework under various cropping patterns, yet the cropping ratio remained constant. By altering the cropping ratio on VQ-GAN, we validate the effectiveness of our controllable unlearning frame- work at different cropping ratios. The results indicate that our controllable unlearning framework is robust to different cropping ratios. Simultancously, compared to larger cropping ratios, the extent of variation in the images generated under our controllable unlearning framework will be smaller for smaller cropping ratios.
1.3.1 INPAINTING TASK
We retain one-sixteenth of the image center and use VQ-GAN for image inpainting. The results in Figure show that our controllable unlearning framework significantly outperforms the baselines in terms of unlearning effect on the forget set, most closely approximating Gaussian noise, and exhibits a lesser decline in unlearning effect on the retain set than the baselines. Simultaneously, can finely control the balance between unlearning completeness and model utility.
I.3.2 DOWNWARD EXTENSION TASK
We crop the bottom 25% of the image and utilize VQ-GAN for image extension from the bottom. As shown in Figure 19 the results demonstrate that our controllable unlearning framework significantly surpasses the baselines in terms of the unlearning effect on the forget set, closely approximating Gaussian noise, and shows lesser reduction in unlearning effect the retain set compared to the baselines. At the same time, we can finely adjust the balance between unlearning completeness and model utility.
1.3.3 RIGHTWARD EXTENSION TASK
We crop the right 25% of the image and utilize VQ-GAN for image extension from the bottom. The results in Figure 2T demonstrate that our controllable unlearning framework significantly surpasses the baselines in terms of the unlearning effect on the forget set, closely approximating Gaussian noise, and shows a lesser reduction in unlearning effect on the retain set compared to the baselines. At the same time, utility.
finely adjust the balance between unlearning completeness and model
J T-SNE ANALYSIS FOR CONTROLLABLE UNLEARNING
In To the main paper, we present the evaluation metrics corresponding different degrees of unlearning completeness solutions (i.e., IS, FID and CLIP) obtained by our controllable unlearning framework mainstream 12I generative models. Here, we analyze the images generated at differ- ent degrees of unlearning completeness for each corresponding model. We use TTSNE analysis to compare the clip embedding distances between the images generated on the forget set and retain set and the ground truth images. As shown in Figure 23 for any model, under the highest degree of unlearning completeness, the distance between the clip embeddings of the images generated on the forget set by the unlearned model and the ground truth images is larger, while the distance the retain set is smaller. Simultaneously, as e increases, the distance between the clip embeddings of the images generated
the forget by the unlearning model and the ground truth images gradually decreases (still significantly higher than the situation of the retain set), and the distance on the retain set also gradually decreases. Lastly, among these three mainstream 12I generation model structures, the effect of VQ-GAN is the most significant.
K EFFICIENCY EXPERIMENTS FOR CONTROLLABLE UNLEARNING FRAMEWORK
In the main paper, analyze the convergence efficiency corresponding to different control func- tions (+) at each phase from a theoretical perspective, and based upon this analysis, we aim to enhance the unlearning efficiency of our controllable unlearning framework. Here, validate
36

---

Published as a conference paper at ICLR 2025
Ground Truth
Input
Original Model
Max
Retain Label
Noisy Label
Composite Loss
(a) Forget Set
Ground Truth
Original Model
Max
Retain Label
Noisy Label
Composite Loss
(b) Retain Set
Figure 17: Generated images of cropping 25% at the center of the image. We crop the center 1/16 of the image. The upper half (a) is the forget set, and the lower half (b) is the retain set. For each set, compare the performance of the baselines and method the inpainting task, where "Ours" represents the extreme case of the unlearning boundary in Phase I, that is, the point of highest degree of unlearning completeness.
37

---

Published as a conference paper at ICLR 2025
Ground Truth
Inout
Original Model
Boundary--e.. fars Highest Degros
Boundary- famc Lowest Dogres
(a) Forget Set
Input
Original
Boundary
Boundary
(b) Retain Set
Figure 18: Generated images of cropping 50% the of the image under different degrees of unlearning completeness requirements. We crop the central 1/16 of the image. The upper half (a) represents the forget set, and the lower half (b) represents the retain set. For each section, compare the effectiveness of our method's unlearning under different values of s. Here, "Highest" and "Lowest" indicate the conditions of the highest and lowest degree of unlearning completeness, respectively.
38

---

Published as a conference paper at ICLR 2025
Ground Truth
Input
Original Model
Max Loss
Retain Label
Noisy Label
Composite Loss
Ours
(a) Forget Set
Input
Original Model
Noisy Composite
(b) Retain Sct
Figure 19: Downward extension by VQ-GAN. We crop the bottom 25% of the image. The upper half (a) is designated as the forget set, and the lower half (b) as the retain set. For each section, we compared the performance of the baselines and our method on the downward extension task, where "Ours" denotes the unlearning boundary condition in Phase I, that is, the point of highest degree of unlcarning completeness.
39

---

Published as a conference paper at ICLR 2025
Ground Truth
Input
Original Model
Boundary-Kair
Boundary-ce Cuos Lowest Dogree
Highest Degroe
(a) Forget Set
Ground Truth
Input
Original Model
Boundary-tosin Highest Degree
Boundary Lowest Degree
(b) Retain Set
Figure 20: Downward extension by VQ-GAN under different degrees of unlearning completeness. We crop the bottom 25% the image. The upper half (a) represents the forget set, and the lower half (b) represents the retain set. For each section, we compare the effectiveness of method's unlearning under different values of e. Here, "Highest" and "Lowest" indicate the conditions of the highest and lowest degree of unlearning completeness, respectively.
40

---

Published as a conference paper at ICLR 2025
Ground Truch
Input
Model
Max Loss
Retain Label
Noisy Label
(a) Forget Set
Ground Truth
Original Model
Retain Label
Noisy Label
Composite
(b) Retain Set
Figure 21: Rightward extension by VQ-GAN. We crop the right 25% the image. The upper half (a) is designated as the forget set, and the lower half (b) as the retain set. For each section, we compared the performance of the baselines and our method on the rightward extension task, where "Ours" denotes the unlearning boundary condition in Phase I, that is, the point of highest degree of unlearning completeness.
41

---

Published as a conference paper at ICLR 2025
Ground Truth
Imput
Original Model
Beun
Boundary- Feorr Lowest Degree
Highest Degree
(a) Forget Set
Ground
Boundary-Sa.
Boundary-Soow
Model
Highest Degree
Lowest Degree
(b) Retain Set
Figure 22: Rightward extension by VQ-GAN under different degrees of unlearning completeness. We crop the right 25% of the image. The upper half (a) represents the forget set, and the lower half (b) represents the retain set. For each section, we compare the effectiveness of our method's s unlearning under different values Here, "Highest" and "Lowest" indicate the conditions of the highest and lowest degree of unlearning completeness, respectively.
42

---

Published as a conference paper at ICLR 2025
Hight Degine
Dog
( Higheen Degrne
pyr 20%
-1 59%
Polee 79%
20 Loveet Degine
@ Highern Degine
pau- 190
Loveent Degin
(3 Diftaiion
Figure 23: T-SNE analysis between images generated by our method and ground truth images under different degrees of unlearning completeness.
analysis on three mainstream 121 generative models. During the two different phases of controllable unlearning, we design the form of the control function (+)) separately.
Specifically, in Phase I, we set (b() = a (f)] where we test the convergence rates of (0) and f2), as well the overall convergence rate, for 5 1, d 2, 5 3, and 5 4. As shown in Figure 24, is apparent that Phase for 2, that is (0) a (0) 2 the overall convergence rate is optimal.
In Phase II, we set (²)) (0) - e)S, where we tested the convergence rates for 8 and 3. Subsequently, we changed the form of (0) to (0) = ((f [ (0)]) and we tested the convergence rates for 8 and 8 = 3. Comparing the aforementioned scenarios, the overall optimal convergence rate in Phase II is obtained when (0)) - ())
43

---

Published as a conference paper at ICLR 2025
(a-1) (01)
(a-ii) (1]1
(a-ii) (1)1
35
0.2
05
36-
065
0.5
86-
04
04
84-
(a) MAE-W/
0.2
02
02
30
(6-1) S(A/ (8) e)²
(e-vi) 5(A (6) -8)
(a-vii) 5(4f (0) -) (+)1 (e-vili) (1/ (6) 9)" (1AA
00-
(b-) 10 (#)1)'
(b-ii) y9llol (0)1I"
(b-ii) (()1
(b-i) 10]l4f (8)11*
MW
(b) VQ-GAN-((()
(b.v) 10(4f) -)²
(b-vi) (0(4f) (0))00)))0 -cy
(b-vii) 10(45 (0) (1S)())) (b-vii)) 10(4fi (0) 110Л
20
20
10
(c-) (F1PP
(o-1i) (e)1
(c-10 [|& (B)1F
(c-li) |laf (#)11
101
M Mmm
(9) Diffusion Model-pp(0)
(A/1 =-
(4/1 (8) 2)²
[c-vi) (2/1 1148100011
(c-vii)) (A/1
(1011
m
The convergence rates under different control functions (0) As illustrated in figure, include three sections: MAE, VQ-GAN, and the diffusion model. Each section contains two rows, corresponding to Phase I and Phase II, respectively. The titles on each subplot indicate the forms of the control function (8).
Figure 24:
44


Reviews:
[{"review_id": "kn9Gs6YsaT", "reviewer": "Reviewer_NJMm", "summary": "The paper proposes a machine unlearning approach for generative image-to-image models. Machine unlearning algorithm in generative domain aims to make the model forget a specific subset samples (for e.g. defined by classes) while retaining its generalization capability on the other samples in order to address issues related to privacy and biases. The paper proposes a controllable unlearning algorithm flexible enough to balance between quality/degree of unlearning concepts and the model’s generalization capabilities. The approach uses a gradient based method to solve a constraint optimization objective where the constrain is to forget a certain specified set while retaining its reconstruction quality on remaining samples. The paper also shows theoretical analysis of its approach using Pareto optimality. The paper shows quantitative and qualitative results on in-painting/out-painting tasks to demonstrate the efficacy of the proposed approach.", "questions": "It would be helpful if the paper can answer/comment on the following question/suggestion:\n\n1. Is it possible to formulate the unlearning objective that simply in-paints with background content ( i.e. instead of predicting a gaussian type patch in the image for in-painting task, the model predicts the background and does not generate the subject that is to be forgotten). Does this require modification in the formulation that uses Divergence(P_Xf | N(0, sigma)) as condition.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper explores an unlearning approach for generative image-to-image models that uses gradient based method to solve a constrained optimization objective. The paper explains the issues present in the current machine unlearning domain and address these issues using a controllable optimization where the users have control over the unlearning optimization (model unlearning while maintaining model generalization). The proposed framework shows better results on ImageNet-1k and Places-365 dataset for in-painting tasks compared to other baseline unlearning approaches. The paper provides detailed ablation experiments and theoretical analysis to explain its proposed algorithm. The paper is well-written, easy to follow and contains a pseudocode that explains the methodology clearly.", "weaknesses": "It would helpful for the reader to see some discussions around the robustness of the concepts removal. For example is it possible to use some attack that resurfaces the forget set, for example as shown in paper Petsiuk, Vitali, and Kate Saenko. \"Concept Arithmetics for Circumventing Concept Inhibition in Diffusion Models.\" arXiv preprint arXiv:2404.13706 (2024).  \nIt would be helpful for the readers if some more related unlearning papers are added as references: \n\n[1] Petsiuk, Vitali, and Kate Saenko. \"Concept Arithmetics for Circumventing Concept Inhibition in Diffusion Models.\" arXiv preprint arXiv:2404.13706 (2024)\n\n[2] Kumari, Nupur, et al. \"Ablating concepts in text-to-image diffusion models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pEJxB56MEG", "reviewer": "Reviewer_icG6", "summary": "This submission formulates the controllable I2I unlearning problem as a $\\epsilon$-constrained problem, which differs from the prior objective. By reformulating the problem as a $\\epsilon$-constrained bi-objective function, two Pareto optimal solutions and the valid range of the control coefficient $\\epsilon$ can be obtained. Furthermore, the authors provide a theoretical analysis of the convergence of the proposed method under various control functions used to govern the direction of parameter updates. The experimental results on two well-known benchmarks show the effectiveness over the mentioned baselines.", "questions": "- Why the results of Composite Loss is different from those reported in [1]? Please provide more details of implementation differences about it.\n- According to Fig.4, why the visualization of the retained set of MAE is changed after unlearning? This is quite different from [1].\n- Can you provide experimental results to demonstrate the proposed enjoys better unlearning efficacy than other methods? The theoretical results sometimes are different from real practice.\n\n[1] Machine unlearning for image-to-image generative models, ICLR 2024", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is sound. The proposed method reformulates the I2I unlearning problem by integrating the $\\epsilon$-constrained method which is widely used in multi-objective optimization. This integration makes the unlearning degree controllable and brings a few theoretical merits, such as convergence analysis.\n- This submission is well written and organized, which reduces the difficulty in reading and comprehending.", "weaknesses": "- This could be an improvement of [1] based on $\\epsilon$-constrained method. Technically, please provide the specific design of $\\epsilon$-constrained optimization for the I2I unlearning problem. And why $\\epsilon$-constrained method is required to integrate with the I2I unlearning problem?\n- Some claims are not evaluated. For instance, in line 70,  how the challenge ``First and foremost, this approach offers a solitary resolution,..’’ is addressed? \n- Evaluation of different crop sizes should be conducted. In practice, not only the degree of forgetting but also the size of crop area is defined by users.\n\n[1] Machine unlearning for image-to-image generative models, ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5PUCp0evO9", "reviewer": "Reviewer_hwTN", "summary": "In this work, the authors study the problem of machine unlearning (MU) in image-to-image (I2I) generative models. Unlike prior studies, this approach diverges from a single objective to better consider the tradeoff between unlearning completeness and model utility, offering more flexibility for varying user needs. Specifically, the authors first reformulate the bi-objective MU problem into a constrained optimization problem and then propose a gradient-based algorithm to find Pareto optimal solutions. The proposed algorithm comes with a theoretical guarantee for convergence. Additionally, empirical results show that the proposed method provides a good balance between the two objectives, performing competitively among baselines.", "questions": "Is the proposed method applicable to text-guided I2I generative models, such as image editing models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed algorithm is well-motivated and comes with a theoretical guarantee for convergence, laying a solid theoretical foundation for application.\n2. The authors identify an overlooked issue in MU for I2I generative models in previous works: the failure to cater to varying user expectations in the real world, $i.e.,$ lack of controllability. Based on this observation, they derive a novel solution to this new bi-objective problem, which has practical significance for improving I2I generative models.\n3. The empirical findings align with the theoretical results, demonstrating that the solutions found by the proposed algorithm achieve good performance in terms of both objectives, $i.e.,$ unlearning completeness and model utility.", "weaknesses": "1. The definition of unlearning completeness in the paper is problematic. The paper uses the KL divergence between distributions of forget data and reconstructed data to evaluate the completeness of unlearning, ultimately approximating it with the L2 loss. However, both losses are not ideal criteria for assessing unlearning performance, as they are defined in pixel space and disregard the original image manifold. Generative models can exploit this by outputting inconsistent pixel values when operating on the forget set, leading to suboptimal unlearning results. This is evident in the artifacts in reconstructed image examples from the forget set in Appendix F, $e.g.,$ in the inpainting task.\n2. The proposed algorithm doubles memory usage, as it requires storing two separate model gradients. It also involves model-level gradient operations (as described in Algorithm 1, Line 8), making it more complex than other baselines. This can be less practical for larger models. A detailed computational complexity comparison and a discussion on memory usage would be helpful.\n3. Confusing evaluation. In Table 1, the Inception Score (IS) appears in both columns for the forget set and retain set. It is unclear why IS should be \"the less the better\" (if that is the meaning of the down-arrow) for the forget set. If unlearning completeness is linked to low-quality generation, then the objective becomes trivial—a simple classifier to detect forget data would suffice. Echoing my previous point in W1, the generative model should at least produce a natural or similar image, even if the input is out-of-distribution. This requirement is completely overlooked here.\n4. Minor typos. For example, in Equation 1, $I_\\theta=D_\\phi(E_\\gamma(\\mathcal{T}(x)))$ should be $I_\\theta=D_\\phi(E_\\gamma(x))$ to be consistent with the rest of the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)