Targeted Unlearning with Single Layer Unlearning Gradient
Authors: Zikui Cai
Yaoteng Tan
M. Salman Asif
Venue: ICML 2025 poster (2025)
Keywords: Machine unlearning
foundation models
CLIP
vision-language model (VLM)
Stable Diffusion
privacy protection
copyright protection
trustworthy and safe machine learning
Primary area: deep_learning->robustness
TL;DR: We propose a highly efficient machine unlearning method for foundation models that requires only one-time gradient calculation and partial model modification (one layer).

Abstract: Machine unlearning methods aim to remove sensitive or unwanted content from trained models, but typically demand extensive model updates at significant computational cost while potentially degrading model performance on both related and unrelated tasks. We propose Single Layer Unlearning Gradient (SLUG) as an efficient method to unlearn targeted information by updating a single critical layer using a one-time gradient computation. SLUG uses layer importance and gradient alignment metrics to identify the optimal layer for targeted information removal while preserving the model utility. We demonstrate the effectiveness of SLUG for CLIP, Stable Diffusion, and vision-language models (VLMs) in removing concrete (e.g., identities and objects) and abstract concepts (e.g., artistic styles). On the UnlearnCanvas benchmark, SLUG achieves comparable unlearning performance to existing methods while requiring significantly less computational resources. Our proposed approach offers a practical solution for targeted unlearning that is computationally efficient and precise. Our code is available at https://github.com/CSIPlab/SLUG

Targeted Unlearning with Single Layer Unlearning Gradient
Zikui Cai 12* Yaoteng Tan
M. Salman Asif'
Abstract
tion (Thiel, 2023) and demands for unlearning. Completely abandoning trained large models and retraining them from scratch using scrutinized dataset is prohibitively expensive and wasteful. Machine unlearning (Cao & Yang, 2015; Nguyen et al., 2022; Chakraborty et al., 2024) is an attractive alternative, which refers to techniques designed to remove targeted information from a trained model.
Machine unlearning methods aim to remove sen- sitive or unwanted content from trained models, but typically demand extensive model updates significant computational cost while potentially degrading model performance on both related and unrelated tasks. We propose Single Layer Un- learning Gradient (SLUG) as an efficient method to unlearn targeted information by updating a sin- gle critical layer using a one-time gradient compu- tation. SLUG uses layer importance and gradient alignment metries identify the optimal layer for targeted information removal while preserving the model utility. We demonstrate the effective- ness of SLUG for CLIP, Stable Diffusion, and vision- Laiiuue models (VLMs) in removing con- crete (e.g., identities and objects) and abstract con- cepts (e.g., artistic styles). On the UnleamCanvas benchmark, SLUG achieves comparable unlearn- ing performance to existing methods while re- quiring significantly less computational resources. Our proposed approach offers a practical solution for targeted unlearning that is computationally efficient and precise. Our code is available https: /ttt:////gtttbttt// cwm/csIP cessspabb//// SLUG.
A good machine unlearning technique should achieve three main objectives: (1) Computational efficiency. The naive approach of re- re-training models achieves exact unlearning but at an enormous computational cost. An effective solu- tion must minimize the computational overhead. (2) Effec- tive and robust unlearning. Recent studies have raised concerns about the robustness of unlearning methods, show- ing that unlearndd concepts can often be recovered through careful probing or adversarial techniques (Zhang al., 2025; Petsiuk & Saenko, 2025; Che et al., 2025). The unlearn- ing process must be robust against such recovery attempts. (3) Targeted removal with minimal side effects. The in- terconnected nature of learned representations means that removing one concept can lead to degraded performance on (un)related concepts and overall model performance (Amara et al., 2025). Unlearning should precisely target specific information while preserving general model capabilities.
Current unlearning methods often fail to meet all three ob- jectives simultaneously. Traditional approaches like fine- tuning (FT) (Warnecke et al., 2023) and gradient ascent (GA) (Thudi et al., 2022) struggle to balance between ef- fective forgetting and utility preservation. More recent techniques such as saliency unlearning (SalUn) (Fan et al., 2024) and selective synaptic dampening (SSD) (Foster et al., 2024) attempt to address this by identifying and updating only salient parameters. While these methods represent the state-of af-the-art, they still face key challenges: (1) High computational cost, as they typically involve iterative gra- dient computation and parameter updates across the entire model. (2) Limited generalizability and fiexibility, as they are often designed for single tasks (e.g. classification, image generation). (3) Side effects, as parameter updates spread over the entire model can cause unintended changes to (un)related concepts and degrade overall model perfor- mance. (4) Human engincering, as they require extensive hyperparameter tuning for learning rate, number of itera- tions, and thresholds for masking in parameter updates. We
Introduction
Modern large foundation models, including large language models (LLMs) (Leiter et al., 2024), text-to-image dif- fusion (Salimans & Ho. 2022: Yang et al., 2023) and lasion-language models (VLMs) (Zhang et al., 2024b; Liu et al., 2024b) leverage vast amounts of data for training. While these large scale datasets enhance performance, they also raise serious data privacy and legal compliance (gdp, 2016; Thiel, 2023) concerns unwanted data influence the trained models, resulting in harmful content genera-
Eqqual contribution Uniiversity of California Riverside, River- side, CA, USA University Maryland, College Park, MD, USA. Correspondence to: M. Salman Asif esasif@uer.edu..
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Targeted Unlearning with Single Layer Unlearning Gradient
Input Prompts
Original Model Fi
Generated Images
Unlearn Request
"A portrait of Elon Musk", "Elon Musk on Mars"
Forget "Elon Musk"
portrait Mark Zuckerberg", "A sea turtle in the ocean"
Train set
Fg (8)
Sample "Elon Musk"
Identify Single Layer
Calculate Single Gradient
Compute Layer Importance
Forget Gradient
Dr
C Cheent De)lla
Importance
togget (0, Dt)
Forget set
0illa Compute Gradient Alignment
8
Retain Gradient
D+
cos(V CVr,Cis 0, D.), Vo, Cetain (, D.))
Gradient Alignment
V V (0, D+)
Retain set
Single Layer Update 0: Ur AV
portrait of Elon Musk", "Elon Nusk Nars"
"Elon Musk" is unlearned
"A portrait of Mark Zuckerberg", "A sea turtle in the ocean"
Other concepts remain unaffected
Unlearned Model
Figure Overview of proposed Single Layer Unlearning Gradient (SLUG) framework. Given an unlcarning request, we curate a forget set and retain set, then compute gradients w.r.t corresponding loss for one-time. We then identify the most critical layer to update that has larger importance for forgetting, and smaller gradient alignment between forget and retain. A binary search helps determine the step size model updating, ensuring targeted forgetting while retaining model utility.
present additional discussion on related work in Section A.
forget set and low relevance to the retain set. (3) Update the targeted layer along linear path using the computed forget gradient. Below we discuss details about the unlearning problem formulation, layer identification, unlearning via single gradient, and generalization to different models. We have included pseudocode for SLUG in Section B.
To address these challenges, we propose Single Layer Unlearning Gradient (SLUG) (see Figure for illustra- tion) to overcome the key challenges listed above. SLUG uses layer importance and gradient alignment metrics to identify single optimal layer for targeted information re- moval while preserving the model utility. SLUG minimizes the computational cost by requiring one-time gradient computation and single layer updates. Furthermore, SLUG concentrates changes in a single layer and introduces strong flexibility, making post-training large scale model updates more modular and efficient compared to existing methods.
2.1 Unlearning problem formulation
Preliminaries. Suppose we are given a model FD(D) with parameters 0 trained on dataset D) with N samples, Our goal is to remove the influence of a specific forget set D, C D. consisting Ng samples, on Fg (D). The challenge is to make this process more efficient than retraining the model the retain set D. = D \ Dr with N, samples. We seek to develop an unlearning algorithm U that produces unlearned model Fe; (FF) D, Dr), which is functionally equivalent to model Fe, (D+) that is retrained only on D,. We can formulate the unlearning problem as minimizing some loss functions defined on the retain and forget sets:
We demonstrate the effectiveness of SLUG across CLIP, Stable Diffusion, and vision-language models in removing concrete (e.g., identities and objects) and abstract concepts (e.g., artistic styles). Our experiments demonstrate SLUG effectiveness across all three key objectives. For efficiency, achieve state-of-the-ar results on the UnlearnCanvas benchmark while requiring only fraction of the computa- tional resources and tiny storage. For precision, we show minimal impact on related concepts and image quality. In terms of robustness, we evaluate against recent vulnerabili-- ties identified by Zhang et al. (2025) and Petsiuk Saenko (2025), demonstrate the effectiveness of our method.
min Cretain (0, Dz) a. (0, De),
(1)
where denotes a regularization parameters.
Loss functions for vision-language alignment. For tradi- tional image classification models, cross-entropy loss can be directly used as both retain loss and forget loss. In this paper, we focus on large multi-modal foundation models such as CLIP (Radford et al., 2021), Stable Diffusion (Rombach et al., 2022), and vision- vision-language models (Liu et al., 2024b) that rely on vasion-language alignment. CLIP, particu-
2 Single Layer Unlearning Gradient
The core principles of SLUG can be summarized follows. (1) Compute one-time gradients for the forget and retain losses. (2) Identify single layer with high importance to the
2

---

Targeted Unlearning with Single Layer Unlearning Gradient
lar, is pivotal in advancing multi-modal models by aligning visual and textual representations through contrastive loss (Chopra et al., 2005). In unlearning, one of our goals is to break these learned alignments, so that one modality is non-retrievable with the corresponding modality.
the &g norm of the layer / parameters 0f:
Importance(() Vo, Liorges (0, Dz)lla
(5)
|0:12
This choice of importance is inspired by the use of Fisher information matrix to measure the influence of model pa- rameters (Foster et al., 2024; Kay, 1993; Hassibi et al., 1993; Kirkpatrick et al., 2017). Note that the Fisher information matrix for O₁ can be defined as
For the retain set, we use the original contrastive loss that can be defined as
No (lozel)) (2),
feetaio (0, D2)
1
Σ
2N,
8 [(00, (00, 00:
c(o. (
(0: =E
C(0: ((O.DD)
(6)
where
exp(cos(v: tz)/T)
where L denotes the logelikelihood loss (i.e., score) function (which can be the forget loss in our case). The diagonal elements reflect the sensitivity of the log-likelihood to the parameter changes, and the & norm of the diagonal entries is identical to the &g norm of the forget loss gradient.
(1) = log
2j=1 exp( f))/T) (3)
exp(cos(t;, VT)
(1)(()) log
2j=1 exp( cuspcoo(tt,
Here, V; is the normalized image embedding from the vision model fv; and is the normalized text embedding from the text model ft. The temperature controls the sharpness of the softmax probability distribution, while cosine similarity is defined as cos(v,, ty) = V, tt. Minimizing this con- trastive loss aligns the vision and language representations in the embedding space.
Importance of layer alone is insufficient for balancing learning and utility retention. We also ensure that forget gradients are nearly orthogonal to the retain gradients by minimizing the gradient alignment:
Alignment()) = cos (Ve, Ctorger Dr), V Cretain (0, D.)).
(7)
Small alignment between unlearn and retain gradients would prevent unlearning updates from negatively affecting the retain set.
For the forget set, we use the cosine embedding loss:
1 N₂ NI
Ciergel (0, (z)
Σ1 1- ty).
(4)
To balance both objectives, we search for Pareto-optima set across all layers (Marler & Arora, 2010), maximizing importance while minimizing alignment of forget and retain gradients. Figure 2 illustrates the Pareto front for unlearning a target identity from CLIP ViT-B/32, where colored dots represent layers that achieve optimal trade-offs between these impporie one metric necessarily worsens the other. We aim to find the optimal balance between unlearning and retention among these selected layers.
This loss directly pushes the embeddings of positive pairs away while not tampering with the embeddings of negative pairs. Using the original contrastive loss as forget loss will result in ineffective unlearning.
2.2 Single layer identification
Inspired by the findings that reveal distinct layers learned distinct features in deep networks from Zeiler & Fergus (2014); Olah et al. (2017): Ghiasi et al. (2022). We aim to identify and modify only the critical layers that contain fea- tures related to the unlearning tasks while avoiding changes to other layers, which capture abstract features unrelated to unlearning but essential for model utility. To minimize the impact on utility when modifying the model, we propose updating parameters along the unlearning direction within the "null space" of the retaining features. In other words, we focus on layers that minimally change retain set outputs while maximally changing the forget set outputs. This ap- proach balances the impact on retained data while precisely targeting features for unlearning, enhancing the precision of model modification.
2.3 Unlearning in a single gradient direction
Existing unlearning methods calculate gradients at each iteration to update model parameters, which significantly increases computational complexity. Inspired by task arith- metic (Ilharco et al., 2023) and the lincar nature of many optimization problems (LeCun et al., 2015), we observe that repeated gradient calculations can be redundant. Instead, we propose calculating the gradient only once for the ini- tial model and updating the parameters On of any layer / in weighh--rithhmen fashion. Specifically, the weights are updated along fixed gradient direction:
of - 0 - A Crorget (0, D:)
(8)
(=
We quantify the importance of a layer / to the forget set Dr using the ratio of the lg norm of the forget loss gradients to
where 03 represents the parameters of layer I for the un- learned model and o.0) represents the initial parameters. The
3

---

Targeted Unlearning with Single Layer Unlearning Gradient
Importance of Layers
10:1
80
Accuracy (%)
Accuracy (%)
10-4
10-9
visualgea
40
11 mipp tak
10-4
Xx
X
Porget Aoc95
Forget Aoeg3
AcoB1
Forput AccipS
Farget Acc
Calbb Accey
out proj
X
ImmageNet
ImageN et Accg]
attn
10-5
10-3
10-2
10"
109
0.0
0.5
1.0
1.5
2.0
2
10
Gradient Alignment
Step Size
Iters
(a) Layers vision model
(b) Unlearn a vision layer
(c) GA on whole model
Importance of Layerss
10-1
Accuracy (%)
Accuracy (%)
10.4
60
10-9
40-
40
10-
Iiipp youj
atinn.ntt proj
in proj
attn
X
X
20-
Forget Acoge]
Forget Ansis
ImsageNot Aoegg
Calobik Accdpo
20.
ImageNatt Anegi
Fonget Acrigo
Fonpui, Accipó
10-5
c.go co
10-2
10 100
1.0
2
10
Gradient Alignment
Step Size À
Iters
(d) Layers of language model
(e) Unlearn a language layer
(f) GAFT on whole model
Figure 2: Layer identification (a.d) and unlearning with single gradient (b,e). The first column shows gradient alignment and importance metrics for vision and language models from CLIP ViT-B-32. 32, highlighting layers the Pareto front for unlearning an identity. The second column demonstrates effective unlearning by updating identified layers along a single gradient direction without significantly impacting retain set performance. The third column shows that iterative methods (GA and GAFT) offer no advantage over a single gradient and require early stopping to prevent over-unlcarning.
gradient Ve, (f, D+)
10=0/o: is calculated only once, based on the forget loss Liunget evaluated on the forget set Dg. The step size X controls the update magnitude.
is a time-dependent noise balance factor, " is the guid- ance scale, e = fe(txt) is the text embedding, and Vx p(x is the gradient of the log-probability of the noisy image conditioning over the text embedding (also known as the conditional score function), guiding the de- noising process. We achieve single-layer unlearning on SD by applying SLUG to the text encoder (i.e., CLIP), enabling a layer-level flexible plug-in unlearner inspired by Zhang et al. (2024c).
Updating weights of a layer along a fixed gradient direction equivalent to linearizing the unlearning trajectory. This approach reduces computational complexity while ensuring effective unlearning. To select the appropriate step size X², we perform binary search along the linearized path, halting when the evaluation metric indicates satisfactory unlearning without harming performance on the retain set. For example, we stop at X R 0.75 in Figure 2b, where the forget accuracy near zero and test accuracy is high. This method strikes balance between computational efficiency and precision, maintaining model utility while achieving unlearning goals.
Unlearn Vision-Language Models. VLMs enable LLMs to process visual modality by employing pretrained vi- sion encoder f-). LLaVA-115 uses a pretrained CLIP vision encoder to extract the visual feature from images e fv(img), which are then projected to visual tokens H. = W- through an MLP W. These tokens are then con- catenated with language tokens Hg as input H = [Hy: Hg] to the language model. Since VLMs rely solely on the vision encoder to understand images, similarly unlearn SDs, we apply SLUG on the vision encoder of VLMs to influence the downstream text generation.
2.4 Generalization to Stable Diffusion and VLMs
Following the effective unlearning in CLIP models, our technique can be further extended to foundation models built on CLIP encoders, such as Stable Diffusion (SD) and VLMs like LLaVA (Liu et al., 2024b;a).
3 Experiments and Results
Unlearn Stable Diffusion. Text-to-image (stable diffusion) models use a pretrained text encoder Je(-) to project text prompts into nigh-dimensional vectors, which serve as guid- ance in the denoising process. A text-guided denoising step at time f is written as
In this section, we first provide a brief overview of the ex- perimental setup for each experiment. We then present key results demonstrating the effectiveness of SLUG in unlearn- ing CLIP, Stable Diffusion, and vision-language models.
Xt-1
Vare (xt Vx (xlog((xeee)) + vi anze
(9)
where x, is the noisy image, 2, is the random noise, Ot

---

Targeted Unlearning with Single Layer Unlearning Gradient
Musk-
0.31
0:15
Elon Musk
4..00
0.01
Mark Zuckerberg
0:17
0.30
0.10
0:00
Mark Zuckerbeng
0:11
0.25
0.34
0.07
Jeff Bezos
0.16
0.20
0.31
0.05
5.10
Jeff Bezos
9.08
0:15
0.26
0.01
0.09
Taylor
Taylor Swift-
Kim Kardashian
0.05
0:25
Kim Kardashian
0.02
0.02
0.14
0.27
(a) Original cosine similarity matrix
(b) Cosine similarity matrix after unlearning
Figure Cosine similarity matrix image-text pairs before after unlearning "Elon Musk" as an example. (a) original CLIP correctly associate images and text of distinct identities with high similarity. (b) after unlearning, the image-text text pair of "Elon Musk" is no longer matched, while other identities are only slightly affected.
Table 1: Performance comparison of different unlearning methods on CLIP zero-shot classification. FA [1, 5) stands for top- 5] forget accuracy (%), i.c., accuracy of unlearned identity. TA_ IN @ and TA CA@ I stands for the top-1 test accuracy (%) on ImageNet and CelebA dataset, respectively. and denotes the number epochs training and iterations for unlearning, respectively (K' 32
and k = 10 in our experiments). N is the training set size, which is much larger than our sampled forget set (N:) and retain set (N..).
METHOD
FA (L)
FA (L)
TA IN (1)
TA. CA@1 (1)
COMPUTE TIME (O)
ORIGINAL
73.05
92.22
60.12
61.38
O(K .N)
LEARNING RATE
10-
FT (WARNECKE ET AI., 2023)
66.08/70.50
90.10/92.22
60.36/60.26
60.70/61.3 .35
O(k N.)
GA (THUDI ET AL., 2022)
0.00/0.00
0.00/4.91
35.88/60.05
24.92/53.8
O(k N:)
GAFT (()) SALUN (FAN ET AL., 2024)
0.00/2.67
0.00/15.89 5.9
55.52/60.15 13
25.71/55.2 5.22
O(k- (N, + +N))
0.00/3.33
0.00/15.69
55.45/60.26
26.11/55.81
O(N.) + Olk (N. + N.))
SSD (FOSTER ET AL.. 2024)
0.00
0.00
51.84
35.96
O N.) O(N, + N.)
SLUG (OURS)
0.00
0.00
59.96
58.32
3.1
Experiment setup
Datasets. We used publicly-availate datasets construct the forget, retain, and validation sets. For identity unlearn- ing, we curated the forget set by filtering the LAION 440M dataset to isolate 1,000 to 6,000 image-text pairs per iden- tity. The retain set consists of a single shard from LAION- 400M, containing approximately 7,900 images (due to ex- piring URLs). To assess unlearning effectiveness, we used the CelebA dataset (Liu al., 2015), sampling 100 fre- quently appearing celebrities from LAION-400M, The util- ity of post-unlearning models were evaluated with ImageNet dataset. UnlearnCanvas was used to test unlearning of artis- tic styles and objects in Stable Diffusion.
Unlearning scenarios. Considering practicality, we explore three key unlearning scenarios: (1) unlearning celebrity identity information address privacy concerns, (2) learning copyrighted content comply with legal standards, and (3) unlearning artistic styles and object concepts in UnlearnCanvas (Zhang et al., 2024d). Our focus is on large- scale foundation models, including CLIP (Radford et al., 2021), Stable Diffusion (Rombach et al., 2022), and VLM (Liu et al., 2024b).
Models. We performed experiments on CLIP, Stable Dif- fuision (SD), and VLM to demonstrate the broad appli- cability of our method. For CLIP, we used architectures ranging from ViT-B-32 to EVA01-g-14, trained on LAION- 400M dataset (Schuhmann et al., 2021), and model weights sourced from the OpenCLIP repository (Cherti et al., 2023). For SD, we used SDv1.5 and SDv2.1 from StabilityAI, which employs CLIP-ViT-H-LE 14 text encoder, trained the LAION-5B dataset. For VLM, we used the improved L.LVVA-v1.5-7 1-577 model from HuggingFace, which employs a CLIP ViT-L/14-336px as vision encoder, from OpenAl. We provide a summary of model sizes in Appendix F.
Evaluation metrics. For CLIP, we measure unlearning per- formance using Forget Accuracy, defined as the zero-shot classification accuracy on unlearned content. Following the standard zero-shot paradigm (Radford et al., 2021), predic- tions are based on the highest cosine similarity between image and text embeddings. Utility is assessed via zero-shot accuracy on ImageNet and CelebA. For SD, we employ established metrics from UnlearmCanvas. For VLM, we de- fine Forget Accuracy as ratio: number of currently predicted instances / total number of instance, detailed in Section 3.4.
5

---

Targeted Unlcarning with Single Layer Unlearning Gradient
Hyperparameters. Our SLUG framework requires no man- ual hyperparameter tuning. We use binary search to de- termine the step size À for the one-step unlearning update (see Algorithm 3) that optimizes the trade-off between un- learning and retention metrics on a small validation subset. Across all experiments, we fix the number of binary search steps to 5 = 10. For validation at each search step, we use 5% of the test set for CLIP, 10 test-time generated images (not present in the forget training set) for SD, and 10- image subset per identity for VLM unlearning. We further discuss the trade-off between validation size and unlearning performance in Appendix B.I.
performing binary search for a step size that minimizes forget accuracy, SLUG effectively preserves model utility, as demonstrated in Figures 2b and 2e.
3.3 Unlearning for Stable Diffusion
In this section, we first present qualitative evaluation on identity unlearning in SD, later, we present comprehen- sive quantitative evaluation of SLUG on the established benchmark UnlearnCanvas.
Unlearning identity. We demonstrate the scenario of re- moving personal information on the latest SDv2.1. SLUG ensures SD from generating content related to the erased identity when given prompts corresponding to that iden- tity. Figure 4 presents examples of images generated by SDs before and after unlearning with different methods. Our method interestingly maps the targeted identity "Elon Musk to electronic circuits, consistently across various prompts, without compromising the model image genera- tion on non-targeted concepts (suffering from ripple effects (Amara et al., 2025)). In contrast, other methods not only struggle with generating images of other identities (e.g., Mark Zuckerberg) but also degrade the quality of generated images on non-targeted concepts. In Section D, we provide additional results on unlearning more celebrity IDs, and other scenarios, including copyright- and novel concepts erasure.
We compare with the state-of-the-er art methods along with classical methods. For CLIP unlearning, we compare with classical fine tuning (FT) (Warnecke et al., 2023), gradient ascent (GA) negative gradient (NG) (Thudi et al., 2022), and recent salient parameters-based SalUn (Fan et al., 2024), and SSD (Foster et al., 2024). We also compare with a two-stage GAFT approach (Fan et al., 2024), which first performs GA for l steps on the forget set, then fine-tunes for è steps on the retain set. For SD unlearning, we compare with 9 methods reported in UnlearnCanvas, detailed in Section 3.3.
Comparing methods.
3.2 Unlearning for CLIP
We demonstrate that modifying a single layer suffices un- learn an identity concept while preserving overall model utility. Figure 3 shows an example of unlearning Elon Musk from CLIP. Before unlearning (Figure 3a), image-text pairs of Elon Musk exhibit high cosine similarity, while after unlearning (Figure 3b), this similarity drops significantly, leaving other identities unaffected. Additional results for multiple identities and CLIP architectures in Section C (Ap- pendix) further confirm the generalizability of our approach.
Evaluation on UnlearnCanvas benchmark. To further demonstrate the unlearning effectiveness and efficiency of SLUG, we also evaluate its performance on the latest bench mark UnlearnCavvas (Zhang et al., 2024d), which focused on unlearning artistic style and object concepts in SDs. introduces a comprehensive set of metries, including UA (Unlearn Accuracy) for unlearning effectiveness, IRA (In- domain Retain Accuracy) and CRA (Cross-domain Retain Accuracy) for utility retention. The benchmark targets un- learning styles and objects on an SDv1.5 model fine-tuned to generate 20 different objects in 60 distinct styles, and focuses on unlearning one object/style at a time, yielding 80 unlearned models for evaluation. For dataset generation, the benchmark inputs the fine-tuned SD with the prompt: "A [object name] in [style name] style," to generate 20 images for each object-style pair, resulting in 24,000 images in total (as there are 1,200 object-style pairs). We curate the forget set for unlearning each style/object using the associated images from the UnlearnCanvas (i.e., 400 images per style and 1200 images per object).
A key strength of SLUG is its ability to maintain perfor- mance on non-targeted tasks/data. Table 1 shows zero-shot classification accuracy on ImageNet and CelebA, where our method outperforms alternatives in both unlearning ef- fectiveness and utility retention. Unlike iterative methods requiring extensive hyperparameter tuning, SLUG performs single gradient computation (O + NN)),, avoiding the trade-offs seen in brasient-based approaches, where high learning rates compromise utility and low rates lead to inef- fective unlearning.
Localizing layers. SLUG efficiently localizes critical layers for updates, reducing the search space from hundreds to just a few Pareto-optimal layers. Figure 2 highlights the critical layers for unlearning within a CLIP model, these layers balance layer importance (sensitivity to forget loss) and gradient alignment (minimizing impact on retained data). Colored dots represent Pareto-optimal layers, exhibiting high importance scores and low gradient alignment. By
In Table 2, we report the unlearning performance of SLUG benchmark metrics, along with other offe-offtheeart un- learning methods reported in UnlearnCanvas. Our method minimizes storage and computational time by only requiring the gradient values of a few layers on the Pareto front to be stored, and performing a one-step update along the gradi-
6

---

Targeted Unlearning with Single Layer Unlearning Gradient
'A photo of astronaut riding a horse mars'
portrait Elon Musk'
"Elon Musk "A portrair of Mark Mars'
cat jumping on a hed'
"A sea turtle the ocean
Zackerberg'
Original SD
Unlcarned Musk" with SLUU (ours)
Unlearned "Elon Musk with SalUn
Unlcarned "Elon Musk"" with ESD
Figure 4: Images generated by different SDs using column captions as prompts. First row: images generated by the original pretrained SD. Second row: outputs of the SD after unlearning "Elon Musk" using SLUG. Bottom two rows: outputs of the SDs after unlearning "Elon Musk" using SalUn and ESD. While all methods unlearned the targeted identity, SLUG superior preserving the utility of the original model.
Table 2: Performance overview of different unlearning methods on UnlearCanvas.. The best performance each metric highlighted in green, and significantly underperforming results, in henchmark criteria, are marked in red. Our method SLUG shows no significant underperforming, and achieves the best trade-off among unlearning, retaining, and efficiency.
EFPECTIVENESS
EFFICIENCY MEMORY (GB)
GAP RATIO
METHOD
UA (1)
STYLE UNLEARNING (%)
IRA
CRA (1)
UA (f)
OBJECT UNLEARNING (%)
IRA (7)
CRA (f)
FID (1)
(S) (L)
TIME
STORAGE (GB) (1)
(%) (1)
MEAN
ESD (GANDIKUTA E] AL., 2023)
98.58
80.97
93.96
92.15
55.78
44.23
65.55
6163
17.8
4.3
18.17
FMN (ZHANG AI., 2024A)
88.48
56.77
46.6
45.64
90.63
73.46
131.37
350
17.9
4.2
1.81
UCE (GANDIKOTA ET AL.,, 2024)
98.4
50.25
47.71
94.31
39.35
34.67
182.07 182.01
434
5.1
1.7
1.72
CA (KUMARI AL., 2023)
60.82
96.01
92.7
46.67
90.11
81,97
$4.21
734
10.1
4.2
2.44
SALUN (TAN ET AL.. 2024)
86.25
90.39
95.08
86.91
96.35
99.39
61.05
667
30.8
4.0
2.78
SEOT (LI ET AL., 2024B)
56.90
94.68
84.31
23.25
95.57
82.71
62.38
95
7.34
0.0
0.46
SPM (Lyu ET Al., 2024)
92.39
84.33
71.25
90.79
81.65
59.79
29700
6.90
0.0
84.73
EDur (Wu AL.,, 2024)
92.42
73.91
98:93
86.67
94.03
48.
81.42
1567
27.8
4.0
5,37
(Wo& HARANDI, 2024)
43.27
80.73
81.15
67.99
31.2
4.62
SLUG (OURS)
29
84.59
83.43
75.43
77.3
81.18
75.97
39
3.61
0.04
0.15
BEST (HYPOTHETICAL)
98.58
96.01
98.93
94.31
96.35
99.59
54.21
3.61
0.00
0.00
ent for unlearning. Despite being extremely efficient, our method does suffer from significant performance degra- there is
represents the normalized distance of method from best performing method; lower values indicate performance closer the best (hypothetical reference) method (see the BEST row of Table 2). To provide the summary statistics for each method (row) in Table 2, we compute the GR for UA, IRA, CRA for both style and object unlearning, FID, Time, and the sum of memory and storage (which are strongly related resources in practical deployments). This provides us 9-dimensional GR vector for each method. We then report the mean of GR vector in the GAP RATIO column in Table 2, which is proportional to the E distance between the GR vector of each method and the BEST (HYPOTHETICAL) method. The results indicate that SLUG has the smallest gap from the best reference method, further demonstrating the superior trade-off between effectiveness and efficiency
dation in any metric or task in UnlearnCanvas no red mark for SLUG row in Table 2. Our method achieves excellent trade-off between unlearning and retaining accu- racy. For qualitative evaluation, we provide visual examples of style and object unlearning in Appendix D.
Unified metric for UnlearnCanvas benchmark. To summa- rize the different performance metries of each method with single quantity, first define the Gap Ratio (GR) for metric and method m
GR(s, m)
8m
Sbest
(10)
Sbest
where 8m and Sbest represent the metric for method m and the best performing method, respectively. Intuitively, GR

---

Targeted Unlearning with Single Layer Unlearning Gradient
Table Quantitative evaluation SLUG identity unlearning LLaVA- 1.5-7B vision- sainu-language model. The table shows forget accuracy and performance retention on standard VLM benchmarks across celebrity identities. SLUG achieves effective unlearning with average forget accuracy dropping from 99.50% to 2.8%, while maintaining competitive performance on utility benchmarks compared to the original model, demonstrating targeted concept removal with minimal impact general model capabilities.
UNLEARN
FORGET
VLM BENCHMARK ScORE (1)
IDENTITY
ACCURACY (L)
MME COGNITION
MME
MMBENCI
GQA
PERCEPTION
(%)
99.50
323.57
1481.21
61.28
62.97
ELON MUSK
3.0
298.57
1354.61
60.70
61.34
TAYLOR SWIFT
2.0
334.64
1336.09
60.72
60.14
MARK ZUCKERBERG
7.0
343.57
1209.55
58.21
56.01
JEFF BEZOS
3.0
314.64
1315.32
60.40
61.43
KANYE WEST
4.0
314.64
1365.53
61.17
61.68
TOM CRUISE
0.0
351.79
1413.04
61.13
61.86
KIM KARDASHIAN
6.0
286.43
1249.54
60.42
60.14
BARACK OBAMA
0.0
288.57
1269.45
60.68
61.08
LADY GAGA
3.0
270.36
1178.45
58.55
55.58
BRUCE LEE
0.0
323.57
1266.21
60.44
60.22
AVERAGE
2.8+2.44
312.68426.57
1295.78473.78
60.24=1.02
39.95+2.22
2024e;
Chin et al., 2024) in Section D.2.
STOP
3.4
Unlearning for VLMs
In this section, we present results for SLUG on remov- ing celebrity identity from VLMs. As there a lack of an established VLM unlearning benchmark, we sample 10 dif- ferent targeted identities from the CelebA, with 100 images per identity, resulting in 1000 images in total, to create comprehensive validation set. We individually unlearn each identity from the original LLaVA-1.5-77 (Liu et al., 2024a), and evaluate the Forget Accuracy (FA) as
What's
What the content the
Image with drising the hackgrund There people walling the sidowall, and lloe...
The this image Don Musk.
The
Original
this image Mark Buckerberg
Asswer Unlearoed
The person this Image is jacksan.
The person this image Mark Ruckerburgg
The image features stop
nearty...
Figure 5: Unlearning "Elon Musk" LLaVA-1.5 with SLUG. After unlearning, the model fails to identify "Elon Musk", whereas other identities/conceps remain unaffected.
number of misidentified images total number of images
FA
(11)
For identification criteria, we input images associated with the targeted identity combined with the question prompt "What is name of the person in the image?" to the model, and check whether the model answer matches the corre- sponding celebrity name. To evaluate the utility retention of unlearned models, employ established VLM utility benchmarks: MME (Fu et al., 2023) GQA (Hudson & Man- ning, 2019), and MMBench (Liu et al., 2025b). These VLM benchmarks quantify performance of vision-language tasks, which cover a broad set of coarse-to-fine-grain questions on visual recognition and visual reasoning, char- acterizing the utility of VLM. The results in Table high- light that SLUG achieves effective unlearning while main- taining performance comparable to the original pretrained model, validating its effectiveness in the VLM context. Fig- ure provides a qualitative evaluation of our method. Our results demonstrate that SLUG successfully unlearned tar- geted identities from the VLM, while preserving utility.
that SLUG achieves. In Appendix D.4, we provide a de- tailed breakdown in terms of the effectiveness and efficiency aspects, along with GR evaluation using & and & norms.
Robust evaluation. Our results demonstrate that SLUG is robust key unlearning vulnerabilities revealed in Petsiuk & Saenko (2025); Zhang et al. (2025), which include black- box attacks that utilize the prompt arithmetic property of Stable Diffusion and model weight quantization. SLUG ef- fectively resists concept arithmetic attacks, causes minimal ripple effects on related concepts (Amara al., 2025), and maintains performance even under 8-bit weight quantiza- tion. The precise modification of single layer allows for targeted concept removal, ensuring controlled downstream text-guided generation tasks while preserving model utility. Detailed experiment setup and results are provided in Sec- tion D. Additionally, we provide robustness evaluations for SLUG against whitebox prompt attacks (Zhang et al.,
8

---

Targeted Unlearning with Single Layer Unlearning Gradient
Table 4:
Ablation studies on parameter selection strategies for CLIP unlearning. Both layer importance and gradient alignment are essential for selecting layer to perform effective unlearning and utility retention under the one-step unlearning update framework.
PARAMETER SELECTION
@A@ 1 (1)
FA@5 (1)
TA IN 1 (t)
TA CA@ (1)
"SALUN" (DISTRIBUTED WEIGHTS, IMPORTANCE ONLY)
4.44
11.33
48.23
37.38
SINGLE LAYER IMPORTANCE ONLY
0.0
0.0
21.04
42.00
SINGLE LAYER ALIGNMENT ONLY
0.0
5.56
31.08
54.16
SINGLE LAYER AT RANDOM
0.0
6.91
33.38
52.90
ALL PARETO FRONT LAYERS
0.0
0.0
59.92
51.64
ALL LAYERS
0.0
0.0
59.70
53.74
SLUG (TABLE I)
0.0
0.0
59.96
58.32
Ablation studies
paper does not provide a rigorous theoretical explanation for the success of single-layer updates for unlearning, and a formal understanding of when and why SLUG is effective remains an open question. This paper primarily focused vision-language models; we did not explore the applica- bility of SLUG to large language models (LLMs), which operate purely in the language modality. While our experi- ments demonstrate that the models seem to forget/unlearn targeted identities and concepts with simple and efficient ma- nipulation by SLUG; we cannot claim that those identities and concepts are completely erased/removed from the mod- els by SLUG. Adversarial attacks and prompts can poten- tially retrieve the unlearned information. SLUG (like many other unlcarning methods) lacks robustness against white- box adversarial attacks and its susceptibility to (whitebox and blackbox) relearning attacks needs further examination.
We conduct ablation studies to evaluate different parameter selection strategies and the effect of updating multiple layers under our one-step unlcarning framework. Following the setup in Table 1, we target five identities for unlearning CLIP and assess forget accuracy (FA) for unlearning effectiveness, as well zero-shot test accuracy on ImageNet (TA IN) and CelebA (TA_ (AA for utility retention assessment. The results are summarized in Table 4.
SalUn row shows that selecting weights across the entire net- work using only importance (SalUn-like strategy) performs worse than SLUG, with higher FA and lower TA. SINGLE LAYER IMPORTA NCE row shows that selecting single layer based gradient importance alone reduces FA to O, but significantly lowers TA_ IN and PA_CA, revealing utility loss. SINGLE LAYER ALIGNMENT row shows that using alignment alone yields FA 555.6%, with notable util- ity degradation, indicating ineffective unlearning. SINGLE LAYER AT RANDOM row, where we randomly select a layer without guidance, performs the worst overall. These results highlight that both gradient importance and alignment are crucial for balancing unlearning effectiveness and utility retention, further justifying that SLUG achieves the best trade-off under the one-step unlearning framework. Beyond single layer, we also explore updating all Pareto-optimal layers and all model layers, as reported in ALI. PARETO FRONT LAYERS and ALL LAYERS rows. While multi-layer updates improve FA by ~2,3%, they significantly increase computational cost. Furthermore, Figure 8 shows that select- ing and updating a single layer per concept enables modular unlearning of multiple identities simultaneously, without requiring multi-layer updates for every concept.
6 Conclusion
SLUG demonstrates that effective machine unlearning can be achieved through targeted single-layer modifications, offering a practical solution to the computational bottle- necks that have hindered large-scale model deployment and editing. Our results across CLIP, Stable Diffusion, and vision-language models reveal that the distributed nature of learned representations does not preclude precise, localized interventions-a. a finding that challenges conventional as- sumptions about the necessity of whole-model retraining extensive parameter updates. The efficiency of SLUG opens new possibilities for dynamic model adaptation, where rapid response to removal requests critical. The robustness of unlearning methods remains underexplored; particular, their resilience against sophisticated prompting strategies or recovery attacks and their stability across different model ar- chitectures and deployment conditions. Future work should prioritize investigating these robustness challenges to estab- lish unlcarning as a reliable model editing technology.
5 Limitations
While SLUG represents the first endeavor to achieve un- learning through single-layer updates, it presents trade-off between efficiency and effectiveness. SLUG demonstrates competitive performance with significant computational ad- vantages, but does not achieve state-of-the-art results simul- taneously across all metrics, tasks, and benchmarks. This
A Acknowledgments
This work is supported in part by an NSF grant (CCFL 2046293) and a UC SoCal HUB seed award.

---

Targeted Unlearning with Single Layer Unlearning Gradient
Impact Statement
Wu, Z., et al. Model tampering attacks enable more rigorous evaluations of Ilm capabilities. arXiv preprint arXiv:2502.05209, 2025.
This paper presents work whose goal is to advance the ef- ficiency of machine unlearning for large-scale foundation models. By improving the ability to selectively remove data influence, method contributes trustworthy AI, ad- dressing privacy concerns and regulatory compliance. While there are many potential societal consequences of our work, none that we feel must be specifically highlighted here.
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C.. Schmidt, L., and Jitsev, Reproducible scaling laws for con- trastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023. bttps://arae org/ abs 2212/22122222 07143.
References
Regulation (eu) 2/66/ of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec (general data protection regulation) (text with eea relevance). Of- ficial Journal of the European Union, vol. 119, pp. 1-88, 2016. bttps:://ur--ee eeree//eeaeeee....a.a..uu legal- httll/ooottnet////////////--///s 3A0J. 2.1.6__221111.... 01..9.....000 011.1.11...1..00..0............. ENG.
Chien, E., Pan, C., and Milenkovic, O. Certified graph unlearning. In NeurlPS 2022 Workshop: New Frontiers in Graph Learning, 2022. URL https: http////ppeeeeeeeeeeeeee net fort/?foum?eem www.::iwwwwwccecccccc
Chin, Z.-Y., Jiang, C. M., Huang, C.-.,, Chen, P.-Y,, and Chiu, W.-C. Promptig4ddeuuging: Red-teaming text-to-image diffusion models by finding problematic prompts. In Forty-first International Conference on Ma- chine Learning, 2024. URL https: http::::////eeeeiiere.e net / www.roorrridiiiio htt.:/oorummiooyyol1555555l5 Seriiddvvyyoooysaooo
Evaluation for the neurips machine unlearn ing competition. August 2023.
https:
Chopra, S., Hadsell, R., and LeCun, Y. Learning a sim- ilarity metric discriminatively, with application to face verification. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR 005), volume 1, pp. 539-546. IEEE, 2005. https: / ieeexplore. ieee org/ document/144777
www.:k/aggle..oom/ommettettteeee select=Mach hww.echhnnenneeeeeinnnnnneonno Metric.pdf.
Amara, I., Humayun, A. I., Kajic, I., Parekh, Z., Har- ris, N., Young, S., Nagpal, C., Kim, N., He, J., Vas- concelos, C. N., et al. Erasebench: Understanding the ripple effects of concept erasure techniques. arXiv preprint arXiv:2501, 098333, 2025. URL https:// arxiv. oww. /bw.. 09833.
Chundawat, V. S., Tarun, A. K., Mandal, M., and Kankan- halli, M. Can bad teaching induce forgetting? unlearn- ing in deep networks using an incompetent teacher. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 37, pp. 7210-7217, 2023. https: / //arxiv. www.vv.org/abbbbob 220..200000 08096.
Basu, S., Zhao, N., Morariu, V. I., Feizi, S., and Manju- natha, V. Localizing and editing knowledge in text-to- image generative models. In The Twelfth International Conference on Learning Representations, 2024, ht https: // //openreview net/ / forum? dm 9 6SOQ.
Dong, P., Bingjie. W. Guo. S., Wang, J., Zhang, J., and Hong, Z. Towards safe concept transfer of multi- modal diffusion via causal representation editing. In The Thirty-eighth Annual Conference on Neural Informaiinn Processing Systems, 2024. https:/ opwnrevien.....e net ftr.morrmmmmmo dd0.mmmss11a.caac.......
Cao, Y. and Yang, J. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on secu- rity and privacy, pp. 463-480. IEEE, 2015. https:// ieeexplore. Leee. org/
Fan, C., Liu, J., Zhang, Y., Wei, D., Wong, E., and Liu, Salun: Empowering machine unlearning via gradient- based weight saliency in both image classification and generation. ICLR, 2024. https: http///wxvvvvr..//./ abs/ hbs/2330.1250S
Chakraborty, T., Shayegani, E., Cai, Z., Abu-Ghazaleh, N., Asif, M. S., Dong, Y., Roy-Chowdhury, A., and Song, C. Can textual unlcarning solve cross- modality safety alignment? In Findings of the Association for Computa- tional Linguistics: EMNLP 2024, pp. 9830-9844, 2024. https:/ aawaarivv....... org/ abs / / 02575.
Foster, J., Schoepf, S., and Brintrup, A. Fast machine unlearning without retraining through selective synap- tic dampening. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 12051, 2024. https: //ttx/aaav.v///////aa/eae/aaaeve wrt. /200 07707. httpps://arxlv.orr///b//////
Che, Z.. Casper, S., Kirk, R., Satheesh, A., Slocum, S., McKinney, E., Gandikota, R., Ewart, A., Rosati, D.,
10

---

Targeted Unlearning with Single Layer Unlearning Gradient
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al. Mme: A com- prehensive evaluation benchmark for multimodal large language models. arXiv preprint preprit 2rw.vv:3306.13339.0 2023. https: wwt.iv..vvvv. www.i.wwgggo........o......ooott httg//b////////////////////// 6. 13394.
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pp. 7514-7528, Online and Punta Cana, Domini- can Republic, November 2021. Association for Computa- tional Linguisties. doi: 10.18663/1/2022..mmlll 595. URL pettps://acaantiinn 595.
Gandikota, R., Materzynska, J.,
J., and Bau, D. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2426-2436, 2023. https: //arxiv.org/ abs/ 2303.3340 07345.
org/ 2021.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adap- tation of large language models. International Confer- ence on Learning Representations, 2022. URL https: / /oopeneeieww net/ for.m?id-dninr forumm2.d=n2eeeeerFY.
Gandikota, R., Orgad, H., Belinkov, Y., Materzyúska, J., and Bau, D. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp- 5111-5120, 2024. https::/ http://pppccccccssceeeeeeoe httpps://penaaccesss. thecvf.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and positional question answering. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pp. 6700-6709, 2019. https: / ('openaccess.theeee www.wf.hcccccccmm http://cccccotttcccccceee 2019 For_Real-Woridd htt_ssuaa_eeessssniaa Reasoning and Compos
com/ http//ooteetttaee http://AC///222/ /agpee/a///accooaaa//////aaaaaan
Caafiee_Corceep__aiiiiingggo Models WACV 2024 11224_paprr.htm.
Ghiasi, A., Kazemi, H., Borgnia, E., Reich, S., Shu, M., Goldblum, M., Wilson, A. G., and Goldstein, T. What do vision transformers lear?? a visual exploration. arXiv preprint arXiv:2212.006727, 2022.
org/abs 2212.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A. Edit- ing models with task arithmetic. In In Proceedings of the 11th International Conference on Learning Represen- tations (ICLR 2023), 2023, https: /ptps://axxvv.rrgg abs Abs/22122 04089.
Goel, S., Prabhu, A., Sanyal, A., Lim, S.-N., Torr, P., and Ku- maraguru, Towards adversarial evaluations for inexact machine unlearning, arXiv preprint arXiv:2201.066 2022. attpps://artee org. abs/
Golatkar, A., Achille, A., and Soatto, S. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 9304-9312, 2020a. https: /awxiv.vvvv/////aa//a..////aaava org/ abs/ /1911991111 04933.
Jia, J., Liu, J., Ram, P., Yao, Y.. Liu, G., Liu, Y., Sharma, P., and Liu, S. Model sparsification can simplify machine unlearning. NeurIPS, 2023. /tttss://axiv...r abs/ 00s//2304.0493
Golatkar, A.. Achille, A., and Soatto, S. Forgetting out- side the box: Scrubbing deep networks of information accessible from input-output observations. In Computer Vision-ECCV 2020: 16th European Conference, Glas- gow, UK, August 23-28, 2020, Proceedings, Part XXIX 16, pp. 383-398. Springer, 2020b. attps://arxiv. arxiv. org/ aass//////a 2003. 02960.
Kay, M. Fundamentals of statistical signal process- ing: estimation theory. Prentice-Hall, Inc., USA, 1993. ISBN 0133457117, https: //dl:///ll../// arg. /di/ abs bbs.11..5555 005.55555111000 1151045.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the na- tional academy of sciences, 114(13):3521-3556 2017. https:/ http//axvvv/..///////e////..a../ .ow.voorg/geoo alss/ 00796.
Trabska-Barwinsse
Guo, C., Goldstein, T., Hanmun, A., and Van Der Maaten, L. Certified data removal from machine learning models. ICML, 2020. https:// http///axivv.v.. wwwp:ii.wwr./////oe//o//oo 03030.
Kumari, N., Zhang, B., Wang, S.-Y., Shechtman, E., Zhang, R., and Zhu, J.-Y. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22691 22702, 2023. https:/ https::/openaccess http//pppnncccccceeeoeoo wtw.eaaccesss.eee......e thecvf. com/ hemm/oontent/cccc2000 /html httl//umaarkk /Kumari_
Hassibi, B., Stork, D. G., and Wolff, G. Optimal brain surgeon and general network pruning. In IEEE inter- national conference on neural networks, pp. 299-299. IEEE, 1993. https: http::://eeeeeporee.... ieee. org/ Socument/29855

---

Targeted Unlearning with Single Layer Unlearning Gradient
Ablladingg_oonpptss___ieeeet---_ tomage Di Llffusioo_Moddee s_t______c______ pawp0reppppppp____ html.
Liu, Z., Luo, P., Wang, X., and Tang. X. Deep learning face attributes in the wild. In Proceedings of the IEEE inter- national conference on computer vision, pp. 3730-3738, 2015. https: / arxii, org/ abs 11///11111 7766.
Kurmanji, M., Triantafillou, P., Hayes, J., and Triantafillou, E. Towards unbounded machine unlearning. Advances in neural information processing systems, 36:1957-1987, 2023. https::/ http:://aaxiivvvo www.ariww.......///ooeea.aeaaa/e wws.//322.. 000.//302000800... 09880.
Lu, S.. Wang, Z., Li, L., Liu, Y., and Kong, A. W.-K. Mace: Mass concept erasure diffusion models. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 6430-6440, 2024. https://openaccss.tbeee com/ content//CVPRR003/tttt///a/aaa
LeCun, Y., Bengio, Y., and Hinton, G. Deep leamning. nature, 521(7553):4364444 2015.
Leiter, C., Zhang, R., Chen, Y., Belouadi, J., Larionov, D., Fresen, V., and Eger, S. Chatgpt: A meta-analysis after 2.5 months. Machine Learning with Applications, 16: 100541, 2024. https:// bttts:///axiv.orgge httgg//bb///000/ 13795.
Lyu, M., Yang, Y., Hong, H., Chen, H., Jin, X., He. Y., Xue, H., Han, J., and Ding, G. One-dimensional adapter to rule them all: Concepts diffusion models and erasing applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7559-7568, 2024. https:// theavf. ctheavf.com/fe http:///oooeett////2///////////
Li, G., Hsu, H., Marculescu, R., et al. Machine unlearning for image-to-image generative models. ICLR, 2024a. https: //tavv///avvv.../aa org/ /ttg//b//////e/ 2ttp://42...0330
Li, S., van de Weijer, J., taihang Hu, Khan, F., Hou, Q.. Wang, Y., and jian Yang. Get what you want, not what you don't: Image content suppression for text-to-image diffusion models. In The Twelfth International Con- ference on Learning Representations, 2024b. https: //openreview net/ /eree/fforuee Eorum? hvVk..
O wttrcinnssioonlAAaaateeeettoooo
All MuL_Concept__Diffussiooooooo @VPR_002 www.pppppp....m
Marler, R. T. and Arora, J. S. The weighted sum method for multi-objective optimization: new in- sights. Structural and multidisciplimarry optimization, 41: 853-862, 2010. https: / /wnkkiinki////// wt.............................. article 0ttpp////....0000000000000000000
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved base- lines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pp- 26296-26306, 2024a. https:
Nguyen, T. T., Huynh, T. T., Nguyen, P. L., Liew, A. W.-
//arxiv.. org/ abs/ 22102210022 03744.
C., Yin, H., and Nguyen, Q. V. H. A survey of ma- chine unlearning. arXiv preprint arXiv:2209 02299, 2022. abs /209 02299.
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tuning. Advances in neural information processing sys- tems, 36, 2024b. https: 08485.
https://arxiv..
2304.
Olah, C., Mordvintsev, A., and Schubert, L. Feature visu- alization. Disuill, 2(11)) 2017.
2017 https://distill
Liu, J., Ram, P., Yao, Y., Liu, G., Liu, Y.. SHARMA, P., Liu, S., et al. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36, wwwp:iiiwoorgragoeeeo
Petsiuk, V. and Saenko, K. Concept arithmetics for circumventing concept inhibition in diffu- sion models. In European Conference on Com- puter Vision, pp. 309-325. Springer, 2025. URL springer. com/ htom//haatee/...../ 1007/978-3-011-753
2024c. https: : / //aaarivvvvv///.. wwwg/brg//....oooooo..e / 2204. .04 944.
Liu, S., Yao, Y., Jia, J., Casper, S., Baracaldo, N., Hase, P.. Yao, Y., Liu, C. Y., Xu, X., Li, H., al. Re- thinking machine unlearning for large language mod- els. Nature Machine Intelligence, pp. 1-14, 2025a. https:/ /tmes://axivvv aaaaiaiiiii.////a///aaa//aaaaava org/ abs/ 2202.422.000 08787.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748-8763, PMLR, 2021. https: / //arxiv. org. abw..a/.//............. 00020.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., et al. Mm- bench: Is your multi-modal model an all-around player? In European Conference on Computer Vision, pp. 216- 233. Springer, 2025b. https:/ /aaxivv..v../ org/ abs/ 2307. 06281.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and
Ommer, B. High-resolution image synthesis with latent
12

---

Targeted Unlearning with Single Layer Unlearning Gradient
diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pp.
Yao, Y., Xu, X., and Liu, Y. Large language model un- learning. ICLR, 2024. https://arxiv..r
10684-10695, 2022. https: / arxiv. org/ abs/ 2112.10755
2310 10683..
Zeiler, M. D. and Fergus, R. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, Septem- ber 6-12, 2014, Proceedings, Part I 13, pp. $18-833, Springer, 2014. https:/ /arxiv..v....////aa//a..////aav. org/ atg/abs//li 1311. 2 2901.
Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. ICLR, 2022. https: //arxiv.org/ otg./ioorrrooooooo abs/ 20..22...2220
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, Laion-400m: Open dataset of clip- filtered 400 million image-text pairs. arXiv preprint 0rXiv:2111.021111 2021. https: httts://arxxv.ork 0ttp/2111.0221111
Zhang, G., Wang, K., Xu, X., Wang, Z. and Shi, H. Forget-me-not: Learning to forget in text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1753-1764, 2024a. https: / thecvf. Cthevf..om/conteeee cttc//////oteeet/////200001 humm://haang_Forret------VLL___
Shaik, T., Tao, X., Xie, H., Li, L., Zhu, X., and Li, Q Exploring the landscape of machine unlearning: A com- prehensive survey and taxonomy. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-21, 2024. doi: 10.1109/TNNLS, STT.TNNNN...........2 //arxiv. wwt./oggg/abaeoeooeoe.oooeeeea/a abs hbtp/ab//23300 /2305 06360.
to_ -uaFrrge_inn_eex-ttt-mmaaeeeee Models_CVPRWWL paper html..
https:
Zhang. J., Huang, J., Jin, S., and Lu, S. Vision-language models for vision tasks: survey. IEEE Transactions on Putlern Analysis and Machine Intelligence, 2024b. https: aaaaaaiiivi...///..a..aaaa .ww.voorgg/ooo. abs / 2304. 006855
Thiel, D. Identifying and eliminating csam in gen- erative ml training data and models. Technical re- port, Technical report, Stanford University, Palo Alto, CA, 2023. URL https:://// /tpp/pull.a stanford / kh752sm9123, 2023.
Zhang, Y., Chen, X., Jia, J., Zhang, Y., Fan, C., Liu, J., Hong, M., Ding, K., and Liu, S. Defensive un- learning with adversarial training for robust concept era- sure in diffusion models. In The Thirty- Anmual Conference Neural Information Processing Systems, 2024c. URL https: id=dkpmfIydrF.
Thudi, A., Deza, G., Chandrasekaran, V., and Papernot, N. Unrolling sgd: Understanding factors influencing ma- chine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pp. 303-319 IEEE, 213.913999...1...
net/ forum?
2022. https: http///xxivv.o/
Zhang, Y., Fan, C., Zhang, Y., Yao, Y., Jia, J., Liu, J., Zhang, G., Liu, G., Rao Kompella, R., Liu, X., and Liu, S. Un- learncanvas: A stylized image dataset to benchmark ma- chine unlearning for diffusion models. NewrlPS, 2024d. https: / /araivriiivv/////i///.aa///aaavv org/ abs /tt///2422.1111
Warnecke, A., Pirch, L., Wressnegger, C., and Rieck, K. Machine unlearning of features and labels. Network and Distributed System Security Symposium (NDSS), 2023. ht
www.aawwwwww...aaaaeaeeeaeoeeeee abs// 2ttp.//21...1111111
Zhang, Y., Jia, J., Chen, X., Chen, A., Zhang, Y., Liu, J., Ding, K., and Liu, S. To generate or not? safety- driven unlearned diffusion models are still easy to gen- erate unsafe images... for now. In European Conference on Computer Vision pp. 385-403. Springer, 2024e. URL https: /ttp://aaxivv.v//v org/ abs wta.b//333 /2100.11002 11868.
Wu, J. and Harandi, M. Scissorhands: Scrub data influ- ence via connection sensitivity in networks. In Proceed- ings of The 18th European Conference on Computer Vi- sion ECCV 2024, 2024. https:/ /atps/////xivv/// org/ abs/ 2401. 06187.
Wu, J., Le, T., Hayat, M., and Harandi, M. Erased- iff: Erasing data influence in diffusion models. arXiv preprint arXiv:2401.0577 2024. https:/ //axivv. org/abs / 240.00779..
Zhang, Z., Wang, F., Li, X., Wu, Z., Tang, X., Liu, H., He, Q., Yin, W., and Wang, S. Catastrophic failure of LLM unlearning via quantization. In The Thirteenth International Conference on Learning Representations, 2025, URL https: wpw.revieneeeeeeepoe/eeoe.ooo..a net /tt.:/.tetttttttte////ee forum? id=1HSeDYamms
Yang, I.., Zhang, 7., Song, Y., Hong, S., Xu, R., Zhao, Y., Zhang, W., Cui, B., and Yang, M.-H. Diffusion models: A comprehensive survey of methods and applications. ACM Computing Surveys, 16(4):1-39, 2023. ht https: 00796.
//a //rxiiv.
13

---

Targeted Unlearning with Single Layer Unlearning Gradient
Appendix A Related work
Machine unlearning (Cao & Yang, 2015; Nguyen et al., 2022) has recently emerged as a critical area of research, driven by privacy concerns and regulatory requirements (gdp, 2016). Existing approaches mainly focus on a single task, like image classification (Liu al., 2024c; neu, 2023; Guo et al., 2020; Goel et al., 2022; Chien et al., 2022; Golatkar et al., 2020b;a; Chundawat et al., 2023; Kurmanji et al., 2023; Jia et al., 2023; Shaik et al., 2024; Fan et al., 2024; Foster et al., 2024), image generation (Li et al., 2024a; Gandikota et al., 2023; Zhang al., 2024a; Gandikota et al., 2024; Kumari et al., 2023; Li et al., 2024b; Lyu et al., 2024; Wu et al., 2024; Wu & Harandi, 2024), and LLMs text generation (Yao et al., 2024; Liu et al., 2025a). In this work, we propose a generic approach that is applicable to a wide range of multi-modal models including CLIP (Radford et al., 2021) for zero-shot image classification, stable diffusion models (Rombach et al., 2022) for text-to-image generation, and lisoon-language models (Liu et al., 2024b) for visual question answering.
For text-to-image diffusion models, particularly Stable Diffusion (SD), the evolution of unlearning approaches reveals increasing sophistication. Early methods such as ESD (Gandikota et al., 2023) and CA (Kumari et al., 2023) focused on modifying the UNet architecture through fine-tuning with negative guidance, but these approaches often resulted in widespread parameter updates across multiple layers. potentially compromising generation fidelity. More recent work has explored more targeted and efficient interventions. UCE (Gandikota et al., 2024) introduced a training-free unified approach using closed-form solutions for simultaneous debiasing, style erasure, and content moderation. FMN (Zhang et al., 2024a) achieved rapid concept removal through attention re-steering loss, redirecting generation from unwanted concepts to pretrained alternatives. SPM (Lyu et al., 2024) proposed an adapter-based approach using Conneptt-Semif BemiPermeeble Membranes" that can be flexibly transferred across different models without re- re-tuning. Other approaches include EDiff (Wu et al., 2024), which formulates unlearning as a constrained optimization problem to preserve model utility, and SEOT (Li et al., 2024b), which focuses on content suppression through text embedding manipulation and inference- sitreenee-time optimization. Despite these advances, existing methods still face challenges in balancing computational efficiency, generalization ability, and preservation of model utility, which our work aims to address through a principled single-layer approach.
Saliency-based methods. Recent advances machine unlearning have seen the emergence of saliency-basee approaches, which aim to identify and modify only the most relevant parameters for concept removal. In image classification, methods like SSD (Foster et al., 2024) employ synaptic importance measures to selectively dampen connections, while SalUn (Fan et al., 2024) takes simple and heuristic threshold- based approach. In text-to-image generation, SalUn (Fan et al., 2024) extend its framework by replacing cross-entropy loss in the unlearning objective to diffusion loss, requiring careful tuning of a gradient threshold for parameter selection. Diff-quickfix (Basu et al., 2024) utilizes causal inference with CLIPSscore (Hessel et al., 2021) as a metric to pinpoint concept-salient model parameters. MACE (Lu et al., 2024) proposes tuning the prompt-related projection matrices the cross- aatsnnnttn blocks in the UNet architecture using LoRA modules (Hu al., 2022). Similarly, CRE (Dong et al., 2024) identifies concept-specific causal denoising time steps in UNet layers and performs representation editing selected layer outputs.
While these saliency- based methods represent the existing efforts in improving the efficiency of unlearning, their scope remains confined to specific tasks, such as image classification or text-to-image generation. Moreover, their parameter modifications often span multiple layers, which limits interpretability and flexibility in practical scenario. In contrast, our approach aims to extend efficient unlearning to foundation models that cover a diverse range of tasks (e.g.. CLIP, Stable Diffusion, and dision-language models). By restricting model edits to a layer-specific scope, our framework introduces modularity to machine unlearning, abstracting the process into distinct layer updates along gradient vectors for tailored unlearning requests.
Appendix B
Algorithm pseudocode
In this section, we present the pseudocode for our method, SLUG, in Algorithm 1, the search process for Pareto-optimal layers in Algorithm 2. and the binary search for the optimal unlearning step size in Algorithm 3. In Appendix B. we discuss the relation between validation size, binary search time cost and unlearning effectiveness.
Our implementation for the corresponding experimental models (i.e., CLIP, Stable Diffusson,, and VLM) and benchmarks (i.e., UnlearnCanvas) has been made publicly available at the anonimized repository: http::/gittub.comCSSiaaaa/sS
14

---

Targeted Unlearning with Single Layer Unlearning Gradient
Algorithm SLUG: Single Layer Unlearning Gradient
Require:
Forget set D and retain set D:
Original model Fo with model weights 0: The set of all layers in the model, as L;
Forget loss function Liorett and retain loss function Cretain: Evaluation metrics forget accuracy and test accuracy TA.
Ensure:
Unlearned model parameters 0
1: Calculate and V tarre (0, D+), Voheetiin (0, D+)
Single gradient calculation
2: for each layer / in L do
3: Importance(1) (T) Vo, Looge (0, 122 1101/1
Calculate layer importance b Calculate layer alignment
4: Alignment(I) = cos V forge (0, D2), V Fretain (0, Dz))
5: end for
6: ParetoOpt(L, Importance, Alignment)
b Pareto optimal algorithm 2 b Set of layers and their performances
Q < 9
8: for each layer in P do
9: Ào= -Importance())/Ul
Initialize step size b Binary search algorithm 3
10: (A,FA, TA)= (
11: Q+QU((L ( TA]}
12: end for
13: FAmin ming FA
D Identify minimum FA
14: Qmin = FA,
FA, TA) E Q FA FAmin] (TA) ("",",FA*,TD TA*) arg
Filter sets with minimum FA D Select set with highest TA
16: return 8: X- Vo. Vo, Ltorgal DE)
Algorithm 2 Pareto Optimal: P ParetoOpt(L, Importance, Alignment)
Require:
The set all layers in the model, as L: Layer importance and gradient alignment all layers
Ensure: The of Pareto optimal layers
1: Initialize P
Set of layers the Pareto front is empty
2: for each layer in L do
3: ParetoDominant
true
4: for each layer /' in L \ 1 do
5: (Importance(? (I') > Importance()) and Alignment(1") < Alignment(I) then
6: ParetoDominant <- false
break
8: end if
9: end for
10: if ParetoDominant then
11: P PU
b> Identified Pareto optimal layer
12: end if
13: end for
14: return P
Return the set of Pareto optimal layers
B.1 Relation of validation size, runtime, and unlearning effectiveness
The runtime of single eval () function, in Algorithm 3, increases linearly with the validation set size. In Table 5, we provide the eval runtime and effectiveness of SLUG versus different validation set sizes, following the setup of Table on CLIP unlcarning. Note that our original choice of 5% validation size already provides good test accuracy on ImageNet, close to that of the original model (which achieves 60. 12%). While increasing the validation size slightly improves utility
15

---

Targeted Unlearning with Single Layer Unlearning Gradient
retention after unlearning. it also increases evaluation time proportionally. Furthermore, smaller validation size (1%
reduces the eval time 3 seconds at the expense of slightly reduced TA.
Table 5: Forget accuracy, test accuracy ImageNet, and runtime of unlearned CLIP models under various validation sizes.
EVAL FUNCTION COST
VALIDATION
NUMBER IMAGES
FA@1 (1)
TA_IN @ (1)
(S)
1%
500
3.05
0.0
58.83
5 % (ORIGINAL)
2500
6.62
0,0
59.96
20 %
10000
24.83
0.0
59.94
50 100 %
25000
59.69
0.0
59.98
50000
119.04
60.04
Algorithm 3 Binary Search for Optimal Step Size: (X*, FA", TA*)
BinarySeaarc(\N..1
Require:
Initial step size λu:
Maximum number of search steps S;
Model parameters 0;
Forget gradient of layer I: G₁ = V Forger (0, D2)
Ensure: Optimal X*, forget accuracy FA, test accuracy TA
Alow < 0
2: Anigh
3: A < λo
4: +- O
5: Initialize P +- 9
D. Performance set
6: while < S do
7: FA, TA- = eval - AG1)
8:
P FA,,,,A TA)}
b. Store results
9:
FF > 0 then
10:
Alow A
Should increase step size unlearn b Should reduce step size to avoid over-unlearning
11:
else
12:
Anigh + A
13:
end if
14:
if Anigh = oc then
15:
x 2A
else
17:
(Atow + \
18: end if
19: + +1
20: end while
21: FAmin -MiNAA FA
D Identify minimum FA
22: Pmin = FA,, TA) FA FAmin)
Filter sets with minimum FA
(X",FA",TA)) TA*) arg Fa (TA)
> Select set with highest TA
return X*, TA*
Select the set with lowest FA which has the highest TA
16

---

Targeted Unlearning with Single Layer Unlearning Gradient
Appendix C More evaluations on unlearning CLIP
C.1
More examples on unlearning identities
Building on the experiment with the target identity "Elon Musk" in Section 3.2, we provide more comprehensive evaluation across broader set of sampled identities. These names, selected from the CelebA dataset, represent diverse range of ethnicities and genders. Our method effectively identifies the key layers associated with each identity, enabling efficient unlearning from the CLIP model. Figure 6 shows that our approach successfully removes the target identities, evidenced by a significant decrease in image-text alignment (cosine similarity). We defer the corresponding pareto-front plots, which indicating the identified layer with SLUG in Section G.
Elon Musk
0.01
Elon Musk
Mark Zuckerberg
Mark Zuckerberg
jeff Bezos
Jeff
Taylor Swift 0.50
0.040.229
Taylor
0.15
Kim Kardashian 0:00 no6 6.09 0.35
0.26
0:28
0.09
Kardashhan 0:c a:44 0.00 0.34 6.26
6.17
0.00 0.12
011 0.24
Kanye West 0.66 0os 6.06 0:30
0.31
0:12
Kanye West
0:33 0.09 0.09
0.10 0:15 0.30
0.33
0.30
0.09
0.15
Barack Obama 0.10 0.14 0.15 0.05 0.05
0.16
0.26
8:12
0.06
0.10
Barack Obama 9:32 6.14 0.15
0.04 0.05
6.15
0.26
0.13
0.00
0.18
Bruce Lee-0 0.04 0.9 0.13 0.02 0.05
0.04
0.08
0.27
0.09
0.08
Bruce Lee-0007 8:05 0.12 0.02 0.05
0.05
0.06
0.26
010
0.07
Fan B Bingbing- 04 0.1 6.05 9.00 0.33
0.01
0.01
6.00
0.32
0112
Fan Bingbing 0.03 9.01 0.02 0.01 0.12 9.02 0.01 0.01 0.04
031
9:13
Lady Gaga- n.m noe 6.10 0.18 0.38 p.ce s
6.11 0.19
0.29
Lady Gaga aidn a.M 0.09 6.17 a.18 on 0.07 0.31 A19
0.29
(a) Original cosine similarity matrix
(b) Cosine similarity matrix after unlcarning
Figure 6: Cosine similarity matrix of image and text pairs before and after unlearning Elon Musk. After unlearning, the image and text pair of Elon Musk are not matched, while other persons are only slightly affected. Here the vision attent ion out pro projection on layer at the Pen resblock (associate with 9, a.ttt... out_orttrrooooo in the pareto front legend) is unleamed, CLIIP model: ViT-8-16
C.2 Joint update for unlearning multiple identities
We extend SLUG unlearn multiple identities simultaneously by computing gradients for each identity's forget set and identifying the most significant layers for joint updates. Following the update scheme in Section 2, initialize step sizes separately for each identity and refine them via binary search based on unlearning effectiveness. Figure 7 demonstrates successful unlearning of (a) Elon Musk, Mark Zuckerberg and (b) Elon Musk, Taylor Swift, showcasing SLUG's ability to handle multiple unlearning tasks efficiently.
Elon Musk-
0.01 0.00 0:05 0:03
Elon Musk 0:03 0.05 0.01 0.10
Mark Zuckerberg 0.00
0.05
8:05
Mark Zuckerberg
jeff Bezos
Jeff Bezos
Taylor Swift
Taylor Swift covift--o.....
Kim Kardashian 0.00 6 00
0.26 6:17
0.08
Kaidashhian 0:03 0:10
6.5 6.14
0.30
617
0.20
Kanye West 6:07 not 6.06
0.30
0:11
6.10
Kanye 0.09 8.6 9.03 6:13 6.25
0.10 0.00
n.00 6.13
Barack Obama 6.07 008 0.12 6.00 0.04
0:38
0.25
0:12
0.04
0.09
Barack Obama 0:33 b:13 0.34
0:31
0.04 0.03
6.14
0.28
0.32
0.09
Bruce 0.09 0.00 0.10
0.00 0.04
0:04
0.08
0.26
0:09
0.07
Lee-b00 8.94 0.11
0.03 0.04
0.05
0.06
0.24
0.09
0.07
Fan Bingbing 0.07 0.04 2:03 0.31 0.01 8.02
0.04
030
Fan Bingbing 0.04 0.02 0.81 0.1 0.02 0.01
0.00
031
8.09
Lady Gaga 9.06 0.1 6.09
0.16
0.18
b.c8 b.d 6.11
0.18
0.29
Lady Gaga 0.04 8:03 0.04 0.00 0.1 0.05 0.
0.00
617
0.24
(a) Cosine similarity matrix after unlearning Flon Musk and Mark Zuckerberg
(b) Cosine similarity matrix after unlearning Elon Musk and Taylor Swift
Figure 7: Cosine similarity matrix of image-text pairs after unlearning multiple identities (see Figure 6a for the original model). (a) Unlcarning Elon Musk and Mark Zuckerberg, (b) Unlcarning Elon Musk and Taylor Swift, In both cases, the selected identities show disrupted alignment, while other identities remain largely unaffected. Based on the Pareto fronts in Figures 22a and 22c, updated the vision layers 9. an. jut_proj for Elon Musk and 11. attn. out_proj for the second identity. Exporiments were conducted on CLIP
17

---

Targeted Unlearning with Single Layer Unlearning Gradient
We further analyze SLUG's performance the number of identities to be unlearned increases. The identified layers updated in parallel to achieve unlearning for N identities. Figure 8 demonstrates effective unlearning across different values of N. The corresponding Pareto-front is detailed in Section G.
Elon Musk 044 0004 6.05
0.00
5,02
0.00
5.21
0.02
0.00
0.02
Elon Musk 2:11 10.2 0.22 0.0 5.00
5.55
0.01
0.05 0.00
Mark Zuckerberg
Mark Zuckerberg
Jeff Bezos 0.13 0.23
9.94
6:0
0.05
9.20
9.9 0,05 0.12
Jeff Bezos 69 0.03 924 022
698
0.05
9.09
0.09
0.05 0:18
Taylor Swift 9606 0.09 6.00
0.01
9.57
033
Taylor Swift 9:53 0.07
0:9
0.07
5.07
0.11
0.15
Kardashlan (6 0:4 6:00
6.26
0.2
0.24
Kardashlan 62
6.26
0.12
0224
Kanye West 5.3 6.6 6.09 0.13 0.30 9.33 9.00
0.00 0.10
Kanya West 9:57 6.1 9 6m 0.30
0:11 0.10
0.00
0.16
Barack Obama 2:z 0.44 6.15 5.66 2.26
015
0.00
0.50
Barack Obama- 0.03
504
0.12
0.05
Bruce 8.0n 0.05 0.32 02 6.05 9.00
026
0.07
Bruce 9.co 4.00
644
0.00
0.26
0:00
0.07
Fan Bingbing pl:a 0. 0.02 0.00 6:32 02 0.01
0.31
Fan Bingbing 9. 0.04 9:31 0.03 681
0.01 0.02
0.00
0.31
Lady Gaga
Lady Gaga
(8)
N
(b) N
Musk
Musk
Mark Zuckerbeng 9. 94 0.01
4.00
5.01
3.54 a.e 0.01 0.00
Mark Zuckerbbrr 3404-0000 os 3 30 00:000000000000 6:01
0.01
3.00
0.07
0.01 0.00
Jeff Decos
0.26 0,02
0.04
6.02
0.02
0.04
6,01 9.05
Jeff Becos 26 0.04 0:22 6.04 592
0.02
5.01
0.04
0,02
loylor Swift 9:05 0.07
0.00
022
5.00
3.07 0:0
0:10
Taylor Swift 9. 00000-000000000000 25 -501
0.00
Klm Kardashian
62 0:00
0.06
0.13
6.25 0.15
9:68 9.10
0.39 23
Klm Ka Kardashian 02 0.00 9:36 0:12 6.25 0.15
9.66
0.11
0.16 23
Kanye West
Kanye West
Barack Obama
dice 0.03 0:23 9:
0.00
2.24
012
0.00
Barack Obama
dece 0.08
82
604
3.24
8:22
0.24 0.00
Bruce Lee pipp -0 0.06 02
3.80
024
Bruce Lee 9:01
6.3
3.09
0.2
Fan Bingbing
6:15 0.00
Bingbing
0. 010
Lady Gaga
5:05 0:03 0.07
0.54 2:37
0.03 5.05 0:10
0.37
0.28
Lady Gaga 5.00 0..2 0.23 0:30
633 5:01 0.07
0.24 022
N
(d) N
Elon Musk 5.05 0.03
0.00
0.00
3.01
0.07
3.22 0.03 0.04 0.00
Elon Musk 3.00 0.000.0000 a 28 0.00-.00 5.05 0.100001--33333 0.07-0.01000 0.00
Mark Zuckerberg
Mark Zuckerberg
Jeff Decos
0.01 0.03
0.22
0.02
0.04
0.01
3.55
0.05
0.02 0.03
Jeff Decos 8.00 22 0 0.01000000000 0.07
0.0a 0.000
0.00
Taylor Switt 9.01-000-4000
27-2
02
Teylor Swift 26-6.000-000
60 -0-------000 6.09 06-000000000 0..55-0.0
Klm Kardashlan
Kardashian
Kanye West 64 0.02 6.03
0.96
6.60
0.2
9:69 0.07
0.00 0.12
Kinye West 62 0.02 91 66 02-002-0000 0.02 0.00 9.00
0.02
0.00
Baradk Obama 9:0e 0.08 8:11
5.00
033
0:50
Barack Obama 0:00 0.12 8:22 0:05 0.02
5.22
0.00
Bruce 9:00 000-0.00 0.07 0.00 6.03 9.69
022
5.07
0.00
Bruce 0: 0.0 a 602 9.69
016
0.00
Fan Bingbing 9::0 0.00 6.05 0.02 5.08 0.04 0.0
0.20
0.00
Fan Bingbing d: 0.09 0 /4 01 605 0.10 5.35 0.00
0.15
0.00
Lady Gage
0:2
Lady Gage
(e) N = 5
(f) N 6
Figure 8: Cosine similarity matrices as we unlearn N identities, where N E {1, 6}. (a) (f) Unlearn Elon Musk, Mark Zuckerberg. Jeff Bezos, Taylor Swift, Kim Kardashian, and Kanye West in joint manner. To unlearn N identities, our method (SLUG) identifies up N layers in the model using the single gradient calculated with the original network weights. The identified layers are then updated in parallel to achieve unlearning of N identities.
18

---

Targeted Unlearning with Single Layer Unlearning Gradient
C33 More CLIP architectures
We performed experiments using an expanded set of model architectures. The results for (ViT-B-16 are discussed above in Figure 6. The results for ViT-I-14, EVA01-9-14] are discussed in Figures 9,10, respectively. These results demonstrate our method offers scalability and effectiveness across a range of model sizes, from 149.62 million parameters (ViT-B-16) to 1.136 billion parameters (EVA01 D0VA01-g-14) This underscores the flexibilityy of our approach to accommodate models of different scales. The Pareto-front of this experiment is included in Section G, where shows the metrics for different layers that our method uses to identify significant layers.
Elon Musk-0.3 0.330 023 0.21
6:12
6:11
0:13 9:13
Elon Musk- 0.01 0.00
Mark Zuckerberg 0:21 034
6:22
Mark Zuckerberg
Jeff Bezos-ni
Jeff Bezos
Taylor Swift 015
0.31
0:25
0.07
Taylor Swift 0.09
0.26
0:13
0:10
0.
0.08
0:18
Kim Kardashian 0:33 014
6.15
0:33
0.32
0:22
0:12
Kardashhan 0:10 612 0:11
0.29
6.19
0.18
0.12
0110
0.24
Kanye West 6:32 813
6.09
0.33
0.28
0.32
0:15
Kanye West 0:09 011 0.10
6.1
0.20
0.33
0.30
0.7
0.14
Barack Obama 0036 u17
6.15
6.05
0.08
0:38
0.31
0:14
0.09
Barack Obama 9:32 0.15 0.34 0.04 0.07
0.34
0.27
0.32
0.44
0.05
Bruce Lee-0:34 0.12
0.16
0.07
0.20
0.32
8:11
0.33
8:12
0.05
Bruce Lee 813 0.13 0.05 0.07
0.10
0.00
0.30
0:11
0.04
Fan Bingbing 9.06 0.14
0.14
0.08
0.29 0:08
0.02
0.17
0:39
0:13
Fan Bingbing 0.00 0.04 0.09 0:83 0.14 8.04
0.05
0.34
0.09
Lady Gaga 9:15 6.3 6.15 0.27 0.28 0:14 0.09
6.09
0:17
0:32
Lady Gaga 0:11 was 0:22 0.14 0.15 9.11 0.07 0.00 0.13
0.28
(a) Original cosine similarity matrix
(h) Cosine similarity matrix after unlcarning
Figure Cosine similarity matrix of image and text pairs before and after unlearning Elon Musk. After unlearning, the image and text pair of Elon Musk are not matched, while other persons are only slightly affected. Here, based on the pareto front in Fig. 23c, we select and update the vision layer 23 ml. p c_ fc for unlearning. CLIP model: ViT-L-14
Elon Musk 0.46 0.29
8.25
0.32
0:33
0:15
1:11
6:14
0:09
0:17
Elen Musk 0.09
0:3
0:16
1616
0:17
0:15
0.18 0:12
Mark Zuckerberg 0.23 0.66
8.22
8.04
0.00
0:05
0.11
6.09
0.03
Mark Zuckerberg
0:05 A29 0:58
10-6
07
0.00 -0 00.0000.00 07-0070-.0
Jeff Bezos 0.26 0.32
6.45
0.09
0.07
0:33
0.14
0.15
00.00 0.10
Jeff Bezos 0.18 0:25
0.37
0:03
0.00
0.04
0.06 0.00 -o de 0.04
Taylor 0wit 0.14 0.05 6.00
0.42
0:31
0.27
5:10
0.40
0.01
0:15
Taylor Swift 0.09 0.04
0.04
0.34
0:05
0.12
0.07
0.06 04 0.00
Kim Kardashian- 0.17 0.08
0.10
0.15
0.39
0:24
8.08
0.11
0.00
0.27
Kim Kardashian 0.10 0.03
0.03
0.07
0.30
8:17
0.04
0.06
0.03
0.19
Kanye West 0.15 0:08
0.12
0.18
0.21
0.46
0.16
0.14
0:03
0:15
Kanye West 0.13 0.04
0:37
0:32
0:15
0.40
0.12
0.30
0.02
0.14
Barack Obama 0:37 618
9.16
0.05 0:00
0:15
0:41
15
0707
Barack Obama- 0:12 914 6:31 01 0.
0.33
0.36
01
0.000
Bruce Lee 1:1- 0.09 0.13
0.07 0.04
0.08
Bruce Lee-0.09 8.05
0.04
0.07
Fan Bingbing 0.04 6.02 0.09
Fan Bingbing 0.02 9.00 6.03 0.02
Lady Gogs
Lady Gaga
(a) Original cosine similarity matrix
(b) Cosine similarity matrix after unlearning
Figure 10: Cosine similarity matrix of image and text pairs before and after unlearning Elon Musk. After unlearning, the image and text pair of Elon Musk not matched, while other persons are only affected. Here, based on the pareto front in Fig. 23f. select and update the language layer 11. attn.
for unlearning. CLIP model: EVA01-G-14.
C.4 Unlearning object concepts in CLIP
In addition to unlearning identities from CLIP, we also sample 7 classes (Basketball, Beach, Castle, Revolver, Rifle, School bus, Sunglasses) from ImageNet to evaluate the unlearning performance of our method on object concepts. For this experiment, we use 10k ImageNet validation images and sample images associated with target classes to create forget sets and compute gradients unlearning different classes from the CLIP model. For evaluation, we use zero-shot accuracy reduction as the metric of effective unlearning target classes from the CLIP. The results, presented in Table. 6, show the CLIP zero-shot accuracy evaluations for both the forgetting of sampled classes and the retention of other ImageNet classes after unlearning. Our findings indicate that our method effectively reduces the CLIP zero-shot accuracy for the targeted classes to 0.0%, while the accuracy for remaining classes remains high, experiencing only minimal degradation (ranging from 0.03% to 2.03%) compared to the original pre- trained model, which indicates that the model's 's original functions are highly preserved after our unlearning.
19

---

Targeted Unlearning with Single Layer Unlearning Gradient
Table Unlearning performance our method on common object concepts. FA@ FA represents the top-1 top-5 forget accuracy (%) of each forget class (i.e.. zero-shot classification accuracy of unlearned class). TA and TA@5 represents the top-1 and top-5 accuracy (%) of all classes of ImageNet except the corresponding Forget class. Each row shows the forget class accuracy and average accuracy over all classes of ImageNet before and after unlearning class. Our method can reduce the forget accuracy of Forget classes 0.0% while keeping the accuracy of the remaining classes close original model (within 0.06 2. 2.03% difference). CL.IP model: Vi --11-B-222 32. @ and
for the original model remains almost the same rows; therefore, we list once in the table.
ORIGINAL
UNLEARNED
FORGET CLASS
FA@1
FA@5
TA@1
TA@3
FA@1 4
FA
TA@1
TA@5 f
BASKETBALL
100.0
100.0
0.0
0.0
59.18
84.48
BEACH
54.55
72.73
0.0
0.0
59.54
84.78
CASTLE
87.50
100.0
0.0
0.0
58.13
83.87
REVOLVER
100.0
100.0
60.16
85.52
0.0
59.94
85.43
RIFLE
42.86
57.14
0.0
0.0
60.08
85.49
SCHOOL BUS
76.92
100.0
0.0
0.0
59.50
89.18
SUNGLASSES
44.44
55.56
0.0
0.0
60.13
85.23
C.5 Impact of unlearning on semantically similar objects
Our method is designed to address precisely this concern by balancing unlearning effectiveness with utility preservation. We identify the most critical layer to update using layer importance and gradient alignment metrics that minimize impact on retained information while maximizing unlearning of targeted concepts. This approach allows for precise targeted removal while preserving general model performance on both related and unrelated tasks. Our experimental results demonstrate this balance. When unlearning specific identities in CLIP, our approach achieves state-of-the-art results while maintaining high accuracy on the CelebA dataset (containing many semantically similar identities) with only minimal degradation compared the original model (58.32% vs. 61.38% top-1 accuracy). This significantly outperforms other methods like SSD, which drops to 35.96% accuracy. SLUG shows minimal impact related concepts and image quality across all experiments, demonstrating its effectiveness at avoiding over-unlearning of semantically similar objects.
To further quantify the impact of unlearning on semantically similar objects, we sampled the "Basketball," "Revolver," and "School Bus" rows from Table and conducted additional zero-shot classification evaluations on the unlearned CLIP models. The semantically related classes were selected based on the ImageNet hierarchy and the top-5 most likely ranks in the logits across all targeted instances. The results in Table indicate that the zero-shot accuracy of unlearned CLIP on both semantically related and top-5 most-likely classes remains high, comparable its performance the full ImageNet zero-shot evaluation. This further demonstrates the strong utility retention of our approach.
Table 7: Additional evaluation of unlearned models classes that are semantically close to the forget class, and top-5 most-likely classes from the classitication logit vectors. SLUG unlearned models maintain high test accuracy classes that closely related the target.
FORGET CLASS
FORGET Acc (J)
TEST ACC IMAGENET (t)
TEST Acc SEMANTICALLY RELATED CLASSES (1)
TEST Acc TOP-5 MOST-LIKELY SELECTED FROM LOGIT (f)
BASKETBALL
0.0
59.18
73.63
58.33
REVOLVER
0.0
59.94
43.59
38.89
SCHOOL BUS
0.0
59.50
86.21
78.85
20

---

Targeted Unlearning with Single Layer Unlearning Gradient
C.6 Linearity of unlearning trajectory of different layers
In addition to the layers presented in Figure 2 (c) and (d), we show in Figure 11 that different layers show similar unlearning behaviors if we update them along their respective gradient direction (computed once for the original model). Nevertheless, the utility performance may vary depending on the selected layer; thus, it is important select the best layer from the Pareto set for the overall best performance.
Acggs
Acci3
Colesá Acogy
Curbb Acci't
imagekert Anng)
imageltet Arrg1
Accuracy (50)
99
Accuncy
10
%
0.020 0.025
0.200
0,073
5,000
0.125
3,200
0.75
0,200
0.020
5,025
0.000
0,073
0.100
0:325
0,250
0.15
0.209
(a) Vision layer vi visuul... proj
(b) Language layer text projectioin
Ferpet
Forpet
Celeba Accde thit Acag1
Geob4 Acool
Imagehet Acag1
Accuracy (S)
Accuracy (%)
63
20
0.00
0.25
6.50
0.75
1.00
1.25
1,50
1.75
2.00
6.5
1.0
25
3.0
(c) Vision layer 11. wwlppppp..........mmmmmm
(d) Language layer at.attn. in_proj
Figure 11: More examples of unlearning different layers. Correspond to Figure 2. The performance changes monotonically with the step
size A.
Appendix D
More evaluations on unlearning Stable Diffusion
To demonstrate the performance and practical utility of method, further conduct robustness study of SLUG in Sec- tion D. providing additional qualitative evaluation on scenarios in Section D.3 (e.g., more identities, copyright characters, novel concepts, artistic styles). Additionally, we provide experimental details for evaluating SLUG on UnlearnCanvas in Section D.5.
D.1 Blackbox adversarial and quantization robustness
Recent research has exposed flaws in the robustness of foundation models unlearning. Notably, the Concept Arithmetic Attack (CRA) (Petsiuk & Saenko, 2025) demonstrates an optimization-free method where attackers exploit concept arithmetic properties of SD to reconstruct "unlearned" content through composite prompts. Zhang et al. (2025) show that loading post-unlearning LLM weights at lower-bit precision (higher quantization) significantly weakens the unlearning effect observed at higher-bit precision. These findings highlight how simple manipulations can undermine unlearning. raising serious concerns about the reliability of existing methods.
To verify effectiveness of SLUG, we applied both the Concept Arithmetic Attack and quantization to SD models unlearned by SLUG. Figure 12 shows that SLUG remains robust to concept arithmetic, as it modifies the encoder component, disrupting concept interoperability and effectively influencing downstream text-guided image generation. In Figure 13, we test an unlearned model trained in 16- floating point (fp16) by loading it in 8-bit unsigned integer (Uint8). The results demonstrate that SLUG maintains its unlearning effect despite quantization.
21

---

Targeted Unlearning with Single Layer Unlearning Gradient
Source prompt
prompt
Subtract prompt
Add prompt
Subtract prompt
portrait
Musk portrait in Van Gogh style"
"Van Gogh sty
'Elon Musk standing the Mars'
"Standing the Mars"
U
Original
Unlearned
"Elon Musk"
Figure 12: SLUG robust to Concept Arithmetic Attacks. The first row illustrates the concept arithmetic property of Stable Diffusion, where distinct concept groups are added or subtracted from the source prompt. Each column presents the generated image using the corresponding arithmetic prompt. The model unlearned by SLUG fails to generate the target ID consistently different arithmetized prompts.
"A portrait Elon Musk fp16
"A portral of Mark Zuckerberg" fp16
"A turtic the ocean"
1p
Original
iil
Unlearned
"Elon Musk
Figure 13: SLUG is robust model weight quantization. The first SD model generates images consistently across different quantization levels. Post-unlearning models with SLUG exhibit negligible differences, consistently failing to generate the targeted ID while preserving utility for other concepts.
sanity check, showing that the original pretrained
D.2 Whitebox adversarial robustness.
In Table 8, we utilize the latest UnlearnDiffAlk (Zhang et al., 2024c) and P4D (Chin et al., 2024). Specifically, we selected the "Nudity" and "Church" from Table 2 and Table 4 of Zhang et al. (2024e) to provide a brief adversarial evaluation of SLUG.
Following the same setup as Zhang al. (2024e), we applied SLUG SDv1.4 unlearn "Nudity" concept and "Church" object, then attack the SLUG-unlearee SDv1.4 .4 using two attack methods: UnleamDiffAtk and P4D that optimized 142 and 50 adversarial prompts for "Nudity" and "Church," respectively.
Having robustness against whitebox attacks is challenging without corresponding adversarial design in the unlearning process. The results indicate that SLUG (like other unlearning methods) is not immune to whitebox adversarial attacks, yet SLUG demonstrates effectiveness on unlearning NSFW concepts and objects that are studied in existing literature.
D.3 More unlearning scenarios
More celebrity names. Beyond unlearning "Elon Musk" from Stable Diffusion, which is presented in Figure 4, here we also provide additional qualitative evaluations on unlearning other celebrity names (Taylor Swift, Je Bezos]
22

---

Targeted Unlearning with Single Layer Unlearning Gradient
Table 8: Evaluation against adversarial attacks. Lower ASR indicates better adversarial robustness. Row-No Attack shows the original performance on unlearning tasks.
CONCEPT/OBJECT UNLEARN METHOD
NUDITY
CHURCH
OURS
ESD
FMN
SLD
OURS
ESD
FMN
No ATTACK P4D UNLEARNDIFFATK
16.90
20.42
88.03
33.10
4
52
ATTACKS: (ASR %)
76.76
69.71
97.89
77.46
46
98
90.32
76.05
97.89
82.39
80
60
96
'A portrait of Taylor Swift'
TTaylor Swift singlng'
'A portrait of Elon Musk'
'A portrait Mark Zuckerberg'
portrait of Jeff Bezos'
sea turtle the ocean"
Original
Unlearned "Taylor Swift"
"A portrait of Bezns
JJff Bezos car"
'A portrait of Musk'
'A portrail Mark Zuckerberg"
portrait of Taylor
sea furtle ocean"
Original SD
ASBON
Unlearned "Jeff Bezos"
Figure 14: Qualitative evaluation unlcarning celebrity names Taylor Swift and Jeff Bezos from the Stable Diffusion.
with our method in Figure 14.
Unlearning concepts and copyright contents. In addition to identity removal for privacy protection, we address copyright concerns that increasingly challenge generative models. For unlearning copyrighted contents from Stable Diffusion models, we generate 500 images using unlearning targets as prompts, and use them as the forget set. The retain set is single shard of LAION-400M dataset, same as for CLIP unlearning.
We successfully apply our method to remove sepyright-rroteceee content, specifically targeting well-known characters such Marvel's "Iron Man" and Walt Disney's "Mickey Mouse." Figure 15 illustrates that technique precisely unlearns the targeted concepts, effectively disabling the generation of images associated with these copyrighted entities while preserving the ability of the model to produce images of other concepts. These results demonstrate the use of SLUG in protecting intellectual property from generative Al.
NSFW concepts. Our experiments in Table 8 follow the setup of UnleammiiffAttack (Zhang et al., 2024e), which considers more practical unlearning scenarios on NSFW (not-safe-for-wone concepts. We applied SLUG to unlearn the "Nudity" concept in SDv1.4. The results in No ATTACK row of Table demonstrate that SLUG is applicable to NSFW concepts previously studied in related work.
Novel concepts. One of the intriguing properties of the Stable Diffusion is its ability to generalize image generation novel concepts that are infrequently or never observed in the real world. In this experiment, we explore the unlcarning of a unique concept, "Avocado chair" from Stable Diffusion. We first generate 500 image using SD with the prompt "An avocado chair" to create the forget set, and use the same retain set as other experiments, which a single shard of LAION-400M dataset. In Figure 16, we show that our method successfully unlearn the concept "Avocado chair" from SD, resulting the model's
23

---

Targeted Unlearning with Single Layer Unlearning Gradient
'Iron
'Iron and Spiderman"
piece fron'
'Mickey Mouse'
portrail
sAra turtle the occan
Original SD
Unlearned
"Mickey Mouse"
'Mickey Mouse and Friends'
'A real mouse'
*Iron man"
'A portrail of Elon Musk"
'A sea turtle in the occan'
Original SD
Unlearmed "Mickey Monse"
Figure 15: Qualitative evaluation of unlearning copyrighted characters "Iron Man" and "Mickey Mouse" from Stable Diffusion. The first shows images from the original pretrained model, while the second and third rows display outputs from the unlearned model using the prompts above cach column. Our method effeetively removes copyrighted concepts while preserving overall image generation quality.
avocade chair" *An avecado"
portrait Mark Zuekerberg'
'A clair
AAn orange"
the occan'
Original
H
Unlearned "avocado chair
Figure 16: Qualitative evaluation on unlearning novel concept "Avocado chair" from the SD.
inability to generate images corresponding to this specific concept.
is noteworthy that the model's 's capability to generate images related to the constituent atomic concepts (namely "Avocado" and "Chair") also compromised. We hypothesize that this occurs due to the model's treatment of novel concepts as compositions of atomic concepts. For example, the concept "Avocado chair" is interpreted by the model as "Avocado" plus
24

---

Targeted Unlearning with Single Layer Unlearning Gradient
"Chair." Consequently, when novel concept unlearned, the associated atomic concepts are inadvertently affected well. This highlights challenge in the model's approach to handling the interoperability of novel and atomic concepts.
Artistic styles and object. In the experiment of evaluating SLUG performance on UnlearnCanvas benchmark discussed in Section. 3.3, we use 400 images that are associated with each style, as the forget set for unlearning style, and 1200 images that are associated with each object concept as the forget set for unlearning object, all images are from the benchmark dataset. We use a single shard of LAION-400M dataset as the retain set.
For qualitative evaluation of this experiment, we provide visual examples of unlearning artistic styles: (Pop Art, Crayon, Sketch, Van Gogh} and object: dog that sampled from UnlearnCanvas, in Figure 17, 18 and 19. These results further show the effectiveness of SLUG in unlearning a broad spectrum of concepts ranging from concrete (e.g,, celebritty name, intellectual property figure, and object) to abstract (e.g., novel concept and artistic style).
D.4 Details on Gap Ratio evaluation
In this section, we provide additional analysis of the Gap Ratio (GR) evaluation in Table 2, by breaking it down in terms of effectiveness and efficiency aspects in Table 9. We follow the same setup as Table 2. where we represent metrics for each method as a 9-dimennsiona GR vector, quantifying its gap from the "Best (hypothetical reference) method". To provide a more comprehensive geometric characterization the performance gap, we report the ε and & distance of the GR vectors, both normalized by the vector length (in this case { equivalent average). the ALL METRIC column of Table 9.
We also report summary metric using only effectiveness efficiency metrics. In the EFFECTIVENESS column, we compute the distances using only the first 7 entries in the GR-vectors (corresponding to UA, IRA, CRA for both style and object, and FID) and their counterparts in the BEST (hypothetical) method of Table 2. In the EFFICIECCY column, we compute the distance using only the last entries in the GR-vectors (Time and the "sum of memory and storage") and their counterparts in the Best (hypothetical) method.
In summary, SLUG offers the best performance across the combined Effectiveness and Efficiency metrics Table While SLUG provides the best results in terms of Efficiency, its Effectiveness is among the top-performing methods. We also acknowledge that different choices of norms and weighting of individual metries provide us different results for the summary metric.
Table 9: Gap Ratio summary of different unlearning methods over metrics of Table 2. Low value means smaller performance gap from the best performing method, the best performance is highlighted. SLUG offers competitive performance in effectiveness metrics.
EFFECTIVENESS
EFFICIENCY
ALL METRICS
METHOD
4 NORM
82 NORM
4 NORM
82 NORM
f NORM
f2 NORM
ESD
0.20
0.11
81.04
78.55
18.17
17.46
FMN
0.47
0.24
6.51
4.72
1.81
1.07
UCE
0.64
0.37
5.50
5.08
1.72
1.17
CA
0.17
0.09
10.37
9.03
2.44
2.01
SALUN
0.06
0.03
12.32
9.11
2.78
2.03
SEOT
0.24
0.13
1.22
0.88
0.46
0,22
SPM
0.16
0.07
380.71
380.27
84.73
84.50
EDIFF
0.20
0.11
23.45
19.97
5.37
4.44
SHS
0.37
0.20
19.50
15.78
4.62
3,51
SLUG (OURS)
0.19
0.08
0.00
0.00
0.15
0.06
D.5 Experiment details UnlearnCanvas
Models. UnlearnCanvas targets unlearning styles and objects from an SDv1.5 model fine-tuned to generate 20 differ- ent objects in 60 distinct styles. The benchmark provides pre-trained SDv1.5 models for evaluation in Diffusers and CompVis implementations. In our experiment, correspondly, focus the CLIP text encoder used in SDv1.5 Di ffusers implementation: openai / wewpraaiilliiiitttiie large -attce14 4 from HuggingFace.
Computational time, memory, and storage. The gradient computational time and memory usage of SLUG depends on several factors: computing resource, batch size, and size of the forget set. Note that while the details the evaluation of efficiency metrics are not well defined in the original UnlearnCavvas, in Table. we are reporting the best performance of SLUG can achieve on our computing resource NVIDIA A100 40GB. Specifically, the batch size is set to 1 for recording
25

---

Targeted Unlearning with Single Layer Unlearning Gradient
the memory usage of SLUG, and to 16 for recording its computational time. This batch size of 16, is consistent with the sizes used in our other experiments. For SLUG storage consumption, method only requires storing the gradient values of few layers the Pareto front, the actual storage consumption is 43 MB (0.043 GB), which by approximation is 0.0 GB in the benchmark scale.
'A [object] in Pop Art style
*A [object] in Crayon style
'A [object] in Sketch style
[object] Van Gogh style
[object] = horse, dog, cat
Original UnlearnCanvas SD
Unlearned Pop Art style by SLUG
Figure 17: Visual examples SLUG performance on UnleamCanvas, Row 3: outputs from original UnlearnCanvas Stable Diffusion
(SD) using column captions as prompts. Row 4 - 6: outputs from UnlearnCanvas SD unlearned Pop Art style. Outputs corresponding to the unlearned style are highlighted by the red bounding box
26

---

Targeted Unlearning with Single Layer Unlearning Gradient
'A [object] in Pop Art style
'A [object] in Crayon style
[object] Sketch style
[object] in Van Gogh style
[object] horse, dog, cat
Unlearned Cravon style SLUG
Unlearned Sketch style by SLUG
Figure 18: Visual examples of SLUG performance on Unleam Row - 3: outputs from UnlearnCanval SD unlearned Crayon
style. Row 4 - 6: outputs from UnlcamCanvas SD unleareed Sketch style. Outputs corresponding to the unlearned style are highlighted by the red bounding box
27

---

Targeted Unlearning with Single Layer Unlearning Gradient
'A [object] in Pop Art style
'A [object] in Crayon style
[object] Sketch style
[object in Van Gogh style
[object] horse, dog, cat
Unlearned Van Gogh style by SLUG
Unlearned Dog object by SLUG
Figure 19: Visual examples of SLUG performance on InlearnCanvar Row 1 3: outputs from UnlearnCavvas SD unlearned Van Gogh style. Row 4 - 6: outputs from UnlearnCanvas SD unlearned dog object. Outputs corresponding to the unleareed style/object are highlighted by the red bounding box
28

---

Targeted Unlearning with Single Layer Unlearning Gradient
Appendix E More evaluations on unlearning VLM
In this section, we present additional qualitative examples of post-unlearning VLM with SLUG. Figure 20 showcases further responses from the unlearned LLaVA VA-11.557 model for the target identity "Elon Musk", beyond examples in Figure the main text.
STOP
Input Image
User Prompt
What's the name of the person in this image?
What's the content of the image?
Describe this image in detail.
The image features a stop sign on street with driving by background. walking on the sidewalk,
image features cats lying pink couch, both sleeping peacefully.
The this image is Elon Musk.
The person this image is Mark Zuckerberg.
Answer Original
people lion
One cat located on the left side of the couch, while the other
The image
Answer Unlearned
The person in this image Michael Jackson,
The person in this image Mark Zuckerberg.
The image features a stop sign on street corner, with red building in the background. There also statue lion nearby.
lying it, slceping peacefully. One cat is located on the left side the couch, while the other is the right side. The
pink, and
Figure 20: Qualitative evaluation on unlearning name "Taylor Swift" from LLaVA 1.5. While "Taylor Swift" mapped to "woman" after the unlearning, the other female celchrity identification remain unaffected, Besides, model's robustness against style distribution shift is also preserved.
In addition to results presented in Section 3.4, we include more qualitative examples on unlearning a different identity "Taylor Swift" from LLaVA-11. 1.5 in Figure 21. We demonstrate that our method can anonymize celebrity names from the pretrained laiguaee models, and simultancously preserve the model's 's ability on image understanding, reasoning and distribution shift robustness on work, cartoon style images.
DEEP LEARNING
Input
Image
User Prompt
What's the name of the person this image?
What's the content of the image?
What unusual obout this image?
The this image is Taylor Swift.
The this image is Queen Elizabeth
in this image is Marilyn
The image is cartoon depiction of baseball featuring various characters elements
The unusual aspect of this image is that the man is reading book while sitting swimming pool. not common engoging
Answer Original
sport
The person in this image is
The person this image is
person this image is Marilyn
The image cartoon or comic strip featuring baseball game, with various characters and animals participating in game...
not common see someone reading in such an environment, as it typically associated with swimming other water-related activities...
Answer Unlearned
Elizabeth II.
Monroe.
Qualitative evaluation unlcarning name "Taylor Swift" from LLaVA 1.5. While "Taylor Swift" is mapped to "woman" after the unlcarning the other female celebrity identification remain unaffected. Besides, model's robustness against stylc distribution shift also preserved.
Figure 21:
29

---

Targeted Unlearning with Single Layer Unlearning Gradient
Appendix F Summary of model sizes
Our empirical results provide strong evidence supporting the claim of updating a single critical layer sufficient/scalable for larger models. We conducted experiments across model scales summarized in Table 10, consistently demonstrating effective unlearning through single-layer updates (see Appendix C.3, Figure 9 and 10).
Table 10: Summary of total and unlearning manipulated parameter size of experimental models. Our approach is scalable the largest model size up 7B parameters.
MODELS
ToTAL PARAMS
SINGLE LAYER PARAMS
CLIP VrT-B-32
151.28 M
1.05 M
CLIP ViT-L,114
427.62 M
2.36 M
CLIP EVA01-G-14
1.14 B
2.35 M
SDV1.5, v2.1
983 M
2.36 M
LLAVA 1,5-7B
7.06B
4.19 M
Appendix G Pareto-fronts of all experiments
In this section, we present the complete set of Pareto-fromt plots for all the CLIP unlearning experiments discussed in Sections C.1, C.2, and C.3. These plots serve as a reference for our method, showing the identified layers for unlearning in each experiment.
Sectioin C.1 More examples on unlearning identities: Figure 22 illustrates the Pareto-front plots that are used identify important layers selected by method for unlearning different identities.
Sectioin C.2 Joint update for unlearning multiple identities:
Figure 22 presents details identifying layers associated with different identities and updating them to achieve unlearning of multiple identities at once.
Sectioin C.3 More CLIP architectures: Figure 23 shows the metrics for different layers that our method uses to identify significant layers for unlearning different CLIP architectures.
30

---

Targeted Unlearning with Single Layer Unlearning Gradient
10-1
10a
Importance of Layers
10-2
10-2
Importance of
10-3
x
10
10-4
10-1
visual.pro
11 attntttt. out, proj
text projection 0.attn.out
x
7.mip.c_fe fc
2.mip.c_fe
10-
10-3
10-1
10-1
107
10-3
10-2
10-1
10a
Gradient Alignment
Gradient Alignment ent
(a) Vision layer Pareto Mark Zuckerberg
Language layer Pareto
Mark Zuckerberg
10-2
Importance of Layers
Importance of Layers
10-2
X
10-1
10-3
x
X
text_ projection
0.tttn.ou proj Lattn.sut pro) S.mip.c.ff ft
10-4
xx
x
pisual.proj
11 aatat.oott aattn.out_pr)
X
x
4. 4.mip.c_fe fc
7.mip.c_pro projj
x*
6.attm.in_proj _poj
10a2
10-3
10-2
10-1
10P
10-3
10-2
10-2
109
Gradient Alignment
Gradient Alignment
(c) Vision layer Pareto Jeff Bezos
(d) Language layer Pareto Jeff Bezos
10-1
Importance Layers
Importance of Layers
10-2
10-2
10-3
10-3
x
visualproi 11 m.m.p...t
10
11. atin pprj
text projection 0.atthouu pro)
out pro]
10-²
10-1
10
10-4
10a
10
10-1
10
10a
Gradient Alignment
Gradient Alignment
(e) Vision layer Pareto Taylor Swift
(f Language layer Pareto Taylor Swift
31

---

Targeted Unlearning with Single Layer Unlearning Gradient
10a1
10-2
Importance of Layers
10-2
Importance of
10"3
x
10"1
x
x
10-4
10-1
pro
11mlpp.c_
K
11 a.t.ttniinpp.. pro)
10-3
7.attn.out.
x
10-1
projection 7.mip.c_fe fe
9.mip.cffe
10-1
10-2
10-4
10a
10
10-3
10-2
10-2
107
Gradient Alignment
Gradient Alignment
gg) Vision layer Pareto Kim Kardashian
(h) Language layer Pareto Kim Kardashian
10-2
Importance of Layers
Importance of Layers
10-2
10
10-2
10-1
text projection
10
Visual.proj
0.attn.out_pre proj 0.attn.in pro)
11 attn.in proj
S.mip.c_f fc
A.mip.c_r
10
0.mip.c_fe fc
x
10-1
10 attn proj
x
S.attn.in_np _proj
10->
10-2
10-1
10a
10->
10-2
10-1
10P
Gradient Alignment
Gradient Alignment
(i) Vision layer Pareto Kanye West
Language layer Pareto Kanye West
10-1
10-1
Importance of Layers
10--
Importance
10
10-2
prou
m.mpp.c_fc
10-4
11 attn praj
attn.in
10
text projection 0.attn.out
10 cattn.out_poe pro]
11 attn. out pro) proj
10-1
S.atthout proj
proj
x
10
8 B.mip.s_ft
X
x
10-1
10-2
10-2
10
10-1
10-2
10-2
10e
Gradient Alignment
Gradient Alignment
(k) Vision layer Pareto Barack Obama
(I) Language layer Pareto Barack Obama
32

---

Targeted Unlearning with Single Layer Unlearning Gradient
10a1
10-
Importance of Layers
10-2
Importance of Layers
10-7-
10-3
10-4
text projectiio 0.attn.out 9.attn.out pro)
visualproj
attn proj
10-3
11 atta.in_prr pro)
10ah
10 attn. out pro)
Xx
10
10
10
10
10
Gradient Alignment
Gradient Alignment
(m) Vision layer Pareto Bruce Lee
(n) Language layer Pareto
Bruce Lee
10-2
Importance of Layers
Importance of Layers
10-1
10-1
10-3
visual.proj 7.
10-1
text projection 0.tttn.out, proj
9.attn.ou out pro)
0.attn.in, pro)
10-1
fo
x
5.attn.out proj
x
10-3
10-2
10-1
10a
10-4
10-3
10-2
10-1
10a
Gradient Alignment
Gradient Alignment
(0) Vision layer Pareto Fan Bingbing
(p) Language layer Pareto Fan Bingbing
Importance Layers
Importance of Layers
10
10-2
10-1
text projection
X
0.attn.oU_
9.attn.out pro)
X
0.tttn.n_ proj
10-4
S.attnout proj
X
x0
visualproj 11 an.ttjiin__pra
Lattn.out
L.ttniin_pee
4.ttnnin_pre proj
Xx
10-3
10-2
10-4
109
10-3
10-2
10-4
10a
Gradient Alignment
Gradient Alignment
(q) Vision layer Pareto Lady Gaga
(r) Language layer Pareto Lady Gaga
Figure 22: Scatter plots of layers for unlearning more identities, same setting as Figure 2. CLIP model ViT-B-32. Figures (a) (1) shows the importance and gradient alignment of different vision model and language model layers as we unlearn different identities.
33

---

Targeted Unlearning with Single Layer Unlearning Gradient
10-1
Importance of Layers
Importance of Layers
10-2
10-7-
xx
10-3
10-3
pisualproj
11.att.out__pe
10-4
S.
10-1
9.attn.out
text, p.pppcecttoon 0.attnouut pro)
7.tttn.out, pro)
a.ttt.out,pp proj
%
x
attn proj
10.5
s.mip.cfte
10-3
0.attn.in pro)
10
10a
10
10a
Gradient Alignment
Gradient Alignment ent
(a) Vision layer Pareto Vi T-B-11
(b) Language layer Pareto -ViT Fii-B--T
10-1
10-2
10-3
10.2
Importance of Layers
10-3
Importance
10-1
visualroj
23 mlpp f
x
10-1
x
mip.c pro)
22 attnttt. out
10
out
text projection 0.attn.out pro)
20 attn. in pro)
18 ci
x
10-3
attnn...u prb]
10-3
10-2
10-1
100
10-3
10-2
10-1
10*
Gradient Alignment
Gradient Alignment
(c) Vision layer Pareto ViT-I-14
(d) Language layer Pareto -iT---14
10-1
10-2
10-2
Importance of Layers
Importance
10-3
x
x
*
10-4
Visual.tunkk
10-4
X
39 attn.qqi givv
pro)
x
X
10-1
36 attn proj
11 attn. pro_
X
proj
proj
Xx
10
10
10-2
10-1
10
10-3
10-2
10-1
10a
Gradient Alignment
Gradient Alignment
(e) Vision layer Pareto EVA01 -4-14
(f) Language layer Pareto EVA01-9-14
Figure 23: More CLIP models, in addition to See 3.2. Unlearning name Elon Musk from different CLIP models built in: (ViT-B-16, ViT-L-14, and EVA01-g-i4;
34


Reviews:
[{"review_id": "3hObRyVAEJ", "reviewer": "Reviewer_Ceck", "summary": "This paper introduces SLUG, an efficient targeted unlearning method that aims to remove specific unwanted information from large-scale models with minimal computational overhead. Unlike conventional unlearning approaches that iteratively update parameters across the entire model, SLUG identifies a single critical layer using metrics of layer importance and gradient alignment. By performing only one-time gradient computations on the forget and retain losses, and updating that selected layer along a linear trajectory (with a carefully chosen step size via binary search), SLUG achieves effective unlearning while preserving the model’s utility. The method is evaluated on a diverse set of models including CLIP, Stable Diffusion, and vision-language models, and experiments on the UnlearnCanvas benchmark demonstrate that SLUG delivers comparable unlearning performance to existing methods but with significantly lower computational cost.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for summarizing the strengths of SLUG (computational efficiency and innovative single-layer update approach). We address your concerns below:\n\n## Domain Generalization\nWhile our paper primarily focuses on CLIP, Stable Diffusion, and VLMs, the principles behind SLUG are agnostic to domains and applicable to LLMs. Nevertheless, a detailed analysis of SLUG for LLMs would require a non-trivial time and effort (due to differences in task and dataset). We believe our comprehensive evaluation across multiple multi-modal foundation models already demonstrates the versatility and effectiveness of our method across different domains \n\n\n## Evaluation on NSFW Benchmark\nThe adversarial attacks evaluation in [2] employed NSFW and object unlearning scenarios. We applied SLUG to unlearn SDv1.4 on the \"Nudity\" concept and \"Church\" object, using 142 and 50 author-provided prompt-generated images, respectively, as forget sets.\nWe present the results in Table R0, where we have the lowest (best) forget accuracy, which confirms that SLUG is generalizable to unlearning concepts and objects previously studied in prior work. We will be happy to expand this evaluation for the complete dataset in the revision.\n\nTable R0 - Additional evaluation on unlearning Nudity (NSFW) and Church (Object) scenarios.\n| Concept/Object |  | Nudity (↓)|  |  | Church (↓)|  |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Unlearn Method | SLUG | ESD | FMN | SLUG | ESD | FMN |\n| Forget Accuracy (%) | 16.90 | 20.42 | 88.03 | 4 | 14 | 52 |\n\n## Ablation Studies\nIn Table R1, we adopt the weight selection scheme in SalUn (i.e., selecting a mask across the entire network). Additionally, we conduct an ablation study on our introduced metrics: layer importance and gradient alignment, and randomly selected layer, to provide further insights.\n\nTable R1 follows the setup in Table 1, focusing on unlearning CLIP and evaluating forget accuracy on 5 target identities, and zero-shot test accuracy on ImageNet and CelebA.\n- \"SalUn\": Selecting model weights distributed across the network (instead of a single layer) using importance-only (SalUn-like selection) performs worse than SLUG, with higher  FA and lower TA. \n- \"Importance\": Selecting a single layer using only gradient importance results in FA reduction to 0 but TA_IN and TA_CA also reduced, due to some layers with high importance for unlearning exhibit high gradient conflict with the retain set.\n- \"Alignment\": Selecting a single layer based on gradient alignment only results in increased FA@5 of 5.56% as well as reduction in TA. \n- ”Random”: Selecting a single layer at random performs poorly, with an increase in FA and a decrease in TA compared to SLUG. \n\nThese results further demonstrate that both importance and alignment are required for achieving an optimal trade-off between effective unlearning and utility retention.\n\nTable R1: Additional studies on different parameter selection approaches for CLIP unlearning.\n| Parameter selection | FA@1 (↓)  | FA@5 (↓)  | TA_IN@1 (↑)  | TA_CA@1 (↑) |\n|:---:|:---:|:---:|:---:|:---:|\n| “SalUn” (distributed weights, importance only)  | 4.44 | 11.33 | 48.23 | 37.38 |\n| Single layer importance only | 0.0 | 0.0 | 21.04 | 42.00 |\n| Single layer alignment only | 0.0 | 5.56 | 31.08 | 54.16 |\n| Single layer at random | 0.0 | 6.91 | 33.38 | 52.90 |\n| SLUG | 0.00 | 0.00  | 59.96  | 58.32 |\n\n## Adversarial Robustness Evaluation\nOur discussions and experiments in Sec. D.1 was limited to robustness against (prompt-based) concept arithmetic attacks and quantization. Guaranteeing robustness against whitebox attacks is challenging, and we did not make any such claim in our paper. Nevertheless, we appreciate the suggestion to incorporate adversarial evaluations. In Table R2, we utilize the latest UnlearnDiffAtk [2] and P4D [1]. Specifically, we selected the \"Nudity\" and \"Church\" categories from Table 2 and Table 4 in [2] to provide a brief adversarial evaluation of SLUG.\n\nFollowing the same setup as [2], we applied SLUG on SDv1.4 to unlearn \"Nudity\" concept and \"Church\" object, then attacked the SLUG-unlearned SDv1.4 using two attack methods: UnlearnDiffAtk and P4D that optimized 142 and 50 adversarial prompts for \"Nudity\" and \"Church,\" respectively.\n\nThe results indicate that SLUG (like other unlearning methods) **is not immune to whitebox adversarial attacks**.\n\nTable R2: Evaluation against adversarial attacks. Lower ASR (%) indicates better adversarial robustness. Row-“No Attack” shows the original performance on unlearning tasks.\n| Concept/Object |  | Nudity (↓)|  |  | Church (↓)|  |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Attack method | SLUG | ESD | FMN | SLUG | ESD | FMN |\n| No Attack | 16.90 | 20.42 | 88.03 | 4 | 14 | 52 |\n| P4D | 76.76 | 69.71 | 97.89 | 46 | 56 | 98 |\n| UnlearnDiffAtk | 90.32 | 76.05 | 97.89 | 80 | 60 | 96 |\n\n\n[1] Prompting4Debugging (ICML 24’)\n\n[2] To generate or not? (ECCV 24’)", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes, it mainly follows previous benchmark settings.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Related to the trustworthy machine learning", "essential_references_not_discussed": "Please refer to weakness part.", "other_strengths_and_weaknesses": "Strengths: \n- SLUG’s one-time gradient computation and single-layer update significantly reduce the computational burden compared to iterative unlearning methods. This efficiency is particularly valuable when dealing with large-scale models.\n\n- The idea of focusing on a single layer, identified via layer importance and gradient alignment metrics, for unlearning is both innovative and practical. Concentrating updates in a minimal subset of the model helps limit unintended side effects on overall model performance.\n\nWeakness:\n\n- Limited Domain Generalization: Although SLUG is evaluated on vision-language and diffusion models, it remains unclear how well the single-layer update approach generalizes to other domains, such as text-only large language models. Additionally, since previous work has used established benchmarks (e.g., for NSFW removal tasks on diffusion models), it would be beneficial to include performance comparisons on those benchmarks to better contextualize the results.\n\n- Lack of Ablation Studies: The paper does not include ablation studies comparing different weight identification methods in combination with the single gradient direction unlearning approach. For instance, if an alternative method like SalUn were applied with a single gradient update, resulting in a comparable time complexity of $O(2·N_f+ N_r)$, it would be informative to assess whether a finer-grained weight localization can yield improved performance.\n\n- Limited Robustness Evaluation: While the authors present one robustness evaluation on diffusion models, incorporating additional evaluation methods (e.g., CCE [1] and UnlearnDiffAtk [2]) would provide a more comprehensive assessment of the method’s resilience under various adversarial or challenging conditions.\n\n[1] Pham M, Marshall K O, Cohen N, et al. Circumventing concept erasure methods for text-to-image generative models[J]. arXiv preprint arXiv:2308.01508, 2023.\n\n[2] Zhang Y, Jia J, Chen X, et al. To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 385-403.", "other_comments_or_suggestions": "Please refer weakness part.", "questions_for_authors": "Please refer weakness part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aHb8XFQbp1", "reviewer": "Reviewer_oTLd", "summary": "The authors propose a novel (saliency-based) unlearning method called SLUG that identifies a single layer in the model and performs only a single update step in this layer to minimize negative side-effects on the model’s utility. Compared to related works such as SalUn [1], they assign values to each layer based on how important they are for the unlearning and how aligned their gradients are with the utility loss, instead of computing a mask over all parameters based on the unlearning importance alone. Then, SLUG figures out a pareto-optimal subset of layers that are the most important but least aligned with the retain loss to minimize interferences with the utility of the model. Within this set, they search for the best single layer and the best step size (learning rate) to take for the single update using a proposed binary search approach. They instantiate their method for CLIP and evaluate it with CLIP-based models like Stable Diffusion (SD), vision-language models (VLMs), or CLIP itself on a variety of different unlearning scenarios.\n\n[1] Fan, Chongyu, et al. \"Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.\" arXiv preprint arXiv:2310.12508 (2023).\n\n\n## Final recommendation and justification\n\nI appreciate the effort made to address my concerns and open questions. My original review was recommending a Weak Reject for this work due to concerns regarding some of the claims, specifically regarding SOTA results on UnlearnCanvas benchmark and SLUG’s adversarial robustness, some clarity issues with their method and contribution, and the lack of a discussion around the downsides of a single-layer approach. In the rebuttal, these concerns were mostly addressed through additional explanations and additional experiments, which is why I raised my rating from Weak Reject to Weak Accept, in the expectation that my points get addressed and that the missing discussion around the potential shortcomings of a minimal method like SLUG will be added to the paper. I share the other points made by other reviewers that the trade-off between the efficiency and the unlearning effectiveness, and the influence of the SLUG method on this is still a bit unclear, especially since the unlearning results of SLUG are consistently worse than, e.g., the ones achieved with SalUn (see Table R7). For that reason, I will stick to my rating of Weak Accept and not raise it further.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful comments. We address the concerns as following:\n\n## SOTA Results on UnlearnCanvas\nThank you for your suggestion of reporting a unified summarizing metric (e.g., harmonic-like mean). \n\nTo unify the scores of different metrics, we use the mean Gap Ratios (GP), defined as: $\\frac{|\\mathrm{current method performance} - \\mathrm{best method performance}|}{\\mathrm{best method performance}}$. For example, according to Table 2, the FID GP for SLUG is calculated as: $= \\frac{75.97-54.21}{54.21}=0.4$. \n\n\nWe compute the GP across effectiveness, FID, time, and memory+storage in Table 2 and report the arithmetic mean of these ratios across these metrics in the following table. In Table R3, SLUG demonstrates the lowest GP among all methods, further supporting our claim that SLUG achieves a well-balanced trade-off between effectiveness and efficiency.\n\nTable R3: Gap ratio (%) averages of different unlearning methods over UnlearnCanvas metrics. Low value means smaller performance gap between the best performing method.\n| Method | SLUG | ESD | FMN | UCE | CA | SalUn | SEOT | SPM | EDiff | SHS |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| Gap ratio mean (↓) | 13 | 4063 | 370 | 342 | 524 | 622 | 71 | 19042 | 1191 | 1013 |\n\n## Adversarial Robustness\nAddressed in response to **Reviewer Ceck, Table R2**\n\n## Methods and Evaluation Criteria\n\n### Identity Erasure Details\nFA columns in Table 1 show the average forget accuracy for unlearning a single identity at a time (score averaged over 5 identities from Figure 3). TA columns report test accuracy on 100 celebrities from CelebA sampled as our validation set. For each identity unlearning experiment, we used forget sets of 1,000-6,000 image-text pairs associated with that identity. We will be happy to further clarify this in the text. \n\n### Object Unlearning\nOur object unlearning is not restricted to specific ImageNet classes. Instead, we leverage the latest UnlearnCanvas, covering 20 object classes for a unified and comprehensive comparison. Our Table 4 focus on CLIP unlearning, differs from Table 2 in SalUn on SD unlearning.\n\n### VLM Metrics\nWe use forget accuracy (FA) to measure successful unlearning of target identities while using standard VLM benchmarks (MME, GQA, MMBench) to evaluate utility preservation. Table 3 shows we reduced FA to an average of 2.8% while maintaining high benchmark scores within 2-4% of original performance. This comprehensive evaluation demonstrates both effective unlearning and utility preservation.\n\n## Theoretical Claims\nIn this paper, we do not make any theoretical claims\n\n## Evaluation Function and Costs\nThe eval function measures both forget accuracy (FA) and test accuracy (TA) on **validation sets** distinct from the forget/retain training sets. A small set of validation images is a sufficient indicator of the unlearning step-size search process. \n- For CLIP, we used 5% of the test size \n- For SD, we used 10 test-time generated images not in the (forget) training set\n- For VLM, we used 10-image subset per identity for validation\n\nOur approach isn't \"multi-step training\" but rather a principled search for the optimal operating point along a single gradient direction, which is fundamentally different from iterative optimization methods. \n\n\n### Computation Requirements\nThe binary search evaluation typically requires less than 5 forward passes through the model (no backward passes) to identify the optimal step size. These costs are included in our reported computational efficiency metrics in Table 2 and are significantly lower than iterative methods requiring multiple forward-backward passes.\n\n## Relation to Broader Scientific Literature\nWe have acknowledged the connections to SalUn and SSD, and discussed them in Section 2 when introducing our approach.\n\n## Methodological Ablations\n\n### Importance-only, Random Selection\nPlease refer to response to **Reviewer Ceck, Table R1**\n\n### Multiple Layers vs Single Layer\nFollowing Table 1 setup, Table R4 presents updating multiple layers on the Pareto front and all model layers. While this slightly improved unlearning effectiveness (~2-3% better FA), it significantly increased complexity and utility degradation risk.\nFigure 8 shows that our approach effectively unlearns multiple concepts by targeting multiple layers in parallel, demonstrating the model’s ability to support modular unlearning without requiring multiple layers for a single concept.\n\nTable R4: Additional studies on updating multiple layers for CLIP unlearning. \n| Method | FA@1 (↓)  | FA@5 (↓)  | TA_IN@1 (↑)  | TA_CA@1 (↑) |\n|---|---|---|---|---|\n| All Pareto | 0.00  | 0.00 | 59.92 | 51.64 |\n| All Layers | 0.00  | 0.00 | 59.70 | 53.74 |\n| SLUG | 0.00 | 0.00  | 59.96  | 58.32 |", "claims_and_evidence": "**SOTA results on UnlearnCanvas (?)**: The claim of the authors to achieve SOTA results and the best trade-off in Table 2 (UnlearnCanvas) appears to be not supported enough through quantitative evaluations. Appreciating that SLUG as a minimal method is performing well, this claim should be supported by some summarizing metric or visualization, e.g., some type of harmonic mean. UnlearnCanvas is a multi-facetted benchmark, where not a single method stands out as the best in all aspects.\n\n**Robustness**: They claim that SLUG is robust to key flaws exposed in recent un-learning studies, i.e. Concept Arithmetic Attacks and quantization, but only show anecdotal evidence for that. They also do not even mention other key flaws like inversion-based attacks [5] for unlearning concepts from Stable Diffusion. It is fine that a minimally invasive method like SLUG is not robust in all regards, but the claims here should then be made more carefully.\n\n[5] Pham, Minh, et al. \"Circumventing concept erasure methods for text-to-image generative models.\" arXiv preprint arXiv:2308.01508 (2023).", "methods_and_evaluation_criteria": "Mostly yes, the authors evaluate their SLUG method in various scenarios with different types of CLIP-based models and thereby show the generality of their method. Unfortunately, they sometimes fail to paint a convincing picture with their evaluation, especially w.r.t. clarity around the choices of FA and TA in the different settings. Appreciating the breadth of their experiments to show a wide applicability of their method and understanding that in-depth evaluation in each of them might be out-of-scope, more clarity around what exactly has been done in each of these experiments will strengthen this work.\n\nFor example, according to Section 3.1, their identity erasure setup involved unlearning 100 different celebrities, which appears to be their major experimental setup but they do not describe the details of that in their results: Does Table 1 show the averages of unlearning one identity at a time or unlearning all of them in parallel?\n\nIn the object unlearning case, it is unclear why they chose their own custom ImageNet classes as opposed to using the ones that prior work already used to evaluate their methods. A quantitative comparison might have been simplified by that. For example, Table 2 in the SalUn [1] paper could have been a great starting point for such a comparison.\n\nIn the VLM identity unlearning case, a more in-domain metric akin to the employed forget accuracy is missing which measures the same for the other non-erased celebrities. The general VLM benchmarks are a good addition but might miss more in-domain forgetting related to other celebrities and thereby underestimate the loss in utility especially with the lack of comparison to any related method. It is also unclear what definition of the test accuracy is used when applying SLUG in this setting.", "theoretical_claims": "The sentence “Small alignment between unlearn and retain gradients would prevent unlearning updates from negatively affecting the retain set” in Sec. 2.2 needs more justification. Intuitively, if both of these gradients are exactly the same and thus perfectly aligned, there would not be any negative interference between the two, right? Another paragraph here or a visualization could help to transport the intuition of the authors to the reader.", "experimental_designs_or_analyses": "Yes, I reviewed the experimental designs and already provided some concerns in Methods And Evaluation Criteria. Beyond that, I am generally confused by the role of their `eval` function in their algorithm that is used to obtain FA (forget accuracy) and TA (test accuracy) to identify the best layer and step size. They claim a single-layer single-step approach but invoking an inner evaluation method can come at a significant computation cost, especially for application scenarios like the VLM or SD. As a reader, it was not clear to me if the final test evaluation was used for that or if the method relied on a validation set to find those step sizes. If no validation set was used for this “inner” evaluation, the numerical results presented in this paper could be overfitted to the specific test set in all of those cases. Moreover, it is unclear if the invocation of the `eval` function is priced into the complexity estimate of your algorithm? As far as I understood, the evaluation within the algorithm does, e.g. for the identity unlearning with the VLM, compute the FA using the definition from lines 423-427, which sounds relatively compute-heavy. The SLUG method effectively evaluates different points in the parameter space and then takes the one that performs the best overall. This is also a form of multi-step training from my POV. A discussion around these questions in the main paper could help to clarify the matter for readers.", "relation_to_broader_scientific_literature": "Their proposed SLUG algorithm is a saliency-based machine unlearning method that is less restricted to a specific type of model (generality) than, e.g.,  ESD [3] or MACE [4] which are restricted to only diffusion models. SLUG is specifically instantiated for CLIP. which makes it more flexibly applicable to related models as the authors also try to transport in their paper by testing it on various CLIP-based models such as Stable Diffusion or a VLM. Their method is in its nature close to SalUn, which also tries to identify a portion of the parameters that are the most important for the update, but SalUn [1] only takes the “forget loss” into account to find the parameters that are the most important for the unlearning without any “alignment” considerations to incorporate the retain gradients into this selection. Instead of using a binary mask (like SalUn), SLUG goes further in a sense and identifies a single layer to restrict the unlearning to.\n\n[1] Fan, Chongyu, et al. \"Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.\" arXiv preprint arXiv:2310.12508 (2023).\n\n[2] Foster, Jack, Stefan Schoepf, and Alexandra Brintrup. \"Fast machine unlearning without retraining through selective synaptic dampening.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 38. No. 11. 2024.\n\n[3] Gandikota, Rohit, et al. \"Erasing concepts from diffusion models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[4] Lu, Shilin, et al. \"Mace: Mass concept erasure in diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "essential_references_not_discussed": "Generally, the related work section appears to be too brief, especially w.r.t. SalUn [1] and SDD [2], which are the closest methods to the proposed SLUG. To understand the relations and differences better, it would help the reader to draw more explicit connections in the main paper.\n\n[1] Fan, Chongyu, et al. \"Salun: Empowering machine unlearning via gradient-based weight saliency in both image classification and generation.\" arXiv preprint arXiv:2310.12508 (2023).\n\n[2] Foster, Jack, Stefan Schoepf, and Alexandra Brintrup. \"Fast machine unlearning without retraining through selective synaptic dampening.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 38. No. 11. 2024.", "other_strengths_and_weaknesses": "The paper presents a minimally invasive yet effective single-layer approach to unlearning that is novel in two different ways: they target only a single layer and also only perform a single step, which effectively linearizes the unlearning update. They specialize their method by instantiating it with CLIP-specific retain and forget losses, which still keeps it generally applicable to all CLIP-based models like Stable Diffusion or VLMs like LLaVa-1.5. This sets it apart from many related works in the area of unlearning in multimodal models, which often specialize their method on a particular model type. Moreover, the teaser figure is well-designed and effective in bringing across the main concepts of their method. Their perspective on finding room for unlearning in the “null space” of the retain loss provides a strong intuition before they introduce their method. \n\nMy main concerns with this work are the lack of depth in experimental evaluations (see Methods And Evaluation Criteria) as well as a lack of methodological ablations to analyze their approach and underline their design choices (e.g., what happens if only the importance but not the alignment measure is used to identify the single layer; what if SLUG would identify more than a single layer to strengthen its unlearning accuracy; what if the pareto-optimal pre-selection would be replaced by a random selection?).\n\nBesides that, some aspects of the evaluations were unclear to me, most importantly the exact choice of FA and TA in the different scenarios and the compute requirements of performing those evaluations as part of their method. This appears crucial to me for a fair comparison to related methods. The existence or non-existence of a validation set for the inner evaluations is also something that should be clarified. A brief discussion around the limitations of SLUG, including the potential shortcomings as a single-layer approach, is also missing.", "other_comments_or_suggestions": "1. **Typo**: “Prato-optimal layers” → “Pareto-optimal layers” (in Line 312)\n\n1. **Line plot colors**: The choice of colors in Fig. 2 is slightly confusing because they do not refer to the same things in the different subplots; consider changing that to make it easier for the reader to understand your helpful visualizations.\n\n1. **Confusion matrix figures**: The “confusion matrix” visualizations (e.g., Fig. 3) help to understand but are also hard to read for the close-to-zero numbers and the darker colors. The small images of the celebrities on the x-axis are a playful addition but lower the quality of the plot when being obviously stretched along one of the dimensions to fit the 1:1 aspect ratio. Consider cropping a square out of the images when recreating your plot.\n\n1. **Naming of hyperparameters**: Using `K` for the maximum number of steps in your binary search algorithm (Algo. 3) and then using `K` as well to denote the number of epochs in Table 1 can confuse the reader. Consider changing one of them.\n\n1. **Single Layer Selection**: As a reader it was not fully clear in the end how the final single layer is selected. Looking at your pseudocode in Algorithm 1 helps but I think readers would appreciate a few more sentences about this part of your method in the main paper. \n\n1. **GAFT**: You mention GAFT as a two-stage combination of Gradient Ascent (GA) + Finetuning (FT) but never really elaborate how you go about that, i.e. when having GAFT with 2 iterations, does that mean 2 GA iterations followed by 2 FT iterations? This should be made clear.", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a16sKHje5z", "reviewer": "Reviewer_C1HE", "summary": "This paper proposes Single Layer Unlearning Gradient (SLUG), a technique for targeted unlearning in large-scale multimodal models by updating only a single critical layer using one gradient computation. The authors demonstrate its efficiency for CLIP, Stable Diffusion, and Vision-Language Models, claiming robust removal of targeted concepts while preserving broader model utility. Their approach hinges on carefully identifying the most relevant layer and then modifying it along a single gradient direction.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful comments and constructive feedback on our paper. We address each of the concerns below:\n## Layer Selection Assumption\nThe reviewer questions whether updating a single critical layer is sufficient/scalable for larger models. Our empirical results provide strong evidence supporting this claim. We conducted experiments across model scales summarized in following table, consistently demonstrating effective unlearning through single-layer updates (Sec C.3, Figure 9-10).\n| Models | Total Params | Single layer Params |\n|:---:|:---:|:---:|\n| CLIP ViT-B-32 | 151.28 M | 1.05 M |\n| CLIP ViT-L-14 | 427.62 M | 2.36 M |\n| CLIP EVA01-g-14 | 1.14 B | 2.35 M |\n| SDv1.5, v2.1 | 983 M | 2.36 M |\n| LLaVA 1.5-7B | 7.06B | 4.19 M |\n\nRegarding the concern about memorization across different layers, our method specifically addresses this through the layer identification procedure (Sec 2.2), which evaluates all layers to find the one most critical for the target concept. Our approach fundamentally differs from LLM memorization studies because:\n- We focus on multi-modal foundation models where concepts are represented differently than in pure language models\n- Our layer importance metric and gradient alignment analysis (Equations 5-7) precisely identify which layer contains the most salient information about the target concept\n- Our experiments across different architectures (ViT-B/L, EVA), different task models (SD, VLM) demonstrate the generalizability of this approach\n\n## Single Gradient Direction\nThe reviewer questions our claim about single gradient direction sufficiency. We provide clear evidence in Figure 2 (b,e) and Figure 11, showing that performance changes monotonically with step size along a single gradient direction. Critically, Figure 2 (c,f) demonstrates that iterative methods (GA and GAFT) provide no advantage over our single-gradient approach while requiring substantially more computation.\nRegarding UnlearnCanvas results, our claim of \"best trade-off\" refers precisely to the balance between unlearning effectiveness, utility preservation, and computational efficiency. While some methods achieve marginally higher UA scores in Table 2, they require orders of magnitude more computation (e.g., SPM: 29700s vs. SLUG: 39s) and memory (e.g., SalUn: 30.8GB vs. SLUG: 3.61GB). Importantly, SLUG shows no significant underperformance in any metric (no red highlights), achieving consistently strong performance across all dimensions.\n## Motivation for Object Unlearning\nWhile unlearning common objects may not seem directly impactful, it serves multiple purposes:\n- Research methodology: Provides a controlled setting to precisely evaluate unlearning techniques using clear metrics..\n- Model customization: Enables tailoring foundation models to exclude specific objects (e.g., restricting non-medical outputs in medical imaging).\n- Data governance: Lays the groundwork for extending unlearning to privacy-sensitive content.\n- Safety & responsibility: Demonstrates technical feasibility in controlling model outputs, a key step toward responsible AI.\n\n## UnlearnCanvas Dataset Details\nThe UnlearnCanvas benchmark considers 20 different object classes, each represented in 60 distinct styles, resulting in a total of 20×60=1200 combinations. For example, consider the combination of \"dog\" and \"Van Gogh.\" The benchmark uses its fine-tuned SDv1.5 model to generate 20 images with the prompt \"A dog in Van Gogh style.\" \nTo construct a complete dataset for the \"dog\" class, the benchmark fixes \"dog\" and iterates through the remaining 59 styles, generating a total of 20×60=1200 images. Similarly, to create a complete dataset for the \"Van Gogh\" style, it fixes \"Van Gogh\" and iterates through the remaining 19 objects, resulting in 20×20=400 images.\nThe benchmark focuses on unlearning either a single class or a single style at a time, ultimately producing 20+60=80 unlearned models for each unlearning method. When unlearning a single style, we use the 400 images available in that style as the forget set and the remaining dataset as the retain set. Similarly, when unlearning a single object, we use the 1200 images available of that object as the forget set and the rest of the dataset as the retain set. \nThe reported metric values in Table 2 are averaged over all 80 unlearned models. We would further clarify the setup of this benchmark in the revision.\n## Adversarial Robustness Evaluation\nWe thank the reviewer for highlighting the latest adversarial robustness studies on LLMs [1,2]. While our primary focus is on vision-language foundation models, we acknowledge that [1,2] vulnerabilities may also apply to these models and are worth further research. \n\nRegarding adversarial prompting, we conducted a brief evaluation during the rebuttal phase, please refer to response to **Reviewer Ceck, Table R2**.", "claims_and_evidence": "The authors' claims are partially supported by their empirical results but require more thorough and concrete substantiation. Please see the details in the Other Strengths and Weaknesses.", "methods_and_evaluation_criteria": "Yes, please see the details in the Other Strengths and Weaknesses.", "theoretical_claims": "This paper does not have a theoretical claim.", "experimental_designs_or_analyses": "I have reviewed their experimental designs and analyses. Please see the details in the Other Strengths and Weaknesses.", "relation_to_broader_scientific_literature": "This work tackles important and crucial problems in unlearning, such as addressing the trade-off by finding a single layer and computing the single gradient direction. The results show a great balance among the three factors, which could potentially help the unlearning community fo future research.", "essential_references_not_discussed": "The authors lack some papers related to their work. Please see the details in Other Strengths And Weaknesses.", "other_strengths_and_weaknesses": "Current unlearning techniques often require extensive model-wide updates and still fail to robustly forget targeted information. The proposed approach addresses these limitations by targeting a single layer for updates, providing a more computationally efficient way to remove unwanted target information. The authors emphasize a balance between efficient forgetting and retaining the model’s utility, which is critical for real-world, large-scale applications. The experiments show the great trade-off between efficiency and effectiveness.\nHowever, the following questions need to be further justified.\n\n1. Layer Selection Assumption\nThe paper claims that updating one critical layer is sufficient to remove unwanted content. However, this claim may not scale to large diffusion models or other foundation models. Could the authors provide any empirical or theoretical evidence of this claim? For example, even in language models, memorization occurs across different layers [1].\n\n2. Single Gradient Direction\nWhile the authors highlight that single gradient computation reduces the computational overhead, there is little evidence or discussion explaining why a single direction suffices or how closely this approximates a multi-step update in practice. For example, if this is the case, this approach should be able to achieve the best forgetting performance in Table 2. The authors show strong efficiency in UnlearnCanvas experiments, but the forgetting scores do not outperform baselines. For example, methods like SalUn or ESD show higher unlearning quality for style removal. Does the best trade-off here refer to considering the efficiency dimension? If so, this claim is quite debatable. While prior works have tried to address the trade-off between forgetting quality and model utility, under which justification we can compromise the effectiveness of unlearning for efficiency remains unclear to me. How do the authors assign weight between the two to determine the best trade-off?\n\n3. Although prior work unlearns object concepts in diffusion models, could the authors clarify the real-world motivation for object unlearning beyond the fact that related studies do so? In particular, for diffusion models, I understand that there is copyrighted or harmful content learned during the training stage, which needs to be addressed properly for responsibility. However, in which perspective do we have to consider object unlearning for large-scale foundation models? For instance, concepts like cats and dogs represent public information readily available in open datasets. Should we consider this from a privacy perspective (e.g., individuals requesting erasure of their data) or a safety perspective? \n\n4. The discussion of UnlearnCanvas could be improved with more specifics on how the forget and retain sets were formed. For instance, the authors mention one example format of the forget data: \"A [object name] in [style name] style\". In this case, do we focus on erasing both information? If so, what does the retained set look like? \nSpecifically, if a target style like \"Da Vinci\" is removed (one-side removal), it would help to see how other styles remain valid, such as \"Monet\". \n\n5. I think it would be very beneficial if the authors could provide quantitative results of the robustness dimension with respect to relearning attacks and adversarial prompting [2,3]. This would better substantiate the claim that updating a single layer is sufficiently robust and that the forgotten knowledge remains inaccessible under adversarial conditions.\n\n[1] Demystifying Verbatim Memorization in Large Language Models\n\n[2] JOGGING THE MEMORY OF UNLEARNED LLMS THROUGH TARGETED RELEARNING ATTACKS\n\n[3] DO UNLEARNING METHODS REMOVE INFORMATION FROM LANGUAGE MODEL WEIGHTS?", "other_comments_or_suggestions": "Please see the details in Other Strengths And Weaknesses.", "questions_for_authors": "Please see the details in Other Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "07Jvz4Wcjt", "reviewer": "Reviewer_BJg7", "summary": "This paper addresses the computational challenge and performance degradation often associated with machine unlearning methods, proposing an efficient technique called Single Layer Unlearning Gradient (SLUG). Instead of extensive updates across the entire model, SLUG strategically updates only a single critical layer, identified through metrics such as layer importance and gradient alignment. Experiments conducted on CLIP, Stable Diffusion, and vision-language models demonstrate SLUG’s effectiveness in unlearning both concrete elements (like specific identities or objects) and abstract concepts (such as artistic styles). Evaluations on the UnlearnCanvas benchmark indicate that SLUG achieves comparable unlearning accuracy to state-of-the-art methods, while significantly reducing computational cost. Thus, SLUG offers a practical, targeted unlearning approach that maintains model utility with minimal overhead.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful feedback. Below, we address each point raised:\n\n## Impact of unlearning on semantically similar objects\nOur method, SLUG, is designed to address precisely this concern by balancing unlearning effectiveness with utility preservation. We identify the most critical layer to update using layer importance and gradient alignment metrics that minimize impact on retained information while maximizing unlearning of targeted concepts. This approach allows for precise targeted removal while preserving general model performance on both related and unrelated tasks.\nOur experimental results demonstrate this balance. When unlearning specific identities in CLIP, our approach achieves state-of-the-art results while maintaining high accuracy on the CelebA dataset (containing many semantically similar identities) with only minimal degradation compared to the original model (58.32% vs. 61.38% top-1 accuracy). This significantly outperforms other methods like SSD, which drops to 35.96% accuracy.\n\nFor precision, we show minimal impact on related concepts and image quality across all our experiments, demonstrating SLUG's effectiveness at avoiding over-unlearning of semantically similar objects. \n\nBelow, we sampled the \"Basketball,\" \"Revolver,\" and \"School Bus\" rows from Table 4 and conducted additional zero-shot classification evaluations on these unlearned CLIP models. The semantically related classes were selected based on the [ImageNet hierarchy](https://observablehq.com/@mbostock/imagenet-hierarchy) and the top-5 most likely ranks in the logits across all targeted instances.\n\n\nThe results indicate that the zero-shot accuracy of unlearned CLIP on both semantically related and top-5 most-likely classes remains high, comparable to its performance on the full ImageNet zero-shot evaluation. This further demonstrates the strong utility retention of our approach.\n\nTable R5: Additional evaluation of unlearned models on classes that are semantically close to the forget class, and top-5 most-likely classes from the classification logit vectors. SLUG unlearned models maintain high test accuracy over classes that are closely related to the target.\n| Forget  class | FA (↓)  | TA_IN(↑)  | TA_Semantic related (↑) | TA_Top-5 most-likely (↑) |\n|:---:|:---:|:---:|:---:|:---:|\n| Basketball | 0.0 | 59.18 | 73.63 | 58.33 |\n| Revolver | 0.0 | 59.94 | 43.59 | 38.89 |\n| School Bus | 0.0 | 59.50 | 86.21 | 78.85 |\n\n## Text-to-image retrieval evaluations for CLIP\nWe agree that text-to-image retrieval evaluation would strengthen our results. In our CLIP unlearning experiments, we evaluate zero-shot classification accuracy on both unlearned content (Forget Accuracy) and retained content (using ImageNet and CelebA). This provides a strong indication of CLIP's alignment between textual and visual representations.\n\nFollowing the standard zero-shot paradigm, predictions are based on the highest cosine similarity between image and text embeddings. Our comprehensive cosine similarity matrices (Figures 3, 6-10) effectively demonstrate the disruption of image-text alignment for unlearned content while preserving alignment for retained content, which directly addresses text-to-image retrieval performance.\n\n## Complex prompt scenarios with multiple objects\nOur experiments focused on SLUG’s computational efficiency, precise unlearning, and minimal side effects.\n\nFor Stable Diffusion, Figure 4 shows that after unlearning, our model successfully replaces an identity (e.g., \"Elon Musk\") with electronic circuits while preserving the \"Mars\" setting—unlike other methods that degrade image quality or affect non-targeted concepts.\n\nAppendix Figure 15 (“Iron Man and Spiderman”) further demonstrates retention of non-targeted objects post-unlearning. Figures 17-19 show that even after style unlearning, models still generate the correct objects in other styles.\n\nWe appreciate the suggestion to test complex prompts with multiple objects alongside unlearned ones, and a more rigorous, quantifiable evaluation worth exploring further.\n\n## Style unlearning setup\nOur UnlearnCanvas experiment comprehensively evaluates style unlearning on an SDv1.5 model fine-tuned for 20 objects in 60 styles.\n\nFigures 17-19 show that after unlearning a style (e.g., \"Pop Art\"), our method prevents its generation while preserving other styles. In Figure 18, while the unlearned model still produces black-and-white images for \"Sketch style,\" classifiers identify them as \"Ink Art,\" not the original Sketch style.\n\nWe further assess our method using UnlearnCanvas metrics: UA (Unlearn Accuracy) for unlearning effectiveness, and IRA/CRA for utility retention.\n\n\n## Computational efficiency comparison\nAs addressed above, we provide a direct comparison of computational efficiency in Table 2 for the UnlearnCanvas benchmark.\n\n## Applying unlearning to multiple layers\nPlease refer to response to **Reviewer oTLd, Table R4**", "claims_and_evidence": "Satisfactory.", "methods_and_evaluation_criteria": "This paper provides a detailed description of the proposed approach, along with extensive experimental results on multiple benchmark datasets, such as UnlearnCanvas, to validate the effectiveness of the method. Additionally, a comprehensive ablation analysis is presented.", "theoretical_claims": "This paper primarily presents empirical results to support its claims; however, it does not provide any theoretical proofs.", "experimental_designs_or_analyses": "1- This paper presents experiments focusing on the unlearning of target objects; however, it does not investigate the potential impact of this unlearning process on closely related objects. There is a concern that excessive unlearning (over-unlearning) could unintentionally affect semantically similar objects, which ideally should be retained. Therefore, further experiments are required to demonstrate clearly that the proposed approach avoids over-unlearning in classes closely related—but not identical—to the target objects.\n\n2- For object/style unlearning in the CLIP model, experiments should also include evaluations on text-to-image retrieval tasks, as these can provide stronger and more compelling evidence of effective unlearning within the CLIP framework.\n\n3- This paper primarily focuses on unlearning a single object, such as \"Elon Musk,\" from a text-to-image diffusion-based generative model. After unlearning, the generated images currently result in random noise instead of depicting Elon Musk. For stronger and more convincing evidence, experiments should include scenarios involving complex textual prompts containing multiple objects alongside the unlearned object. The ideal outcome would be generating realistic images that accurately depict all other objects from the prompt while naturally omitting the target (unlearned) object, rather than producing random or noisy images. Including such observations would significantly strengthen the paper's empirical validation.\n\n4-  A more intriguing experimental setup for style unlearning could be demonstrated as follows: suppose we aim to unlearn the \"sketch\" style from a text-to-image generative model. After unlearning, prompts explicitly requesting the sketch style—such as \"a sketch of a dog\" or \"a sketch of a cat\"—should fail to generate sketch-style images of these animals. However, if prompts requesting other styles—such as \"a photo of a dog\" or \"a cartoon of a dog\"—are provided, the model should correctly produce images consistent with those styles. Additionally, when using a neutral prompt like \"generate a dog\" or \"generate a cat\" without specifying any style, the resulting set of images should exclude any sketch-like representations. Conducting and presenting such observations would offer compelling evidence and significantly strengthen the findings on style-specific unlearning.", "relation_to_broader_scientific_literature": "The primary contribution of this work, compared to prior approaches, is the effective unlearning of the target object by applying the unlearning process only within a single layer. This strategy enhances both the effectiveness and computational efficiency. However, a direct comparison highlighting computational efficiency against previous approaches has not been included, which would further strengthen the claims and clearly demonstrate this advantage.", "essential_references_not_discussed": "The paper effectively covers recent and essential related approaches. However, it would be beneficial to also include text-to-image retrieval methods to demonstrate the performance of the CLIP model after unlearning target objects. Including such results and references would offer stronger and more comprehensive empirical evidence.", "other_strengths_and_weaknesses": "Strengths:\n1- This paper proposes a computationally efficient approach for target object unlearning by applying the unlearning process to only a single layer.\n\n2- The paper applies unlearning techniques to diffusion-based generative models and the CLIP model, conducting comprehensive experiments on both object and style unlearning. Additionally, it evaluates the proposed approach using recent and more challenging datasets, such as UnlearnCanvas.\n\n3- A detailed ablation analysis is included to support the claims presented in the paper.\n\nWeaknesses:\n\n1- A comparison demonstrating the computational efficiency of the proposed approach relative to existing methods should be included.\n\n2- Experiments demonstrating the impact of unlearning target objects on semantically related retained objects should also be included. For example, in a fine-grained dataset, the top 5-10 closely related objects to the target (unlearned) object can be identified using similarity scores calculated from the retained set after unlearning. Subsequently, the generative capability of the model for these closely related objects should be evaluated. Additionally, for the CLIP model, conducting text-to-image retrieval tasks involving these closely related objects after target object unlearning would provide insightful observations and strengthen the evidence presented.\n\n3- To provide stronger and more compelling evidence, the experiments should include scenarios with complex textual prompts involving multiple objects, including the object targeted for unlearning. Ideally, the generated images should realistically represent all other mentioned objects while naturally omitting the target (unlearned) object, rather than producing random or noisy outputs. Incorporating such experimental observations would greatly enhance the empirical validation of the paper's claims.", "other_comments_or_suggestions": "Would applying unlearning to multiple layers yield more effective results?", "questions_for_authors": "Please see the concerns highlighted in the Weaknesses and Experimental Designs or Analyses sections and address all.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces an unlearning algorithm, SLUG, that promises removal of specific information with significantly lower computational overhead compared to some other methods, while preserving model utility. SLUG achieves this by efficiently identifying a single layer to update. Reviewers initially raised several concerns, such as robustness of unlearning to white box adversarial attacks, and the scope of experiments, which were mostly addressed during the rebuttal process with new experiments. In the camera-ready version, the authors should incorporate these new results, improve their discussion of computational costs, trade-offs, and single-layer editing approach, and also accurately state the limitations of their robustness evaluations (and consider expanding these evaluations).
Decision: Accept (poster)