CASteer: Cross-Attention Steering for Controllable Concept Erasure
Authors: Tatiana Gaintseva
Andreea-Maria Oncescu
Chengcheng Ma
Ziquan Liu
Martin Benning
Gregory Slabaugh
Jiankang Deng
Ismail Elezi
Venue: ICLR 2026 Poster (2026)
Keywords: steering
diffusion
control
erasure
Primary area: generative models
TL;DR: CASteer is a training-free method that steers cross-attention in diffusion models to precisely control concepts: enabling erasure, suppression, and broader semantic manipulation without retraining or degrading image quality.

Abstract: Diffusion models have transformed image generation, yet controlling their outputs for diverse applications, including content moderation and creative customization, remains challenging. Existing approaches usually require task-specific training and struggle to generalise across both concrete (e.g., objects) and abstract (e.g.,4 styles) concepts. We propose CASteer (Cross-Attention Steering), a training-free framework for controllable image generation using steering vectors to influence a diffusion model’s hidden representations dynamically. CASteer precomputes concept-specific steering vectors by averaging neural activations from images generated for each target concept. During inference, it dynamically applies these vectors to modify outputs only when necessary, either removing undesired concepts from images where they appear or adding desired concepts to images where they are absent. This selective activation ensures precise, context-aware adjustments without altering unaffected regions. This approach enables precise control over a wide range of tasks, including removing harmful content, interpolating between desired attributes, replacing objects, all without model retraining. CASteer outperforms state-of-the-art techniques while preserving unrelated content and minimising unintended effects.

Published as a conference paper at ICLR 2026
CASTEER: CROSS-ATTENTION STEERING FOR CON- TROLLABLE CONCEPT ERASURE
Tatiana Gaintseva 12, Andreea-Maria Oncescu², Chengcheng Ma Ziquan Liu', Martin Benning", Gregory Slabaugh', Jiankang Deng" I.smail Elezi
'Quenn Mary University of London Huawei Noah's Ark
CASIA
Uniiversity College London
'Imperial College London
ABSTRACT
Diffusion models have transformed image generation, yet controlling their outputs to reliably erase undesired concepts remains challenging. Existing approaches usually require task specifci training and struggle to generalize across both concrete (e.g., objects) and abstract (e.g., styles) concepts. We propose CAStteer (Cross- Attention Steering), a training-free framework for concept erasure in diffusion mod- els using steering vectors influence hidden representations dynamically. CASteer precomputes concept-specific steering vectors by averaging neural activations from images generated for each target concept. During inference, it dynamically applies these vectors to suppress undesired concepts only when they appear, ensuring that unrelated regions remain unaffected. This selective activation enables precise, context-aware erasure without degrading overall image quality. This approach achieves effective removal of harmful unwanted content across wide range of visual concepts, all without model retraining. CASteer outperforms state-of state-of-the-ard concept erasure techniques while preserving unrelated content and minimizing unintended effects. Code is available at httpss//gitubbcom/AAmyyCCCCASSS
INTRODUCTION
Recent advances diffusion models Ho et al. 2020); Rombach et al. (2022) have revolutionized image Podell et al. (2024) and video generation Girdhar et al. (2024). achieving unprecedented realism. These models operate by gradually adding noise to data during a forward process and then learning to reverse this noise through series of iterative steps, reconstructing the original data from randomness. By leveraging this denoising process, diffusion models generate high-quality, realistic outputs, making them a powerful tool for creative and generative tasks.
However, the same capabilities that make diffusion models transformative also raise profound ethical and practical concerns. The ability generate hyper-realistic content amplifies societal vulnerabilities. Risks range from deepfakes and misinformation to subtler effects such as erosion of trust in digital media and targeted manipulation. Addressing these challenges requires not only reactive safeguards (e.g., blocking explicit content) but proactive methods to constrain or remove harmful concepts at the level of the model itself. Current approaches to moderation often treat symptoms rather than causes, limiting their adaptability as risks and applications evolve.
Existing methods for concept erasure in diffusion models remain narrow in scope. LoRA-based fine- tuning Hu et al. (2022) is effective for removing specific objects styles but struggles with abstract or composite concepts (e-g., nudity, violence, or ideological symbolism), and scales poorly when multiple concepts must be removed, requiring separate adapters costly retraining. Prompt-based interventions Yoon et al. (2024) offer greater flexibility for abstract harm reduction but lack precision in suppressing concrete attributes, often failing generalize across concept variations. As result, existing strategies fall short of delivering reliable, efficient, and broad-spectre rroad-spectrum concept erasure.
In this work, introduce CASteer, training-free free method for controllable concept erasure that leverages the principle of steering to influence hidden representations of diffusion models dynamically. Our method builds on recent findings that deep neural networks encode features into approximately

---

Published as a conference paper at ICLR 2026
linear subspaces Elhage al. (2021); ; Wu al. 2023). Prior research has shown that intermediate subspaces of diffusion backbones also exhibit this property, with directions that modulate the strength of particular features Kwon al. (2023) Park et al. (2023); Si et al. (2024); Tumanyan et al. (2023);
Li al. 2024). Yet, these techniques remain limited in scope, often restricted to specific subspaces,
requiring training, or offering only coarse control.
Our approach departs from this paradigm. We show that multiple subspaces within diffusion models exhibit linear properties that can be harnessed for precise concept erasure. For each concept of interest, we generate k positive images (where k 2 1) containing the concept and k negative images not containing it, and compute the steering vectors by subtracting the averaged hidden representations of the network across negative images from those of positive ones. During inference, these precomputed vectors are applied directly to the model activations, allowing us to selectively suppress undesirable concepts without retraining or degrading the overall image quality. Experiments demonstrate that CASteer achieves fine-grained erasure of harmful or unwanted concepts (e.g., nudity, violence), while maintaining robustness across wide range of diffusion models, including SD 1.4, SDXL Podell ct al. (202)), Sana Xie et al. (2025), and their distilled variants (e.g.. SDXL-Turbo Sauer et al. (2022).
Sana- Sprint Chen et al. (2025)).
In summary, our contributions are the following:
We propose novel training-fres framework for controllable concept erasure diffusion models, leveraging steering vectors to suppress unwanted image features without retraining.
We demonstrate that CASteer effectively handles both concrete (e.g., specific characters) and abstract (e.g., nudity, violence) concepts, and scales to multiple simultaneous erasures.
We achieve soite--ofthheaat performance concept erasure across diverse tasks and diffusion backhones, validating the robustness, versatility, and practicality of our approach.
2 RELATED WORK
Data-driven Al Safety. Ensuring the safety of image and text-lo-image generative models hinges on preventing the generation of harmful or unwanted content. Common approaches include curating
training data with licensed material Rao (2023);
Schuhmann et al. (002)), fine-tuning models to
suppress harmful outputs Rombach et al.
(2022); Shi al.] (2020), or deploying post-hoc content
detectors [edapudi (022): Rando et al. (002).. While promising, these strategies face critical
limitations: data filtering introduces inherent biases Shi et al. (2020), detectors are computationally efficient but often inaccurate or easily bypassed Gandikota et al. (202)); SSmithMano (2022), and model retraining becomes costly when new harmful concepts emerge. Alternative methods leverage text-domain interventions, such as prompt engineering Shi et al. (2020) or negative prompts (romptssMMyykee jet al. (2023); Schramowski et al. (2023). Yet these remain vulnerable to adversarial attacks, lack flexibility and precision as they operate in the discrete space of tokens, and often fail to address the disconnect between text prompts and visual outputs- can still generate undesired content even when text guidance is "safe". Our approach instead operates in the joint image- latent space of diffuuson models, enabling more robust and granular control over generated content without relying solely textual constraints.
Model-driven AI Safety. Current methods Gandikota et al. (0033) y Kumari et al. 2023) Heng &
Soh (2023): ; Zhang al. (2024)) Huang al. (2024); Lee al. (2025) erase unwanted concepts by fine-tuning or otherwise optimising models and adapters to shift probability distributions toward null
or surrogate tokens, often combined with regularization or generative replay (Shin et al. (2017). Other methods, such as Gandikota et al. (2024); Gong et al.] (2024). use direct weight editing to remove unwanted concepts. Although effective, these approaches lack precision, inadvertently altering or removing unrelated concepts. Advanced techniques like SPM Lyu et al. (2024) and MACE Lu et al. 2024) improve specificity through LoRA adapters Hu et al. (2022), transport mechanisms, or prompt-guided projections to preserve model integrity. However, while promising for concrete concepts (e.g.. Mickey Mouse), they still struggle with abstract concepts (e.g., nudity) and require parameter updates. Another group of methods focuses on interventions into internal mechanisms
of generative models. Methods like Prompt-to- Prompt-to-Promptt Hertz et al. (2023) enable fine- grained control over text-specified concepts (e.g., amplifying replacing elements) through interventions to cross-attention maps, yet fail to fully suppress undesired content, particularly when concepts are implicit absent from prompts. This task-specific specialization limits their utility for cafety-critical
2

---

Published as a conference paper at ICLR 2026
erasure, where complete removal is required. CASteer bridges this gap. enabling precise, universal concept suppression without relying on textual priors or compromising unrelated model capabilities.
Another area of research focuses removing information about undesired concepts from text embeddings that generative models are conditioned on Yoon et al. (2024); Zhang et al. (2024b); Qiu et al. (2024). However, these methods operate on discrete space of token embeddings, their trade-off between the effectiveness of crasure and the preservation of other features is limited. Zhang et al. (2024c) proposes using adversarial training for concept unlearning; however, training this method is computationally intensive. In contrast, CASteer eliminates training entirely, enabling direct, non-invaan non-invasive concept suppression in the model's latent space without collateral damage to unrelated features.
Utilizing directions in latent spaces. This area research focuses on finding interpretable directions in various intermediate spaces of diffusion models K won et al. (2023); Park et al.] (2023); (Si et al. ( (Tumanyan et al. (2023), which can then he used to control the semanties of generated images. Based on this idea, SDID Li et al. 2022)) recently proposed to learn vector for each given concept, which is then added to the intermediate activation
bottleneck layer of the diffusion model during inference to provoke the presence of this concept in the generated image. However, this method is highly architecture-specinn and fails to deliver precise control over attributes. In our work, we propose training-free method for constructing interpretable directions in intermediate activation spaces of various diffusion models for more precise control of image generation. SAeUron Cywinski & Deja (2025) utilizes Sparse Autoencoders Olshausen & Field (1997) (SAEs) find interpretable directions in the activation space of the diffusion model. However, SAES are unstable, require extensive training, and do not provide initial control over the of attributes that can be erased. In contrast, CAStcer
does not require training and provides direct control over the manipulated attributes.
3 METHODOLOGG
The main operating principle of CASteer to modify outputs of certain intermediate layers during inference in order to affect the semantics of generated images, thus preventing the generation of desired concept. These outputs are modified using specially designed steering vectors. In this section, we begin by justifying the choice of the intermediate layers that CASteer modifies (Sec. (Sec.3.1), then proceed with the procedure of construction of steering vectors (Sec.3.2) and after that describe how these steering vectors are used during inference control the generation process (Sec. (Sec.3.3). Finally, we elaborate on practical aspects regarding the calculation and use of the steering vectors (Sec. 3.4
3.1 CHOICE OF LAYERS TO STEER
Most modern diffusion models use U-Net or Diffusion Transformers (DiT) Peebles & Xie (2023) backbone. They consist of a set of Transformer blocks, each having three main components: cross-attention (CA) layer, self-attention (SA) layer, and MLP layer, all of which contribute to the residual stream of the model. Among those, CA layers are the only place in the model where information from the text prompt goes into the model, guiding text-to-image generation. For every image patch and prompt embedding, each CA layer generates a vector matching the size of the image patch embedding. After summation, these vectors transmit text rext-prompt information to corresponding image regions Hertz al. (2023).
As the semantics of the resulting image is mostly determined by the text prompt, we modify the outputs of the CA layers during inference, which results effective, yet precise, control over the features of the generated image. Thus, CASteer constructs steering vectors for the outputs of every CA layer in the model. the appendix, we present experiments on applying CASteer to steer outputs of other layers (SA, MLP, and outputs of intermediate layers inside CA blocks).
3.2 CONSTRUCTION OF STEERING VECTORS
We propose to construct steering vectors for each concept we aim to manipulate. These vectors correspond to the cross-attention (CA) outputs we modify. Each steering vector matches the size of the CA outputs and encodes the desired concept's information. For preventing the concept from being present the generated image, we subtract steering vectors of an unwanted concept from cross-attention outputs during generation.
3

---

Published as a conference paper at ICLR 2026
Generation with steering
DM
DM
DM
cal,
anime style
City
anime style
anime
timestamp
Transfomer block:
timestamp. -
Transformer block
Text emcoder
1000 1000
A cal, anima style
Calo
dag. style
Kngy
A
over patches
Steering vector for layer timestamp
Text encoder
Col
Modifying CA outputs on current timestamp steering vector
timestamp
Transtoreer block
timestarp
Computation of steering vector
Figure 1: Main pipeline. (Bottom left, gray background) For computing a steering vector, we prompt diffusion model with two prompts that differ in desired concept, c.g., "anime style" and save CA outputs at each timestamp and each CA layer i. We average these outputs over image patches and get averaged CA outputs pas___vg and ca neg_aavg for each and i. We subtract the latter from the former, getting steering vector for the layer i and timestamp / cagnime, (Right) For deleting concept X from generation, at each denoising step t, we subtract stecring vector cafe by intensity from the CA outputs of the layer
animme multiplied
We construct steering vectors follows. Given concept X to manipulate, create paired positive and negative prompts differing only by the inclusion of X. For example, X "baroque style", example prompts Ppos "A picture of man, baroque style" and Pneg "A picture of man". Assume a DiT backbone has N Transformer blocks, each containing one CA layer, totaling CA layers. We generate images from both prompts, saving outputs from each of the N cross-attention layers across all denoising steps. This yields NT cross-attention output pairs (ca ca neg, for 1 & i N and & t T, where denotes the layer and is the denoising step. Each ca and caje has dimensions patch_numy X emb size, corresponding to the number of patches and embedding size layer i. We average ca and ca neg over image patches obtain averaged cross-attention outputs:
pos
Epatch nontch_num; numy pos
patch num; neg
pos_ovg catt
co
neg_ovg
patch_num,
patch_remm
where ca pos_a09 and co. co neg_oog are vectors of size emb_ mmb_size,, Then, for each of these N layers and each of T denoising steps, we construct a corresponding steering vector carrying a notion of X by subtracting its averaged cross-atio aots-attention output that corresponds to the negative prompt from that corresponding to the positive one as:
ca fnorm ( (catt -CO
pas_avg
neg_aog
where Jnorm is Ln-normalization function: fnorm(
T
3.3 USING STEERING VECTORS TO CONTROL GENERATION
Computed steering vectors can be seen as directions in a space of intermediate representations of model (in the space of CA activations) that represent notion of X. Thus, should be able to control the expressiveness steering vector representing X. That is. we can prevent concept from appearing the generated image by subtracting some amount of steering vectors for that concept from corresponding CA outputs of a model during inference:
certain feature X by steering the model representations along the
out_new
X
CO
4

---

Published as a conference paper at ICLR 2026
Figure 2: Heatmaps of dot product values between CA outputs of SDXL model (layer 15, denoising step 0) and steering vector for Snoopy concept". Top: generated im- ages, bottom: heatmaps. Images are gener- ated based on the prompt "A bright photo a X", where X E Snoopy". Pikachu", dog". We see that for the image that con- tains Snoopy (left), dot product values are high for those image tokens that correspond to the image tokens that actually produce Snoopy. For images that do not contain Snoopy, dot product values are low for all image tokens.
Here 1 < k & patch n and a is hyperparameter that controls the strength of concept suppres- sion. Larger values of lead higher suppression of the concept X in the generated image. Below we propose a way to adjust a dynamically based on activations of diffusion model during generation, achieving effective and precise erasure of unwanted concepts in the resulting image.
Choice of alpha. Most often when we aim to suppress the concept of X, our goal is to completely prevent it from appearing on any generated image given any input prompt. This the case of such tasks as nudity/violence removal or privacy, when we do not want the model to ever generate somebody's 's face or artwork. However, there might be different magnitudes for concept in the original text prompt (e.g,, prompts "an angry man" or "a furious man" express different levels of anger). concept X can have different magnitudes of expression different patches of the image being generated. Consequently, if we use Eq. 3 for suppression, different values of a are needed to completely suppress X for different prompts and individual image patches while not affecting other features in the image.
We propose to estimate a for concept deletion by using the dot product between ca and correspond- ing CA output out (feu) cm as an assessment of amount of X that present the image part corresponding to ttth patch of ca (see Fig. 2). As ea is normalized, the value of this dot product the length of the projection of the CA output ca onto the steering vector caty As eaty can be seen as a direction in a linear subspace corresponding to the concept X, the length of the projection can be seen the amount of X that is present ca That said, for removing information about X from out we propose to subtract the amount of cate proportionate to the dot product between out_new i.e., define a B(ca,) ). Consequently, Eq. 3 becomes the cosy and cast from co
following:
out_new new = stk - Bleaxt out cat co
out
Here 1 < k < patch_num, and 3 is hyperparameter that controls the strength of the suppression.
Note that Eq. caan be reformulated in matrix form as a projection operator onto the subspace orthogonal to steering vector 8 = catt
new
Saatete (c, 8) = (I - "--ss=") e
Here new
co
out_new
oa
out ith
8 = ca I an identity matrix.
Intermediate clipping. We now introduce mechanism of clipping the value of to get better control over concept suppression. Note that using Eq. 4 we only want to influence those CA outputs ca which have positive amount of unwanted concept X. As dot product (ca ca measures the amount of X present CA output oa out we only want to steer those CA outputs cagu which have a positive dot product with ca So the equation becomes the following:
a = max((3((aa out, ) 0)
oot_new = co itk
out occ.
Note that if intermediate clipping is used, Eq. 6 can no longer be formulated in matrix form. In the experiments section, we present results of applying CASteer for concept erasure both with and without intermediate clipping (i.e. using Eq. and Eq. 6).
5

---

Published as a conference paper at ICLR 2026
3.4 PRACTICAL CONSIDERATIONS
Multiple Prompts for Steering Vector. We described in the previous section how to construct and use steering vectors alter one concept, based on one pair of prompts, e.g., "a picture of man" and "a picture of a man, baroque style". As mentioned, steering vector can be seen as the direction in the space of intermediate representations of a model that points from an area of embeddings not containing a concept X, to an area that contains it. In order for this direction to be more precise, we propose to construct steering vectors based on multiple pairs of prompts instead of one. More neg ang
precisely, we obtain P > pairs of pass_ang and ca
< &
P. then average them over P:
Ep
pos_ang
Epm
P neg
ca
pos_avg aog st
P
00
it
avg
P
and obtain steering vectors as case
pos_avg
ca it
neg_avg
Steering multiple concepts. It is easy to erase multiple concepts during a generation by either mutually orthogonalizing set of steering vectors corresponding these concepts and applying them to the cross-attention output successively, or constructing single steering vector corresponding to multiple concepts by simply averaging individual steering vectors. In the experimental section, we show results on applying a steering vector constructed for multiple concepts using averaging approach to prevent generation of inappropriate concepts. Additionally, in the appendix (Sec. MM) we show results on erasing multiple concepts by orthogonalizing their corresponding steering vectors.
Efficiency: Transferring vectors from distilled models. Adversarial Diffusion Distillation (ADD) Sauurr et al. (2022) is a fine-tuning approach that allows sampling large-scale foundational image diffusion models to steps, while producing high-quality images, with many methods such SDXL and Sana having distilled versions (SDXL- Turbo and Sana-Sprint). We observe that steering vectors obtained from the distilled models can successfully be used for steering generations of its corresponding non-distilled variants. More formally, having a pair of prompts, we obtain cafos pos_avy and ca, neg_arg from the distilled model using 1 denoising step. Note that there is no second index t concept X as cnt"" = ca. pos__an ang ca,
use only one denoising iteration, i.e. T = 1. We then construct steering vectors for the neg meg_ang ang and then use it to steer non-Turba variant of the model
by using ca." for each denoising step 1 < j < T.
Injecting CASteer into model weights. Note that when steering more advanced models (SDXL or Sana), we use steering vectors from Turbo/Sprint model versions, where we have only one steering vector per model CA layer. Also note that the last layer of CA block in SDXL/SANA is Linear layer with no bias and no activation function, i.e., essentially is a matrix multiplication: hout W prrof_ on Here W Wprojoout is weight matrix of the last proj_oul layer of CA block of SDXL/SANA, hin and hout are input and output to that layer, hout being the final output of CA layer. In this case, by combining last layer of CA block with CASteer formulation in a matrix form (Eq. [5) we can incorporate CASteer directly into weights of the model, by multiplying weight matrix of the last layer of CA block with I ss matrix from Eq. 5
hout = (1 - ss W WPrrooouthhe = Wi proj_out han
(8)
Wiroj_ j a matrix of the same size as W proj This results in having zero inference overhead compares to original SDXL/SANA model similar to LoRA-like tuning approaches.
4 EXPERIMENTS
We evaluate the performance of our method on the task of erasing different concepts. We show that our method succeeds in suppressing both abstract (e.g.. "nudity", "violence")) and concrete concepts (c.g., "Snoopy"). Moreover, we demonstrate the advantages of our method in removing implicitly defined concrete concepts (c.g., a concept is "Mickey", prompting "a mouse from a Disneyland" should not result in generation of Mickey).
Implementation details. For a fair comparison, we report our main quantitative results using StableDiifusion-v// (SD-v1.4) RoDmbaah et al. (2022). SD-1.4 model does not have a Turbo version, so for these experiments we use per-step steering vectors computed from the original SD-1.4. We apply steering to all of the CA layers in the model. We set 3 2 for the concept erasure in all

---

Published as a conference paper at ICLR 2026
Table Quantitative results on nudity removal based on 12P Schramowski et al. (2023) dataset. Detection of nude body parts is done by Nudenet at a threshold of 0.6. F: Female, M: Male. The best results are highlighted in bold, second- best are underlined.
Method
Nadiyy Detection
Bivest
GeniralocF)
BeasstM)
GestallaMM)
Bumooes
Feet
Belly
Ampits
Treal
6+45
DoC'
163
74
65
64
127
148
750
1435
Abla me
205
23
45
65
180
153
835
117
424
SI
M61
19
73
47
213
LX Smn honrraa
35
20
62
20
182
SAParr 9.30 Purned
39
20
15
163
Mabrrammmn-- Trtse
121
MAC Tertei
109
RECT (irggn pund
3
23
17
66
CPE Wort (atraa 205
11
15
13
15
66
(ooss wand PacCeat
18
Ous (w07t)))
12
Ouss (clig)
Table 2: Quantitative results on inappropriate content removal based on 12P Schramowskii et al. (2023) dataset. Detection of inappropriate content is done by Q16 (Schramowski al. (2022) classifier. The best results are highlighted in bold, second- best are underlined.
Teoppropeite propertonn (R)(C)
FMI
Ablain
ESD-x
SLD
ESD-u
UCE
Reeele
(ssx(wibr
Ouai (clp)
Hate
37.2
43.5
31:1
22.5
25.5
36.4
$5.5
13,00
08.5
34.1
17.75
Scll
47.9
68.8
47.4
99.8
38.3
55.7
30.5
34.8
20 22
Scual
60.2
00.1
00.5
01.2
92.4
560
25.5
20.4
22.90
28:73
Skodkigg
65-2
40.5
35.
experiments. This choice is motivated by the fact that with 8 2, the 54..omecmmee Householder operator (reflection) of the CA activation vector e across the hyperplane orthogonal to the steering vector This operation preserves I--norm of the vector thus keeping relative and absolute values of all the information present in that is orthogonal to s intact after transformation. In appendix (see Sec. H.3 we ablate the choice of 3. We show 3 leads to lower level of concept suppression, while still producing high quality images, enabling control on the level of concept erasure. We also show that for SDXL and SANA models, values of B > 2 lead to stronger concept erasure, while also leaving general quality of images high. We use 50 prompt pairs for generating steering vectors for concrete concepts (e.g., "Snoopy")), and 196 prompt pairs for generating steering vectors for abstract concepts (e.g., "nudity"). Information about prompts used for generation of the steering vectors is in the appendix.
We also show effectiveness of CASteer bigger models, such SDXL and SANA, and present results on these models the appendix. For SDXL Podell al. (2024) and SANA, we use steering vectors obtained from SDXL-Turbh and SANA-Sprint models, respectively.
4.1 RESULTS
Abstract concept erasure. In this section, we present results on inappropriate content erasure based on 12P dataset Schramowski et al. (2023). I2P is dataset of 4,703 curated prompts designed to test generative models, where most prompts lead to images containing inappropriate content. Following prior work, we test CASteer on two 12P-based tasks: 1) removing nudity, 2) removing all inappropriate content at once. For nudity removal, we utilize CASteer with steering vectors generated for the concept of "nudity". For inappropriate content removal, we use CASteer with steering vectors obtained as average of steering vectors generated for each type of inappropriate content, i.c., hate, harassment, violence, self-harm, shocking, sexual, and illegal content.
We compare our method with the following state-of-the-art approaches: Ablating (CA) Kumari 2023). FMN Zhang et al. (2024a), SAeUron Cywiński & Dejal (2025) SLD Schramowski
et al.
ct al. 2023 ESD Gandikota et al. (2023), UCE Gandikota et al. (2024) MACE Lu et al. (2024).
SA Heng & Soh 2023), Receler Huang et al. (2024), RECE Gong al. (202)) CPE Lee et al. 2025
Adv Un ZZhang et al. (202cc) and. Following prior art, we utilize the NudeNet to detect nude body

---

Published as a conference paper at ICLR 2026
parts on generated images for nudity erasure task, and the NudeNet with the Q16 detector detect inappropriate content.
We present the results for CASteer versions with and without intermediate clipping applied in Tab. []
and Tab, 2 We show that both versions of CASteer outperform all prior models on nudity erasure, with CASteer version with clipping having more than times fewer images with detected nudity than the second-best result. On the inappropriate content removal, CASteer version with clipping also achieves state-of-the-ars result, surpassing second-best model Receler by 1.42% overall.
To assess general generation quality of CASteer, we follow prior work and run CASteer with "nudity" steering vectors on prompts from COCO-30k [in et al. (2014). We report FID Heusel et al. (2017) for general visual quality and CLIP score Hessel et al. (2021) for image-prompt alignment. Results
are presented in Tab. Both versions of CASteer have better FID than prior art.
Thus, CASteer clearly is capable of deleting unwanted information while maintaining general high quality. Note that these datasets feature adversarial prompts, i.e., the "nudity" concept encoded in the prompts implicitly.
Table 3: Evaluation of nudity-erased mod- els. Robustness is measured with nudity prompts from the I2P dataset, while locality is assessed using COCO-30K prompts.
Locality
Method
CLIP-30K(1)
FID FID-30K())
SD
31.34
14.04
FMN
30.39
13:52
CA
31.37
16.25
AdvlIn
17.18
Receler
Figure 3: SPM and DoCo failure in removing im- plicitly defined concepts (SD-1.4). Top: CASteer, Middle: SPM, Bottom: DoCo. Prompt: "a mouse from Disneyland," CASteer erases Mickey con- cepts despite not being explicitly named, while SPM and DoCo fail.
MACE
29.4]
13,42
CPE
31.19
13.89
UCE
30.85
14,07
SLD-M
30.90
ESD-x
30.69
ESD-u
30.21
15.10
SAcUron
30.89
14.37
Ourz Iwfo clip)
30.69
13.28
(clgg)
Concrete concepts erasure. To assess ability of CASteer to remove concrete concepts, we follow the experimental setup of Lyu et al. (2024). In this setting, the concept to be erased is Snoopy, and images of five additional concepts (Mickey, Spongebob, Pikachu, dog and legislator) are generated to test the capability of the method to preserve content not related to the concept being removed. The first four of these are specifically chosen be semantically close to the concept being removed to show the model's 's ability to perform precise erasure. Following SPM Lyu et al. (024) and DoCo Wu al. (2025), we augment each concept using 80 CLIPPRadfon et al. (2021) templates, and generate 10 for each concept-template pair, so that for each concept there are 800 images. We evaluate the results using two metrics. First, we utilize CLIP Score (CS) Hessel et al. (202I) to confirm the level of the existence of the concept within the generated content. Second, we calculate FID Heusel et al. 2017) scores between the set of original generations of SD-1.4 model and a set of generations of the steered model. We use it to assess how much images of additional (non-Snoopy) concepts generated by the steered model differ from those of generated by the original model. A higher FID value demonstrates more severe generation alteration. We present the results in Tab. 17 In Fig. 4 we also show two types of plots. Fig. 4a pictures normalized clip score of source concept, i.e. "Snoopy" (the lower the better) versus mean normalized clip scores of other concepts (the higher the better). Normalization is done to ensure equal importance of all the concepts in the mean, and done by dividing clip score of images produced by erasing method by clip score of images produced by vanilla SD-1.4. Methods on the left of the plot erase Snoopy well, and methods on top of the plot preserve other concepts well. Fig. [bb pictures normalized clip score of source concept versus mean FID scores of other concepts (the lower the better). Methods on the left of the plot erase Snoopy well, and methods on top of the plot tend not to affect images of other concepts much.
Results show that CASteer maintains good balance between erasing unwanted concept, while pre- serving other concepts intact. ESD Gandikota et al.; (2023) and Receler erase Snoopy well, but also highly affect other concepts, especially related ones such Mickey Spongebob. Note that their CS
8

---

Published as a conference paper at ICLR 2026
Table Comparison of Artist Concept Removal tasks: Famous (left) and Modern artists (right).
Remeve Vias Cogh"
Remose
"Belly McKerman"
Methed
JLPEPS T LPIPS.
Acc.
Acc, 1 LPPIS,. 1 LIPPPS
Acc.
Aee,
SD-v14
0.95
Nunrita 2011
0.30
01:15
0165
01.00
0.32
0.11
0.76
REC
0.31
0.08
0.30
0.93
0.29
0.44
0.55
0.76
(ennierr 2054
0.29
11:85
0.9
0138
0.75
0.83
0.80
0.11
MLl-M Sctnapon at 2023
0.21
0.91
0.22
0.47
048
Dam (wbbclip)
0.46
0.55
0.88
0.54
0.21
0.00
Durs (elip)
0.44
0.30
0.35
0.86
0.54
0.26
0.85
0.5
of unrelated concepts (e.g. "Mickey" or "Spongebob") are significantly lower than that of original SD-1.4, ..4 indicating that these concepts are also being affected when erasure of "Snoopy" is done. High FID of these methods on these concepts supports this observation. SAFREE shows reduced level of Snoopy erasure compared to that of CASteer, and has lower CS and higher FID on all the other concepts. SPM keeps unrelated concepts almost intact (High CS and low FID), but has a much lower intensity of Snoopy crasure. Moreover, SPM fails to crase implicitly defined concepts (see Fig. and Sec. Din in the appendix). We provide qualitative results on comparisons in Fig. 5 and in the appendix.
Sprralae Seerge (omee Cocaeoto
SM
Duk
(oher Concestt)
k30-
Gat k0:00
A42
Normalizes Snocpy
Smoway
(a) Normalized CLIP score on "Snoopy" vs normalized CLIP scores of other concepts
(b) Normalized CLIP score on "Snoopy" vs
FID scores of other concepts
Figure 4: Comparison of various methods on concrete concept erasure (removing "Snoopy")
Style erasure. Following SAFREE Yoon et al. (0024) and ESD Gandikota et al. (2023), also evaluate our method on two artist-style removal tasks. One task focuses on the styles of five famous artists (Van Gogh, Picasso, Rembrandt, Warhol, Caravaggio) and the other uses five modern artists (McKernan, Kinkade, Edlin, Eng, Ajin: HemmiHHmmnn,, with the task being removing the style of Van Gogh and McKernan. Following SAFREE Yoon et al. (2024), we use LPIPS Zhang al. (2018) and prompt GPT-4o to identify an artist on generated images as evaluation metrics.
We follow SAFREE Yoon et al. (024) evaluation procedure, please refer to it our appendix (Sec.E) for more details on the procedure and metrics. From Tab. 4 we see that CASteer achieves the best results in style removal (see columns LPIPS, and Acc,), while preserving other styles well (see columns LPIPS,, and Acc,). Among all the approaches, CASteer achieves great balance between target style removal and preservation of other styles.
CASteer is capable of erasing implicitly defined concepts. We check what happens if we define the prompts implicitly. e.g.. "A mouse from Disneyland". We run CASteer, SPM and DoCo trained on the Mickey concept on these prompts and show the results in Fig. 3 We clearly see that SPM and DoCo fail to erase the concepts when they are not explicitly defined. In contrast, our method does a much better job of crasing the concepts, despite being implicitly defined. This is also supported by the results on nudity crasure in Tab. [ as considered datasets contain specially selected adversarial nudity prompts. We provide additional results on implicitly defined prompts in the appendix.
Overall experimental results show that CASteer performs precise erasure of both concrete and abstract concepts and concepts defined implicitly while leaving other concepts intact and not affecting the overall quality of generated images. More qualitative results showing the performance CASteer on prompts related and not related the target concept can be found in the appendix.

---

provide more Published as a conference paper at ICLR 2026
Figure 5: Qualitative results on SDXL (left) and SANA (right) on removing "Snoopy". Top: original model generations, bottom: generations of model steered to remove "Snoopy"
Picasso
Rembrandt
Warhol
Caravaggl
(romoved
Figure 6: Qualitative results on SANA (right) removing style of "Van Gogh". Top: original model generations, bottom: generations of model steered to remove "Van Gogh" style. Left 2 images correspond prompts with "Van Gogh style", images on the right correspond to prompts mentioning other artists ( ("Picasso", "Rembrandt", "Warhol", "Caravaggio")
4.2
ABLATION STUDY
Steering other layers. As mentioned in Sec. 3.1 the DiT backbone steering is most effective. We show in the appendix that steering the CA outputs is the most effective. We also ablate steering only fraction of CA layers in Sec. H.5
we ablate to determine for which type of layer in
Number of prompt pairs to construct steering vectors. the appendix, we provide an ablation on the number of prompt pairs needed to produce high-quality outputs after steering. We find that as little as 50 prompts is enough for the steering vectors to capture the desired concept well.
Interpretation of steering vectors. Here, propose way to interpret the meaning of steer- ing vectors generated by CASteer. Suppose we have steering vectors generated for concept X { & < 1,1 & & T, where I is the number of model layers and T is the number of denoising steps performed for generating steering vectors. To interpret these vectors, we prompt the diffusion model with placeholder prompt "X" and at each denoising step, substitute outputs of the model's CA layers with corresponding steering vectors. This conditions the diffusion model only on the information from the steering vectors, completely suppressing other information from the text prompt. Results are presented in Fig. [1 and in the appendix.
UMap. We generate steering vectors for all vocabulary tokens of SDXL text encoders and apply UMap Mclnnes & Healy (2018) on these steering vectors. We present the results in the appendix, showing that structure emerges in the space of these steering vectors, similar to that Word2Vec Mikolov et al. (2013), supporting that steering vectors carry the meaning of the desired concept.
Modern models. We show qualitative results SANA and SDXL in Fig. G qualitative and quantitative results in SDXL (Sec. F) and SANA (Sec. G
User studies. In Sec, D we give several user studies, showing that in most cases, the users prefer our results compared to SPM and Receler.
5 CONCLUSION
We presented CASteer, novel training-free method for controllable concept erasure in diffusion models. CASteer works by using steering vectors in the cross-atee aoossatttention layers of diffusion models. We show that CASteer is general and versatile to work with different versions of diffusion, including distilled models. CASteer reaches state-of-tha the-ttt results in concept erasure on different evaluation benchmarks while producing visually pleasing images.
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGMENTS
This work was supported by the Engineering and Physical Sciences Research Council [grant number EP/Y009800/1], through funding from Responsible Ai UK (KP0016).
REFERENCES
Praneeth Bedapudi. Nudenet: Neural nets for nudity detec- tion and censoring. 2022.
Junsong Chen, Shuchen Xue, Yuyang Zhao, Jincheng Yu, Sayak Paul, Junyu Chen, Han Cai, Song Han, and Enze Xie. Sana-sprint: One-step diffiusion with continuous-time consistency distillation, 2025.
Bartosz Cywinski and Kamil Deja. SAeuron: Interpretable concept unlearning in diffusion models with sparse autoencoders. In ICML, 2025.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfeld-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. http://taannfommmr-crrruetue www.euitspp.///21111111 /ttp///01//rammewo///nonoaa//
Rohit Gandikota, Joanna Materzynska, Jaden Frotto-Kaufman and David Bau. Erasing concepts from diffusion models. In ICCV, 2023.
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzynska, and David Bau. Unified concept editing diffusion models. WACV, 2024.
Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. In ECCV, 2024.
Chao Gong, Kai Chen, Zhipeng Wei, Jingjing Chen, and Yu-Gang Jiang. Reliable and efficient concept erasure of text-to- image diffusion models. In ECCV, 2024.
Alvin Heng and Harold Soh. Selective amnesia: A continual learning approach to forgetting in deep generative models. In NeurlPS, 2023.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt- to-prompt image editing with cross-attention control. In ICLR, 2023,
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. EMNIP,, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In NeurlPS, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation large language models. ICLR, 2022.
Chi-Pin Huang, Kai-Po Chang, Chung-Ting Tsai, Yung-Hsuan Lai, Fu-En Yang, and Yu-Chiang Frank Wang. Receler: Reliable concept erasing of text-to-image diffusion models via lightweight erasers. ECCV, 2024.
Nupur Kumari, Bingliang Zhang, Sheng- Yu Wang, Eli Shechtman. Richard Zhang, and Jun-Yan Yan Zhu. Ablating concepts in text-to-image diffusion models. In ICCV, 2023.
Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have A semantic latent space. In ICLR, 2023.
11

---

Published as a conference paper at ICLR 2026
Byung Hyun Lee, Sungjin Lim, Seunggyu Lee, Dong Un Kang, and Se Young Chun. Concept pinpoint eraser for text-to-image diffusion models via residual attention gate. In ICLR, 2025.
Hang Li, Chengzhi Shen, Philip Torr, Volker Tresp, and Jindong Gu. Self-discovering interpretable diffusion latent directions for responsible text-to-image generation. CVPR, 2024.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014.
Shilin Lu, Zilan Wang. Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. MACE: mass concept crasure in diffusion models, In CVPR, 2024.
Mengyao Lyu, Yuhong Yang, Haiwen Hong, Hui Chen, Xuan Jin, Yuan He, Hui Xue, Jungong Han, and Guiguang Ding. mie-dimensional adapter to rule them all: Concepts, diffusion models and erasing applications. In CVPR, 2024.
Leland Mclnnes and John Healy. UMAP: uniform manifold approximation and projection for dimension reduction. CoRR, abss/180..3346,, 2018. URL [RR::tp//aaxxx.oor ab88
03426
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation word representa- tions in vector space. In ICLR, 2013.
Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka. Negative-prompt inversion: Fast image inversion for editing with text eext-guided diffusion models. CoRR. abs/2305.16807, 2023.
Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vision Research, 37:3311-3325, 1997. URL https://api
semanticscholar wttp:r//ooooooss:11000....
Yong-Hyun Park, Mingi Kwon, Jaewoong Choi, Junghyo Jo, and Youngjung Uh. Understanding the latent space of diffusion models through the lens of riemannian geometry. In NeurlPS, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. In ICLR, 2024.
Huming Qiu, Guanxu Chen, Mi Zhang, and Min Yang. Safe text-to-image generation: Simply sanitize the prompt embedding. CoRR, 0os/2411.1032) 2024. doi: D04850/ARXII.241..000 URL ht httpps://dii.oo org/O0..rgg..... wtt.or/./....000........ www.a.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramèr. Red-teaming the stable diffusion safety filter. CoRR, abs/2210.0461111 2022.
Dana Rao. Responsible innovation in the age of generative ai. 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. In ECCV, 2022.
Patrick Schramowski, Christopher Tauchmann, and Kristian Kersting, Can machines help us answer- ing question 16 in datasheets, and in turn reflecting on inappropriate content? In ACM FAceT, 2022.
Patrick Schramowski, Manuel Brack, Bjöm Deiseroth, and Kristian Kersting. Safe latent diffusion: Mitigating inappropriate degeneration in diffusion models. In CVPR, 2023.
12

---

Published as a conference paper at ICLR 2026
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: an open large-scale dataset for training next generation image-text models. In NeurIPS, 2022.
Zhan Shi, Xu Zhou, Xipeng Qiu, and Xiaodan Zhu. Improving image captioning with better use of captions. CoRR, 1807 2020.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay, In NeurIPS, 2017.
Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. In CVPR, 2024.
SmithMano. Tutorial: How to remove the safety filter in seconds. 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusson implicit models. ICLR, 2021.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and and-play diffusion features for text-driven taage-to-image translation. CVPR, 2023.
Yongliang Wu, Shiji Zhou, Mingzhuo Yang, Lianzhe Wang, Heng Chang, Wenbo Zhu, Xinting Hu, Xiao Zhou, and Xu Yang. Unlearning concepts in diffusion model via concept domain correction and concept preserving gradient. In AAAI, 2025.
Zhengxuan Wu, Atticus Geiger, Thomas Icard, Christopher Potts, and Noah D. Goodman. Inter- pretability at scale: Identifying causal mechanisms in alpaca. In NeurIPS, 2023.
Enze Xie, Junsong Chen, Junyu Chen, Han Cai, Haotian Tang, Yujun Lin, Zhekai Zhang, Muyang Li, Ligeng Zhu, Yao Lu, and Song Han. SANA: Efficient high-resolution text-to-image synthesis with linear diffusion transformers. In ICIR, 2025.
Jachong Yoon, Shoubin Yu, Vaidehi Patil, Huaxiu Yao, and Mohit Bansal. SAFREE: training free and adaptive guard for safe text-lo-image and video generation. In ICLR, 2024.
Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. CVPRW, 2024a.
Hongxiang Zhang, Yifeng He, and Hao Chen. Steerdiff: Steering towards safe text-to-image diffusion models. CoRR, abs/2410.02710,, 2024b. doi: 104850ARXXI2240007700. URL
480..44555500 48550/5550000 arxiv. www.:iiww........... 02710
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu. Defensive unlearning with adversarial training for robust concept erasure in diffusion models. In NewrIPS, 2024c.
13


Reviews:
[{"review_id": "YEYhV6DLqT", "reviewer": "Reviewer_ZNEe", "summary": "This paper introduces CASteer, a training-free framework for erasing specific concepts from diffusion models. The core idea is to compute steering vectors that represent the direction of an unwanted concept in the model's cross-attention layers. During inference, these vectors are subtracted from the model's activations without requiring any model retraining. The authors did comprehensive empirical experiments to demonstrate the effectiveness of the proposed method across various concepts, prompts, and model architectures.", "questions": "- the hyperparameter $β=2$ is used for all experiments. How does the method's performance and stability vary with this parameter, especially across different concepts or model architectures?\n- the paper uses a large number of prompt pairs (50 for concrete, 196 for abstract concepts). How sensitive is the method to the quality and diversity of these prompt pairs? poorly chosen set of prompts could lead to an imprecise steering vector.\n- for multi-concept erasure, how do you resolve conflicting steering directions? is orthogonality assumptions still valid?\n- the results on using steering vectors for adding concepts (and style transfer) are rather limited, can you discuss why certain applications fails with the proposed method?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- a training-free framework using precomputed steering vectors to remove concepts from diffusion models.\n- comprehensive empirical evaluations across various concepts and models.", "weaknesses": "- limited novelty regarding method, with strong assumptions on steering vectors’ linear compositionality.\n- ad-hoc parameter selection - steering vector scaling hyperparameters are fixed empirically.\n- no discussion of computational or memory trade-offs when constructing per-layer, per-step steering vectors.\n- somewhat limited improvements, for example, CASteer with clipping only surpassing second-based model Receler by 1.42%, which is not significant.\n- did not compare with unlearning baselines, such as [1,2]. \n\n[1] Wu, Yongliang, et al. \"Unlearning concepts in diffusion model via concept domain correction and concept preserving gradient.\n\n[2] Alberti, Silas, et al. \"Data unlearning in diffusion models.\" arXiv preprint arXiv:2503.01034", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "al2XYzu5xD", "reviewer": "Reviewer_vj4f", "summary": "The paper introduces CASteer, a training-free method for concept erasure in text-to-image diffusion models. Given an input prompt and a concept prompt, CASteer guides image generation to follow the input prompt while avoiding visual content associated with the concept prompt. The method computes steering vectors from the difference between cross-attention outputs when prompting the model with and without the concept. These vectors are applied at inference to steer generation in all the cross attention layers of the diffusion transformers. Experiments show that CASteer significantly outperforms prior baselines and is able to remove both abstract (e.g. nudity, inappropriate content) and concrete concepts (e.g. Mickey, Spongebob) and to generalize across models such as Stable Diffusion v1.4, SDXL, and SANA.", "questions": "1. How does CASteer perform when erasing multiple unrelated concepts or when steering vectors are different from each others?\n2. How critical is cosine-similarity weighting? Could fixed or learned scalars achieve similar results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The works presents a novel *training-free* approach to solve the problem of content erasure which improves over existing baselines both in term of performances and of practicality (no need to run an ad-hoc training). \n2. I find the idea of using steering vectors to *remove* concepts rather than *adding* them very interesting.\n3. The paper is well written and easy to follow.", "weaknesses": "I only have some minor concerns:\n\n1. The *extension to multiple concepts* (L272–L276) is not experimentally validated. It would be useful to assess performance as the number of erased concepts increases and/or their individual steering vectors substantially differ from each other.\n2. Equation 4 introduces *cosine-similarity weighting* between the text prompt and the erased concept; the paper does not analyze how crucial this weighting is in practice. An ablation could clarify whether simpler weighting performs comparably.\n3. Some *figures* could be improved. Figures 2 and 4 could benefit from clearer labeling of rows/columns and spacing can be increased among the sub captions of Figure 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jSi5f90n9J", "reviewer": "Reviewer_ezmt", "summary": "The paper proposed Cross-Attention Steering (CASteer), a training-free framework for concept erasure in diffusion models using steering vectors to influence hidden representations dynamically. Specifically, CASteer cleverly designs an algorithm for constructing steering vectors for new concepts and leverages these steering vectors to suppress unwanted image features without retraining. Extensive experiments demonstrate the effectiveness of the proposed method.", "questions": "See 'Weaknesses'.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The writing is fluent and logically coherent, exhibiting strong readability.\n* The proposed method is highly efficient, requiring no training or fine-tuning while achieving excellent performance.\n* The proposed method exhibits strong generalization and is applicable to various text-to-image models that incorporate cross-attention mechanisms.\n* Comprehensive qualitative and quantitative experiments demonstrate the effectiveness of the proposed method.", "weaknesses": "* As shown in Figure 1, CASteer computes a steering vector for the output of every cross-attention layer at every timestep in the generation process and applies a correction. Could this be somewhat excessive? Would it be possible to experimentally analyze whether the number of corrected timesteps and CA layers can be reduced to improve efficiency?\n* The reviewer examined the examples provided in the main paper as well as the prompt set in the appendix, and observed that the subjects used when constructing the prompts are relatively simple. This may lead to high variability in the generated outputs, which in turn could make the steering vector estimation unstable. For example, in Figure 1, when using “dog” as the subject, the model generates a brown dog as the positive sample and a white dog as the negative sample; this could cause the steering vector to incorrectly treat other attributes of the dog as part of the target concept. Would using more specific subjects yield a more stable and robust steering vector?\n* It would be very helpful to provide some positive visual examples of erased abstract concepts, such as “Van Gogh style.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ffvcgbRI0A", "reviewer": "Reviewer_dKH9", "summary": "The paper proposes CASteer, an activation steering method for concept erasure in diffusion models. It extracts steering vectors from cross-attention layers in the diffusion model for the concept to be erased and applies them during inference for erasure. Experiments are conducted for erasure of abstract concepts (nudity, violence) and objects (snoopy, mickey) on I2P benchmark and COCO dataset.", "questions": "See weaknesses above.\n\nHow are the multiple prompts generated?\n\nDo the steering vectors generated for SDv1.4 work for SD v1.5 across different checkpoints of the same model?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed activation steering method for concept unlearning is training-free. \n\nAdapting the method to novel concepts is straightforward and does not require retraining. \n\nExperiments on I2P benchmark for SDv1.4 model shows the effectiveness of the proposed method over a comprehensive set of prior methods.", "weaknesses": "The proposed method is not new as it applies the idea of activation steering which is well-established in LLMs to concept unlearning in diffusion models. So, the novelty is limited. \n\nThere is no theoretical justification on why operating in the latent space of diffusion models is better. It does not fully justify why unlearning in the cross-attention activation space (latent) is fundamentally better than guidance or prompt-based methods. \n\nIs it sensitive to the noise seed of the diffusion model? Are the steering vectors extracted for random noise seeds?\n\nHow does the method work if there is a mismatch between the number of steps used in steering vector extraction and during inference?\n\nMissing ablation study on steering with only one vector per CA layer in SDv1.4.\n\nConcept erasure based on guidance or steering is susceptible to adversarial attacks via concept addition or subtraction. Is this robust to adversarial attacks based on concept arithmetic [1] ?\n\nResults are only reported for few concepts (nudity, violence, snoopy, mickey). Missing erasure results on art styles (e.g., van gogh) or more abstract concepts (e.g., summer, mosaic style etc).\n\nComparisons are only reported for SDv1.4 model. Missing comparisons to prior methods on more recent models such as SDXL, SANA or SD3.5\n\nPaper can be better organized. Important ablations from supplementary need to be included in the main paper or the appendix instead of supplementary paper.\n\n[1] Petsiuk et. al. Concept Arithmetics for Circumventing Concept Inhibition in Diffusion Models, ECCV 2024\n\nMinor: \nReference to tables in main paper (e.g., table 15) is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)