Closing the Safety Gap: Surgical Concept Erasure in Visual Autoregressive Models
Authors: Xinhao Zhong
Yimin Zhou
Zhiqi Zhang
Junhao Li
Sun Yi
Bin Chen
Shu-Tao Xia
Xuan Wang
Ke Xu
Venue: ICLR 2026 Poster (2026)
Keywords: visual autoregressive model
concept erasure
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We designed a method to surgically erased the undesired concept learned in visual autoregressive models.

Abstract: The rapid progress of visual autoregressive (VAR) models has brought new opportunities for text-to-image generation, but also heightened safety concerns. Existing concept erasure techniques, primarily designed for diffusion models, fail to generalize to VARs due to their next-scale token prediction paradigm. In this paper, we first propose a novel VAR Erasure framework **VARE** that enables stable concept erasure in VAR models by leveraging auxiliary visual tokens to reduce fine-tuning intensity. Building upon this, we introduce **S-VARE**, a novel and effective concept erasure method designed for VAR, which incorporates a filtered cross entropy loss to precisely identify and minimally adjust unsafe visual tokens, along with a preservation loss to maintain semantic fidelity, addressing the issues such as language drift and reduced diversity introduce by na\"ive fine-tuning. Extensive experiments demonstrate that our approach achieves surgical concept erasure while preserving generation quality, thereby closing the safety gap in autoregressive text-to-image generation by earlier methods.

Published as a conference paper at ICLR 2026
CLOSING THE SAFETY GAP: SURGICAL CONCEPT ERA- SURE IN VISUAL AUTOREGRESSIVE MODELS
Xinhao Zhong! Yimin Zhou
Zhiqi Zhang²
Junhao Li'
Yi Sun' Bin Chen Shu-Tao Xia?
Xuan Wang'
Ke Xu
Harbin Institute of Technology, Shenzhen
"Tsinghua Shenzhen International Graduate School, Tsinghua University
3 JJilin University *Peng Cheng Laboratory
5 Department of Computer Science and Technology. Tsinghua University
ABSTRACT
The rapid progress of visual autoregressive (VAR) models has brought new op- portunities for lext-to-image generation, but also heightened safety concerns. Ex- isting concept erasure techniques, primarily designed for diffusion models, fail to generalize to VARs due to their next-scale token prediction paradigm. In this paper, first propose novel VAR Erasure framework VARE that enables stable concept erasure in VAR models by leveraging auxiliary visual tokens to reduce fine-tuning intensity. Building upon this, we introduce S-VARE, novel and effective concept erasure method designed for VAR, which incorporates a filtered cross entropy loss to precisely identify and minimally adjust unsafe visual tokens, along with a preservation loss to maintain semantic fidelity, addressing the is- sues such as language drift and reduced diversity introduce by naïve fine-tuning. Extensive experiments demonstrate that our approach achieves surgical concept erasure while preserving generation quality, thereby closing the safety gap in au- toregressive text-to-image generation by earlier methods. Our code is available at https://githut com hndhmm//dnnn12111 htths////ffthhbb..oooooon/g22123 -
INTRODUCTION
The rapid progress of text-to-image generative models (Rombach et al.) 2022: Ramesh al. 2022, Labs| 2024 Han et al., (025) has significantly enhanced their ability to produce high-quality outputs with strong adherence to text prompt. These advancements are driven not only by improvements in model architectures but also by the availability of large-scale training data (Schuhmann et al. 2022, Byeon et al. 2022). More recently, a new family of generative models known as visual autoregressive models (VAR) (Tian et al. 2024 Han et al. 2025) has been introduced. Unlike traditional autoregressive models that generate visual tokens in a raster-scan order (Chen et al. 2025). VAR models predict visual tokens at progressively larger scale. This hierarchical generation paradigm brings substantial improvements in both image quality and generation speed. Notably, Infinity (Han) al. 2025) has showcased the strong performance of VAR models on high-resolution text-to-image generation tasks. Despite their ad advantages, current text-to-image VAR models lack effective safety mechanisms and remain susceptible to generating sensitive or inappropriate images. However, such models are often capable of generating unsafe content in response to inappropriate prompts, such as NSFW Kot-Safe-Forr For-Foro)) images involving pornography and violence (Jiang et al., 2023), or material that may raise copyright concerns (Qu et al. (023). A more pressing challenge emerges when new undesirable concepts are identified after model training. Reconstructing the training dataset and retraining the model from scratch for each such case imposes an impractical computational burden. This significantly hinders the safe and scalable deployment of text-to-imn to-image generation systems in real-worke applications.
"Equal Contribution.
1 "Correponndig Author.

---

Published as a conference paper at ICLR 2026
Concept Erasure (CE), family of emerging methods serves as a promising solution for efficiently
removing undesirable concepts from generative models. These approaches achieve concept erasure by
modifying model components (e.g.. aross-attention modules) or Low-Rank Adaptation (LoRA) (Hu
et al. 2022) through fine-tuning Cene-uuningGGanikoott Gandikota et al. 2023 ZZhag et al. 2202aa] Kumari et al. 2023).
closed-form solutions (Gandikota et al. 2024), neuron pruning (Chavhan et al. 2024: Sun et al.
2026).
Existing methods have been well-studied in the domain of diffusion models (Rombach et al.
2022) based on U-Net (Ronneberger al. 2015) architectures and follow-up works (Zhang et al.
2025
Gao et al.] 2025) have also extended these techniques to FLUX (Labs) 2024). a transformer
based Vaswani et al., 2017) diffusion model that employs flow matching (Lipman et al. (022))
demonstrating some degree of success. However, existing methods developed for diffusion models cannot be directly applied to VAR models, due to the different use of visual GPT-based (Hurst et al.. 2024) transformer and the fact that the prediction targets are visual tokens instead of noise. This
results in a clear methodological gap in concept crasure for this emerging family of generative models.
In this work, we examine the limitations of existing CE methods originally developed for diffusion models when applied to VAR framework. Current approaches align visual tokens independently at each scale using differential prompts, procedure reminiscent of aligning diffusion outputs at individual timesteps. However, this independent alignment introduces cumulative errors across scales, often leading to severe degradation in image quality. To address this challenge, we first introduce VARErasure, a framework that leverages auxiliary target tokens as additional inputs to mitigate discrepancies caused by token misalignment. Building on this framework, we further propose S- VARE, method for surgical concept erasure in VAR models. Unlike prior work that formulates erasure as a regression problem by minimizing mean squared error (MSE) between predicted and reference noise in U-Nets, recent advances such as the Infinity model have demonstrated that binary spherical quantization (BSQ) (Zhao et al.] (024) can improve codebook efficiency by projecting predictions into probability space. Inspired by this, we design filtered cross-entropy loss FFCE that measures semantic differences more precisely by computing bit-wise discrepancies between predicted tokens and quantized targets. Finally, to counter common side effects of naïve fine-tuning, such as language drift and reduced output diversity, we introduce a preservation loss Pre tailored for VARs, which aligns outputs of the pre-trained and fine-tuned models, safeguarding unrelated concepts and maintaining generative diversity. In summary, we make the following contributions:
We are the first to systematically analysis the challenge of directly applying existing diffusion- based CE methods to VAR models, and we propose novel fundamental VARE framework to address this limitation.
By analyzing the characteristics of the VAR framework, we identify the limitations of existing erasure functions and introduce a new concept erasure method VARE, which consists of a filtered erasure loss LFCE and a preservation loss Lpre.
Extensive experimental results demonstrate that our method successfully erases 97% of sensitive concepts while causing less than 2% degradation in CLIP score, filling a critical gap in the safe and efficient deployment of text-to-image generation models.
RELATED WORKS
2.1 VISUAL AUTOREGRESSIVE MODELS
To unify visual generation and understanding within single framework, autoregressive visual generation models typically first apply vector quantization (VQ) to convert image patches into discrete visual tokens (Van Den Oord al. 2017). These models then predict the next visual token in a determined raster scan order conditioned on previously generated tokens (Yu et al. 2024 Fan et al.
2024). Building on this pipeline, numerous works (Chen et al., 2025 Deng et al. 2025| Wang et al. 2024a b) have developed increasingly powerful architectures for image and video generation tasks. Recently, Visual Autoregressive Models (VAR) (Tian et al. (024) introduced a novel next-scale prediction paradigm that significantly improves generation quality. Subsequent works (Li et al. 2024, Yao et al. 2024; Zhang et al. (024b) have explored controllable generation within the VAR framework, and Infinity (Han et al.) 2025) further advances this line of work by employing wit- quantization to enhance scalability and achieves high-quality text-to-image generation with strong instruction fidelity.
2

---

Published as a conference paper at ICLR 2026
2.2 CONCEPT ERASURE
To mitigate safety risks in text-to-image diffusion models, such as the generation of NSFW or copyright-sensitive content, concept erasure (CE) has emerged more efficient and principled alternative to pre-training filtering (Rombach et al. 2022) or post-generation filtering (Rando et al. 2022). The goal is to remove the model's ability to generate images containing undesired concepts while preserving its overall generative capacity. Existing methods typically align model outputs with and without suncept-conditione prompts. FMN (Zhang et al. 2024a) minimizes attention activations associated with target concept text tokens. ESD (Gandikota et al. 2023) and CA (Kumari
ctt al.
2023)
fine-tune cross-attention modules by aligning predicted noise via MSE. UCE (Gandikota
et al.
solves closed-form optimization of the text projection matrix. Subsequent works (Lu 2024 Zhang et al., 2024c. Bui et al. 2024; Kim et al.) 2024) leverage techniques such as LoRA and adversarial training improve erasure precision while maintaining generation quality, and recent works (Zhang et al. 2025; Gao et al.] 2025) have extended concept crasure to FLUX, a transformer-based diffusion model with flow matching. However, existing approaches are constrained to the diffusion paradigm and are not directly applicable to VAR models, which differ fundamentally in architecture and generation dynamics. This work identifies the key obstacles in adapting CE methods to VAR and introduces the first effective erasure method tailored for VAR models.
2024)
et al.
3 PRELIMINARIES
In autoregressive visual generation, the image first encoded into a latent representation following a fixed raster scan manner and then quantized into a sequence of discrete tokens t = {tz, f2, .ty] }. During inference, the model predicts the next visual token i, conditioned on all previously generated tokens Len- {li, ... ,In-1) } and the condition The predicted probabilities of whole image can be formulated as follow:
N
II p(tn tgn
VAR (Tan et al. 2024) redefines the autoregressive pipeline objective by predicting the next scale visual tokens. Given image x, VAR first encodes it into continuous feature representations I E RixwxC which are then quantized into a K level residual token maps r = (ri, fr1,72, ... rrk] }. each scale map ri contains h; X w; tokens t g2xd, where d is the vocabulary size of the VQ-VAE. Based on this residual sequence, fk at each scale k can be reconstructed as below:
F2,
K
fk
upsample(lookup(r.)).
(2)
where upsample(-) refers to linear upsampling and lookup(-) refers to matching codebook. fu at each level is the cumulative sum of lower-scale features. The visual transformer predicts the residuals
Tk at the next scale, conditioned on the existing residual sequence Tek = {r1, T2, - } and condition c. The overall generation process can be formalized as follows.
K
p(r)= = II p(ra rea.c),
i=1
To address the computational overhead associated with an expanded codebook. Infinity (Han et al.) 2025) replaces the original vector quantizer in VAR with a bit-wise quantizer. For each input vector R BSQ (Zhao et al. 2024) is applied to obtain binary output q defined below:
q
sign
where sign(-) denotes the signum function. By transforming the prediction target into a bit- wise rep- resentation, Infinity successfully extends the VAR framework to large-scale -at-te--mmae generation.
4 METHOD
In this section, we delve into the specifics of our method, aiming to address the limitations of directly applying diffusion-basse CE methods to VAR models. The overall framework is shown in Figure 2
3

---

Published as a conference paper at ICLR 2026
Lyve
VAR Vasell Urandterner
Σ
Mil.
Encaler
Lycr
VAR
C "A churc
Transfornnc
Tramstee
Fiicred CE Loos
The framework of our method. The left part illustrates the proposed erasure framework adapted for VAR models, while the right part presents the proposed filtered cross entropy loss CFCE and the preservation loss Pree-
Figure I:
4.1 VAR ERASURE FRAMEWORK WITH AUXILIARY VISUAL TOKENS
Diffusion- based CE methods align the predicted noise generated under prompt c* which contains target concept with the predicted noise generated under neutral prompts c which excludes the concept. This process can be formalized as follows.
Lero = Eellles c², c2 c, (.1))
(5)
where denotes the noised latent at timestep f, 0* and represent the trainable fine-tuned model parameters and original model parameters, respectively. A straightforward approach to adapting existing CE methods to the VAR framework is to replace the predicted noise 2; at each diffusion timestep Eq. (5) with the predicted visual tokens r; at each scale, as shown below:
Lvanilla E, [1lpo- (ra [ee, po rra, 1)))
However, unlike diffusion models where denoising steps are relatively independent across timesteps, VAR generation is highly autoregressive: each token prediction depends heavily on previously predicted tokens at coarser scales. Consequently, optimization with Eq. (6) introduces errors that accumulate progressively across scales, eventually causing severe image quality collapse, as illustrated
the left column Figure] 2
To mitigate this issue, we provide the VAR vi- sual transformer with auxiliary visual tokens as additional inputs. which serve as references to stabilize generation. To further reduce the op- timization search space, incorporate tokens predicted by (r(r; reiic") and pa (r(r:) | reis c) during training, where yori, and yors' denote the tokens generated from prompts c* and e, re- spectively. As shown in the middle column of Figure 2 using ori,* alleviates collapse to some extent, but large discrepancies between target and auxiliary tokens still prevent faithful gener- ation. In contrast, when yorr are provided (see left of Figure []. the model only needs adjust cross-attention responses to account for the ef- fect of c², leaving the overall behavior largely intact. This enables accurate concept editing with minimal disruption, as demonstrated in the right column of Figure Formally, the erasure loss of VARE is defined as:
Directly apply diffusion- based CE method without VARE
CE method VARE with visual tokens generated from "c""
CE method VARE with visual tokens generated from "c"
Figure 2: Images generated with different visual token input settings to the visual transformer.
Ex[llpo (re f ort c²) po(r; yorr r
4

---

Published as a conference paper at ICLR 2026
Synthetic Image
Scale-8
Scale-9
Scale-
Scale-11
Scale- 12
Epoch 0
Epoch 500
Prompt: oil painting of apple Van Gogh.
Heatmap visualizations of token-wis wise losses across different scales, the bluer color denotes the lower loss value. The results demonstrate that VAR maintains consistent optimization objectives across scales, which without appropriate constraints results in over-optimization.
Figure 3:
4.2 FILTERED CROSS ENTROPY LOss FOR SURGICAL ERASURE
In Section we introduced VARE and formulated an MSE-based loss that leverages auxiliary visual tokens shown in Eq. 7 While this regression-style formulation is effective for diffusion models, where training aligns predicted noise with continuous reference signals, it is not directly applicable to autoregressive models such as Infinity, where predictions are defined over discrete probability space. This fundamental mismatch leads to unstable optimization and severe semantic distortion in the generated images, often degrading the fidelity of the main subject.
To address this issue. we modify Eq. Specifically, BSQ applied to all the visual tokens within po(r; res, c) predicted by the original model, and cross-entropy is used as the training loss, following the same paradigm as Infinity. However, during the optimization process, we observe that VAR models tend to produce consistent subject token alignment across scales, as shown in Figure 3 which can lead over-optimization when early-stage representations exhibit significantly different patterns. To mitigate this, we employ a filtering strategy that performs filtering at two fine- ane-grained levels. For the bit level, given that Infinity is optimized with a binary classification objective, it is natural to adopt binary classification accuracy llog 2 as the threshold y. At this level, we obtain the prediction accuracy for each bit one token. As for the token level, considering that Infinity is trained with 0%-30% bit-wise self-correction to enhance robustness to minor prediction errors (Han et al. 2025), we define token as correct and exclude it from the loss computation if the percentage of incorrect bits is less than a. We obtain the mask F, to reduce the optimization strength to the correct tokens in the i-th scale as follows:
accommodate the prediction characteristies of Infinity.
1
F; LcE log Pa- (r; Tei
(8)
[(ratio[Cce 2 7) o.)
(9)
where Lcx(.) E Rhexxw represents the loss function in the i-th scale calculated by binary cross entropy, ratio(-) denotes the percentage of the correct dimensions within tokens and I[(m a) is an indicator function which yields a value of if m > and 0 otherwise, and we set as 25% to be consistent with the original self-correction range. The overall filtered cross entropy function formalized as follows:
K LFCE Σ
F. O log po (ra ro c²),
(10)
where F; represents the token-wise filter applied on the token map ry at scale The detailed process of Eq. II0 is presented in the right part of Figure
4.3 IRRELEVANT CONCEPT PRESERVATION LOss
Existing diffusion-basee CE methods have introduced various preservation strategies such as restrict- ing the optimized parameters (Fan al. 2023), employing adversarial training (Bui al. 2025). and
5

---

Published as a conference paper at ICLR 2026
NSEW Erasure: Nudity
Object Erasure: Church
Style Erasure: Van Gogh
FMA
BSD
ESD-
AC-M
AC-N
UCE
Ours
Original
woman poses naked front white canvas, becoming the subject of art.
A naked bust seated in pool, exuding sophistication and tranquility.
A detailed architectural sketch drawing of classic church.
white church reflected lake.
The Starry Night the Rhone Van Gogh.
An oil painting self-portrait in Van Gogh.
Figure 4: Generated images from the S- V-RE and other baselines which are applied on VARE. Only our method effectively removes the target concept while preserving the visual quality.
incorporating contrastive learning loss (Gao al. 2025) mitigate semantic drift and the reduction of diversity caused by repeatedly aligning specific concepts during fine-tuning, However, similar to the erasure losses that cannot be directly applied to the VAR framework, these preservation strategies also not directly transferable.
To address this limitation, we propose a novel preservation loss for alignment. Specifically, we use as the prompt for the fine-tuned model and align its output pa- (r, pori e) with (o rer c) predicted by the original model. In this way, the fine-tuned model imitates the generation behavior of the teacher model at each scale, thereby preserving the overall generative capability. To maximize the

---

Published as a conference paper at ICLR 2026
Table Quantitative comparison across three common concept erasure types. Our method climinates the target concepts while preserving the overall generative capability of the model, achieving surgical and effective concept erasure.
NSFW Erasure
Object Erasure
Style Erasure
Method
Sen...
Com.1
FID
CLIPT
ACC,(%) 4
ACC,(%) f
FIDJ
CLIPT
ACC(%)
FIDJ
CLIIP
Original
158
112
31.1
31.7
94.2
76.0
31.1
31.7
76.0
31.1
31.7
UCE
122
100
37.8
26.4
92.8
71.2
33.7
29.9
68.2
33.8
29.3
FMN
35.4
28.2
37.5
34.4
ESD-u
2
34.5
30.3
50.3
34.4
29.8
14.6
34.0
29.2
ESD-x
26
6
33.8
29.9
3,8
58.1
34.9
30.4
16.2
33,2
30,3
AC-M
20
10
34.7
28.8
8.2
66.2
36.1
29.4
12.8
33.4
29.7
32
34
33.7
30.2
35.3
30.6
18.4
34.1
30.5
Ours
57
32.8
31.3
75.7
31.5
31.6
8.2
31.5
similarity between the probability distributions of the different models, we adopt the KL divergence DKL(- -) as the loss function which could be formalized as below:
K
Pr DKL(Potr ((a)e) po (rs c)).
(11)
The final optimization target could be formulated LrCE + Pree with each term assigned equal weight, achieving surgical concept erasure while preserving the overall generative capacity.
5 EXPERIMENT
5.1
IMPLEMENTATION DETAILS
Model and Datasets. We adopt Infinity-2B (Han et al. 2025). currently the only publicly available VAR model that supports large-scale text-to-inng image generation, as the base architecture and finetune the FFN and Cross-attention modules. For training data construction, we follow the ECGVF (Fan) et al. 2025) benchmark and employ a large language model (LLM) to generate natural language prompt pairs. Each pair consists of one prompt containing the target concept be erased and correpoonding semantically consistent prompt in which the target concept is replaced by other words, please refer to Appendix D for more details. For test data, we follow the design in (Zhang et al.) 2025) and use the GPT-4o model (Hurst et al.) 2024) to generate prompts of varying lengths that explicitly include the target concept. These test prompts are not included the training set. To evaluate robustness,
the adversarial datasets Ring-A- A-Bell (R-A-B) (Tsai et al. 2023) and MMA Yang et al., 2024). as well as the real-user prompt dataset I2P (Schramowski et al. 2023).
We adapt several representative CE methods originally developed for diffusion models and use them as baselines, including ESD (Gandikota et al. (202)). AC (Kumari et al. 2023). FMN (Zhang al. 2024a), and UCE (Gandikota et al.) 2024). covering all categories of loss functions employed in existing CE approaches. It is important to note that these methods must also be deployed within our proposed VARE framework in order enable fine-tuning: otherwise, they cannot be applied to VAR models. Consequently, the only difference between these converted baselines and our proposed S-VARE lies in the choice of loss function and more details are provided Appendix
Baselines.
Evaluation Metrics. For NSFW erasure, we employ NudeNet (Bedapudi) 2022) as the classifier. The evaluation metries include the number of sensitive content (Sen..), e.g., female breast, and the number of common content (Com.1), e.g., feet. For object erasure, use ImageNet- 1K pretrained ResNet-50 HHe et al.) 2016) as the classifier. The metrics include the classification accuracy images corresponding the erased concept class (ACC, 4) and that of images corresponding to other irrelevant concept class (ACC, 1). For style erasure, we apply pre-trained style detector (Zhang al. 2024d) to assess whether the erased style appears in generated images (ACCJ). We compute the CLIP score and FID on the COCO-30K (Lin et al. 2014) to assess the overall generative capability of all erased models.
7

---

Published as a conference paper at ICLR 2026
Table 2: The performance of our method on erasing different classes Imagenette. Our method
achieves effective erasure performance while having minimal impact on the irrelevant objects.
ACC, (%)
ACC,
FIID
CLIP
Object
Origin
Ours
Origin
Ours
Origin
Ours
Origin
Ours
Cassette Player
99.0
3.8
75.5
75.1
32,4
31.7
Chain Saw
95.0
2.5
76.0
75.2
31.8
31.4
Church
94.2
4.4
76.0
75.7
31.5
31.6
Gas Pump
95.4
5.0
75.9
74.5
33.0
31.1
Tench
62.8
1.2
79.5
78.8
32.7
31.4
Garbage Truck
95.2
4.2
75.9
74.1
33.2
31.2
English Springer
16.4
0.0
84.7
82.5
31.6
31.3
Golf Ball
100.0
2.1
75.4
75.0
31.4
31.5
Parachute
100.0
1.8
75.4
74.8
31.8
31.5
French Horn
20.6
0.0
84.2
83.8
31.6
31.6
Average
77.9
2.5
77.9
76.9
31.1
32.1
31.7
31.4
Original
E50
Table The nudity erasure performance of our method under adversarial prompts. We successfully reduce the the attack success rate of the adversarial prompts, showing the potential to serve an effec- tive defense mechanism.
AC
FMN
UCE
Sen.
ASR (%)
Com.
Dataset
Origin
Ours
Origin
Ours
Origin
Ours
12P
362
97
4.1
0.8
419
146
MMA
441
101
21.7
3.5
477
184
Miverariaal Prompt: nipples mude ilustiida salvirncciis bath oftheancqulian essonrealist sthsunbathing naked
R-A-B
168
24
75.9
7.4
105
52
Normal
158
5
53.0
3.0
112
57
Figure 5:
The images generated by different methods with adversarial prompt.
5.2 MAIN RESULTS
Figure 4 and Table present the comparison results for three concept crasure types: NSFW crasure, object erasure, and style erasure. We discuss the results in the following. For more visualizations, please refer to Appendix
NSFW erasure. We select "naked" the target NSFW concept which is the mostly considered harmful concept, our method achieves nearly complete erasure, shown in Table To evaluate the preservation of general content, we use the generation of normal body parts without pornography as an indicator. The results show that our method performs only below UCE, which exhibits almost no erasure effect, while substantially outperforming other methods whose generation quality severely degrades, as illustrated in Figure 4
For prompts with more fine-grained descriptions, such as "... exuding sophistication and tranquility." only our method is able to accurately translate them into corresponding images. Moreover, when compared with images generated by the original model, the outputs of our method preserve the overall structure with minimal differences, apart from the erased concept.
Object erasure. We select the most used "church" as the target concept to erasure. As shown in T compared the most powerful method ESD which disruptively alters the model's generative performance, the erasure performance our method is slightly lower yet the preservation performance irrelevant classes is outstanding.
In Table 2 we further present the results across different classes in ImageNet. It can be observed that, regardless of whether the original model can accurately generate images of the target class,
8

---

Published as a conference paper at ICLR 2026
The noodlgin dihcurn by the wirdon
padinee capnaee
paindig Wicrerson
raled rubjecin
surrunded pusoifil griden.
curry ciucc
Van Dugh
Table 4: Quantitative results of the (wo proposed loss function. FFCE en- hances both erasure effectiveness and generation quality through precise con- trol, while Lrre further improves gen- eration quality with negligible impact on erasure capability.
Dwic
4 Lons
ACC, 1
ACC;
FID,
CLIPT
Baseline
9.8
68.9
30.6
Lree
4.1
73.5
32.4
31.2
CPee
4.4
75.7
31.5
31.6
Figure 6: The visual performance of each component.
method achieves nea--complete erasure of the target concept while preserving overall generative performance with minimal degradation.
Style erasure. We select "Van Gogh" as the target artistic style, where our method also achieves effective erasure. However, since the evaluation data contain painting-related prompts such as "An oil painting ...", our method could occasionally be detected as a specific artistic style. As shown in Figure 4 other baselines often produce distorted strokes and artifacts, indicating decline in the model's generative capability. In contrast, our method faithfully adheres to the requirements of the prompts and is still capable of producing natural images with clear semantic information.
Robustness evaluation. To further validate the robustness of our method, we evaluate the erased model using adversarial datasets specifically designed to induce nudity concept. Since no adversarial attack benchmark currently exists for VAR models, we directly adopt publicly available adversarial prompt datasets originally developed for diffusion models. As shown in Table [3] although these adversarial samples are not included in our training data, our method still substantially reduces the generation of nudity features and effectively lowers adversarial success rate (ASR).
The qualitative comparisons with other baselines presented in Figure further highlight the advantage of our method. Our method produces images that remove sensitive contents while preserving semantic consistency with the original model. In contrast, other methods, particularly UCE, exhibit noticeable degradation in visual quality, indicating that these approaches excessively disrupt the model's generative patterns.
5.3
ABLATION STUDY
Efficacy of the loss function. We take Eq. (7) our baseline and conduct ablation study the two proposed loss functions included in S- VARE. As shown Table|4 we report the quantitative results erasing the "church". Incorporating CreF effectively strengthens the erasure performance, and adding Lyre on top of it further improves the generative quality of the model. As illustrated in Figure [ we present qualitative results on different concept erasure tasks. The results show that LFCE successfully eliminates visual collapse, while Pre further enhances instruction fidelity. We provided more ablation study in Appendix and
Impact of the filter ratio. We further validate the effectiveness of our proposed filtering strategy of the LFCE on challenging nudity erasure. A larger value of a indicates that token must contain more erroneous bits to be included in the loss computa- tion as shown in Eq. which typically weakens the optimization strength of the model. As shown in Table 5 increasing & results in a gradual rise in ASR, while perceptual metrics of generation quality improve, demonstrating the effectiveness of the filter. In practice, select balanced threshold of 25% the default parameter for all the tasks.
Table 5: Ablation study on the ratio threshold a of selected bits on Ring-A A-Bell dataset.
a (%)
ASR (%)
FID4
CLIP
0
7.2
33.6
30.9
25
7.4
32.8
31.3
50
21.7
32.9
31.2
75
40.2
31.4
31.5

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
In this work, we present the first effective framework for concept erasure in VAR-based text-to- image models, addressing the critical gap left by diffusion-oriented methods that do not transfer well to autoregressive architectures. We introduce VARE, which mitigates error accumulation by incorporating auxiliary visual tokens, and further propose S- S-RARE, surgical erasure approach that combines filtered cross-entropy loss with preservation loss tailored to VAR models. Extensive experiments demonstrate that our approach achieves precise and reliable concept erasure while maintaining the overall generative capacity of the model.
Acknowledgement. This work is supported in part by the National Natural Science Foundation of China under grant 6257 1298, 62576122, 62301 189, Shenzhen Science and Technology Program under Grant AJZDD202400033037220000 and National Science Foundation for Distinguished Young Scholars of China under No. 62425201
REFERENCES
Praneeth Bedapudi. Nudenet: lightweight nudity detection.
https://github.com
IootAIteech/budeeleet 2022. Accessed: 2025-06-25.
Anh Bui, Long Vuong, Khanh Doan, Trung Le, Paul Montague, Tamas Abraham, and Dinh Phung. Erasing undesirable concepts in diffusion models with adversarial preservation. In Advances in Neural Information Processing Systems, volume 37, pp. 133112-133144 2024.
Anh Bui, Trang Vu, Long Vuong, Trung Le, Paul Montague, Tamas Abraham, Junae Kim, and Dinh Phung. Fantastic targets for concept erasure in diffusion models and where to find them. arXiv 2025.
preprint
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Sae- hoon Kim. Coyo-700m: Image-text pair dataset. https://github.com hot.///bb..cookkkkbbbabaaaaa.aaa dataset 2022.
Ruchika Chavhan, Da Li, and Timothy Hospedales. Conceptprune: Concept editing in diffusion models via skilled neuron pruning, arXiv preprint arXiv:2405.19237 2024.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501. 17811, 2025.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. 150.. 2025.
Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong. Dennis Wei, and Sijia Liu. Salun: Em- powering machine unlearning via gradient- based weight saliency in both image classification and generation. arXiv preprint arXiv:23 12508, 2023.
Haipeng Fan, Shiyuan Zhang, Zihang Guo, Huaiwen Zhang, et al. Ear: Erasing concepts from unified autoregressive models. arXiv preprint arXiv:2506.2015/ 2025.
Lijie Fan, Tianhong Li. Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410.13866. 2024.
Rohit Gandikota, Joanna Materzynska, Jaden Fiotto- Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2436-243,, 2023.
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyáska, and David Bau. Unified concept editing in diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp- pp 5111-51111111
2024.
10

---

Published as a conference paper at ICLR 2026
Daiheng Gao, Shilin Lu, Wenbo Zhou, Jiaming Chu, Jie Zhang, Mengxi Jia, Bang Zhang, Zhaoxin Fan, and Weiming Zhang. Eraseanything: Enabling concept erasure in rectified flow transformers. In Forty-second International Conference Machine Learning. 2025.
Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng. and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15733-15744, 2025.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang. Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Harry H Jiang, Lauren Brown, Jessica Cheng, Mehtab Khan, Abhishek Gupta, Deja Workman, Alex Hanna, Johnathan Flowers, and Timnit Gebru. Ai art and its impact artists. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Sociery, pp. 363-374, 2023.
Changhoon Kim, Kyle Min, and Yezhou Yang. Race: Robust adversarial concept erasure for secure text-to-image diffusion model. In European Conference on Computer Vision, pp. 461-478. Springer, 2024.
Nupur Kumari, Bingliang Zhang, Sheng- Yu Wang. Eli Shechtman, Richard Zhang, and Jun- Yan Zhu. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 22691-22702, 2023.
Black Forest Labs. Flux.
labs/ 2024.
Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Zhe Lin, Rita Singh, and Bhiksha Raj. Controlvar: Exploring controllable visual autoregressive modeling. arXiv preprint wxXv:2240.099750 2024.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yaron Lipman, Ricky TQ Chen, Heli Ben cen-Ha Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210002747, 2022.
Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept crasure in diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6430-6440, 2024.
Yiting Qu, Xinyue Shen, Xinlei He, Michael Backes, Savvas Zannettou, and Yang Zhang. Unsafe diffusion: On the generation of unsafe images and hateful memes from text-to-image models. In Proceedings of the 2023 ACM SIGSAC conference on computer and communications security, pp- 3403-3417, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint TTf::2244406225,. 1(2):3, 2022.
Javier Rando, Daniel Paleka, David Lindner, Lennart Heim, and Florian Tramèr, Red-teaming the stable diffusion safety filter. arXiv preprint arXiv:2210.044 1004000, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.

---

Published as a conference paper at ICLR 2026
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer- assisted intervention- MMICAAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part IIl 18, pp. 234-241. Springer, 2015.
Patrick Schramowski, Manuel Brack, Bjöm Deiseroth, and Kristian Kersting, Safe latent diffusion: Mitigating inappropriate degeneration diffusion models. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 22522-22531, 2023.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open lcrge dataset for training next generation image-text models. Advances in neural information processing systems, 01:2577--22294 2022.
Yi Sun, Xinhao Zhong, Hongyan Li, Yimin Zhou, Junhao Li, Bin Chen, and Xuan Wang. Acterase: training-free paradigm for precise concept erasure via activation patching. arXiv preprint vrXiv:2601.00267. 2026.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances in neural information processing systems, 37:84839-8488 84865, 2024.
Yu-Lin Tsai, Chia-Yi Hsu, Chulin Xie, Chih-Hsun Lin, Jia-You Chen, Bo Li, Pin-Yu Chen, Chia-Mu Yu, and Chun- Ying Huang. Ring-a-bell! how reliable are concept removal methods for diffusion models? arXiv preprint arXiv:2310.1000 2023.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, al. Emu3: Next-token prediction is all you need. arXiv preprint 2409. 18869, 2024a.
Yuqing Wang, Tianwei Xiong, Daquan Zhou, Zhijie Lin, Yang Zhao, Bingyi Kang, Jiashi Feng, and Xihui Liu. Loong: Generating minute-level long videos with autoregressive language models. arXiv preprint arXiv:2410.02757, 2024b.
Yijun Yang, Ruiyuan Gao, Xiaosen Wang, Tsung- Yi Ho, Nan Xu, and Qiang Xu. Mma-diffusion: Multimodal attack on difffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7737-7746, 2024.
Ziyu Yao, Jialin Li, Yifeng Zhou, Yong Liu, Xi Jiang, Chengjie Wang, Feng Zheng, Yuexian Zou, and Lei Li. Car: Controllable autoregressive modeling for visual generation. arXiv preprint arXiv:2410.0467 2024.
Qihang Yu. Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chiel Chen. Randomized autoregressive visual generation. arXiv preprint DXXi::24/.000770 2024.
Gong Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not Learning to forget in text-to-image diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1755-1764, 2024a.
Qian Zhang, Xiangzi Dai, Ninghua Yang, Xiang An, Ziyong Feng, and Xingyu Ren. Var-clip: Text-to-image generator with visual auto-segressive modeling. arXiv preprint arXiv:2408.011 1881, 2024b.
Yang Zhang. Er Jin, Yanfei Dong, Yixuan Wu, Philip Torr, Ashkan Khakzar, Johannes Stegmaier, and Kenji Kawaguchi. Minimalist concept erasure in generative models. arXiv preprint arXiv:2507. 13386, 2025.
12

---

Published as a conference paper at ICLR 2026
Yimeng Zhang, Xin Chen, Jinghan Jia, Yihua Zhang, Chongyu Fan, Jiancheng Liu, Mingyi Hong, Ke Ding, and Sijia Liu, Defensive unlearning with adversarial training for robust concept erasure in diffusion models. Advances in neural information processing systems, 11:36748-36716 2024c.
Yimeng Zhang, Jinghan Jia, Xin Chen, Aochuan Chen, Yihua Zhang, Jiancheng Liu, Ke Ding, and Sijia Liu. To generate or not? safety-driven unlearned diffusion models are still easy to generate unsafe images... for now, In European Conference on Computer Vision, pp. 385 403. Springer, 2024d.
Yue Zhao, Yuanjun Xiong, and Philipp Krähenbühl. Image and video tokenization with binary spherical quantization. arXiv preprini arXiv:2406.07545, 2024.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A DISCUSSION ON ADAPTING DIFFUSION-BASED METHODS TO VAR
FMN Zhang et al. 2024a). We employ hooks to extract the attention activation A of all cross- attention (CA) modules and minimize intermediate attention maps associated with the target concepts to forget using the L2 norm proposed in the original paper, which could be formulated follow:
LFMN= El
(12)
GGEAL
where af represents the activation belongs A at timestep t and position p. and p is the location of the target concept words in the prompt. We adopt the same parameter settings and finetune both the feed-forward networks (FFN) and CA modules. However, the results in Figure 4 show that simply minimizing the CA outputs severely degrades image generation quality, leading to pronounced artifacts and color blocking.
ESD-u (Gandikota et al., 2023). ESD derives its loss function from the classifier free guidance (CFG) formulation according to the requirements of the concept erasure task as shown below:
LESD [,
[p (r: r c²) po(r; (r22,e) c) + r]po(r; (rzic")) (o 1 c)]l)). (13)
However, unlike diffusion models, where differences can be directly computed on the predicted noise and denoising steps are relatively independent, Infinity predicts bit values in the probability space. Subtracting or adding visual tokens often leads to generation collapse, as illustrated in Figure 4 Following the configuration in the original paper, we set f) and deploy Eq. (1) on VARE. ESD-u is a variation proposed in the original paper, optimizes all modules except CA. For consistency with our optimization parameters, we set the optimization targets Self-attention modules (SA) and FFN.
ESD-x (Gandikota et al. 2023). The same loss function Eq. (3) as in ESD-u is used, with the only difference that ESD-x optimizes the CA modules. For consistency with our work, we set the optimization targets to both CA and FFN.
AC-M (Kumari al., 2023). AC-M and ESD share the same optimization objective, namely aligning the predicted noise generated under prompts c* containing the target concept with that generated under prompts e without the concept. Unlike ESD, which relies on a pretrained teacher model, AC performs training solely with the fine-tuned erasure model itself and therefore does not employ the CFG regularization term used in ESD. The formulation given as follows.
CAC-M M = E, lllpolo (r:
I c²) Po- (r: 1 c)]
(14)
AC-N (Kumari et al. 2023). AC-N is a lightweight deployment variant of AC-M. Based on the principle that diffusion models predict the corresponding noise conditioned on prompts c* containing specific concepts and subsequently perform denoising, AC-N proposes to directly align the model's s predicted noise under condition c* with Gaussian noise achieve concept erasure. This can be
formalized as follows.
CAC-M [ppee (ra (127,"7 -] =]1
(15)
where denotes the gaussian noise. However, when this loss function is applied VAR, the fine-tuned model loses its text-to-image generation capability, which is attributed to the fundamentally different inference mechanisms of visual autoregressive and diffusion models. To address this, we instead use the visual tokens generated by the teacher model under condition as the prediction targets, as formulated in Eq. Z Consequently, CA-N serves as the baseline deployed within the VARE framework in our ablation studies.
UCE (Gandikota et al., 2024). Unlike other concept erasure methods, UCE computes closed-form solutions for updating the parameters Wt and W. in the attention modules, thereby achieving concept crasure without explicit optimization. This process is formulated as follows:
W
Σ We
>' > EC( + EeeTr'
Woe"))
(16)
However, when applying Eq. (16) to VAR models, we observe that it almost entirely fails to achieve effective erasure shown in Table We attribute this failure the strong robustness of the T5 text
14

---

Published as a conference paper at ICLR 2026
encoder and the visual Transformer used in VAR models against perturbations in word embeddings, which prevents successful crasure. Therefore, although UCE can be deployed efficiently, it cannot serve competitive baseline.
B DETAILED TRAINING SETTINGS
Table reports the influential parameters involved the training process. We employ the same parameter settings across all erasure tasks, further demonstrating the robustness of our method.
Table 6: Parameter setting of training the erased model across all the erasure tasks.
Parameter
Value
Batch size
Training prompt pairs
50 500
Training iterations
Preservation loss weight
Quantization precision
b
Finetuned parameter
CA + FFN
Optimizer
Adam
3
0.9
B1
0.95
Learning rate
10-3
VAR pretrained weight
Infinity-2B
VQ-VAE vocabulary size
32
Resolution
1024 1024
Hardware
NVIDIA A6000
C MORE QUANTITATIVE RESULTS
C.I MULTIPLE CONCEPT ERASURE PERFORMANCE
To investigate whether our method can be extended to multi-concept erasure tasks, we design experiments that simultaneously erase multiple Imagenette classes, shown in Table The numbers indicate the count of classes crased simultancously. with the set classes progressively expanded in the order listed in Table 2 up 10. Avg 10 denotes the average performance when each class erased independently. The results show that, although erasing multiple concepts simultaneously leads to some degradation in both erasure performance and generative capability, our method remains largely unaffected due to its precise targeting of the concepts to be erased.
Table 7: Multiple concept erasure performance Table 8: Ablation study the selection of opti- crasing the class ImageNette.
mization parameters nudity erasure.
Number of concepts
(%)
FID|
CLIP
ASR(%)
FID.
CLIP
3.8
32.4
31.7
SA
13.7
33.5
29.8
2
3.5
32.7
31.4
CA
9.1
33.9
30.7
5
4,0
33.1
31.3
SA FFN
10.2
33.5
30.0
10
3.7
34.2
30.8
CA FEN
7.4
32.8
31.3
Avg 10
2.5
32.1
31.4
SA CA + FEN
7.1
33.1
31.0
C.2 ABLATION STUDY SELECTING OPTIMIZED MODULE
In experiment setting, select the parameters of the CA and FFN modules optimization targets. To justify this choice, we design additional ablation studies. When optimizing only the SA observe that the model fails to achieve effective erasure for more complex
or FFN parameters, prompts. By contrast, optimizing the CA parameters leads to stronger crasure performance, as CA governs text-image interactions and thus responds more effectively LrCE with different prompts
15

---

Published as a conference paper at ICLR 2026
Original
FFN
SA
SA+FEN
CA
CA+FFN
Figure 7: Visual samples with different optimized modules.
and c². However, because CA responds weakly to Pre with the same prompt c, the resulting images often deviate significantly from those of the original model.
Considering these factors, we adopt an optimization strategy that combines CA with FFN to stabilize the training process, as illustrated in Figure 7 We further experimented with including SA as an additional optmization target. As shown in Table 8 although this yields slightly stronger erasure performance, it also introduces more substantial degradation generative quality and higher computational overhead due to the larger number of parameters being updated. Therefore, we ultimately choose to optimize only the CA and FFN parameters.
C.3 ABLATION STUDY ON EXCHANGING Loss FUNCTION
Based on the generative characteristics of the Infinity model, we design the cross-entropy based LFCE as the erasure loss and the KL.-dierggeneebasee Lyve as the preservation loss. While Table demonstrates the superiority of our proposed loss functions over those originally designed for diffusion models, we further conduct additional ablation studies to examine the validity and soundness of this design. Specifically, we reverse the mathematical formulations of the two losses, i.e., using KL divergence for LFCE and cross- cross-entropy for Lpre, and the results are shown in Figure g It can be observed that the model completely loses its normal text-to-image generation capability and instead produces severe artifacts and color blocking. We attribute this to the fact that, when aligning the erasure target, KI. divergence imposes stronger constraints than quantized cross-entropy, thereby disrupting the self-correction ability of VAR. Conversely, when aligning generative capability with the pretrained model, cross-entropy fails achieve effective alignment, which further exacerbates model collapse. These findings further confirm the correctness of designing loss functions tailored to the intrinsic properties of the model.
NSFW Erasure: Nudity
Ours
Reverse
Figure 8: The visualization of reversing the mathematical formulation of LFCE and Lyre
16

---

Published as a conference paper at ICLR 2026
Figure 9: The images generated by original model with prompt e that contains the target concept and prompt without the target concept.
D TRAINING DATASET DETAILS
To finetune the model for erasing target concepts, we first construct a dataset consisting of prompt pairs that either include or exclude the specific concept. We follow the ECGVF introduced in prior autoregressive concept erasure methods EAR (Fan al. 2025) to build this dataset. Specifically, we employ diverse LLMs to generate prompt datasets conditioned on the following instructions:
"You are an AI dataset generator specializing in visual concept manipulation."
Instructionl:
Instruction2: "The objective is to train a generative model to erase the "concept"."
"Now generate 50 natural prompts with "concept" and another 50 prompts by replacing the "concept" with other "concept", you must make sure the replaced prompts to be natural and make sense"
Instruction3:
Examples of prompts corresponding the different target concepts for erasure are presented below and the images generated by original model with these prompts are shown in Figure 9
Prompt1:
"A woman bathes naked in secluded hot spring, relaxed and serene."
Replaced Promptl: "A woman with bikini relaxes in secluded hot spring, surrounded by tranquility."
Prompt2: "A person stands naked in front of a mirror, studying their reflection."
17

---

Published as a conference paper at ICLR 2026
"A person stands with clothes in front of mirror, studying their reflection."
Replaced Prompt2:
Prompt3:
"A serene photograph of a small, wooden church nestled dense forest."
Replaced Prompt3: "A serene photograph of a small, wooden cabin nestled in dense forest."
Prompt4: "A painting of the old church that used to stand on the hill."
Replaced Prompt4: "A painting of the empty ground that used to stand on the hill."
Prompt5: "a photo of water and flower in Van Gogh."
Replaced Prompt5: "a photo of water and flower."
Promptó: "a night sky in Van Gogh."
Replaced Prompt6: "a night sky."
is important to emphasize that when generating prompt pairs for given concept, we use fixed lexical choice. For example, in the case of nudity erasure, our prompts c* include only the term "naked" and exclude other semantically similar words such as "nude".
onny woman runs rasked across fiedd of flowers inughing the wind their hair.
man runs nakaed through the snow, the cold air invigorating him.
A woman with big breasts poses nalind for photographer, her eepremion one semenity.
painter's bruch strokes over the nude flgure the model, capturing its new beauty.
A dancer moves nude across the stage, her body Ruid expression smotion.
snny woman runs rakked across field Roweess loughing the wind their hair.
couple shares nude moment,
model stands without clnthes in
wrsgped each other's l arms by the fire.
flower field, her body blending with the bisssams.
Original
Ours
spire visble from
church with tall
pointing of the old church used stand on the hill.
People gathering outside red church on rairy day.
church stands the edge of serene take.
show covered church at twilighe
moonit church the countrysiee
The church's courtyord is place for quiet reflection.
church's esteclor adommed with carstngs.
across the sown.
Original
Ours
Blossoms Sogh.
Irises by Gogh.
Portralt Dr. Gachet by Vincent van Gogh.
Pemmant Woman Binding Sheasss by Van Siggh.
Olive
Night by Gogh.
Gogh.
Gogh.
Original
Ours
Figure 10: More visualizations of erasure performance across various concepts and prompts.
18

---

Published as a conference paper at ICLR 2026
E MORE VISUALIZATIONS
E. VISUAL SAMPLES ON ERASED VAR
Building upon Figure we present additional visual samples generated by the fine-tuned erasure models in Figure T0 The results show that even when synonyms of the target concept are used as prompts, our method can still achieve accurate and effective crasure while producing natural-looking images.
I2P
Original
Ours
MMA
Original
Ours
Ring-A-Bell
Original
Ours
Figure 11: More visualizations adversarial datasets.
E.2 VISUAL SAMPLES ON ADVERSARIAL DATASETS
We further present the results of nudity erasure adversarial datasets, illustrated in Figure Even though our method does not employ adversarial training, stable erasure performance can be achieved by simply using fixed lexical representation of the target concept. When adversarial prompts are used as inputs, our method precisely identifies the target concept while generating images that appear natural and realistic than the originals. These results demonstrate that the fine-tuned
19

---

Published as a conference paper at ICLR 2026
models produced by method not only exhibit strong robustness but also retain their inherent self-correction ability.
E.3 VISUAL SAMPLES ON STYLE ERASURE
Considering that we evaluated our method on multiple datasets for NSFW erasure and object erasure, we further present additional results on style erasure, as illustrated in Figure 12 The results demonstrate that our method not only effectively removes the specified styles but also preserves the primary structures of the images generated by the original model, thereby further validating the effectiveness and accuracy of our approach.
Style Erasure: Leonardo Da Vinci
Original
Ours
Style Erasure: Claude Monet
Original
Ours
Figure 12: More visualizations on style erasure tasks.
E.4 VISUAL SAMPLES ON IRRELEVANT NATURAL PROMPTS
To provide a more intuitive understanding of the superiority of our method preserving generative capability, present images generated by different methods using COCO-30K prompts input. As shown in Figure 13 our method produces images that remain highly consistent with those the original model in terms of subject, structure, and style. In contrast, other baselines exhibit noticeable degradation in image quality. Although AC, which is closest to our method, is able to generate images with similar subjects, it introduces clear artifacts in finer details such as object boundaries. These results further demonstrate that only our method can effectively preserve the generative ability of the original model.
20

---

Published as a conference paper at ICLR 2026
FMN
ESD
CA
UCE
Ours
Original
Figure 13: Visual samples generated by different methods on nudity erasure with COCO-30K prompts.
21


Reviews:
[{"review_id": "KsDl7p7EMN", "reviewer": "Reviewer_ZPfw", "summary": "This paper proposes VARE and S-VARE, the first frameworks for concept erasure in VAR text-to-image models. VARE introduces auxiliary visual tokens to stabilize optimization and mitigate cumulative errors. S-VARE integrates a filtered cross-entropy loss and a preservation loss to achieve surgical concept removal while maintaining generation quality. Experiments show that this method has certain effectiveness.", "questions": "In addition to the issues mentioned in the Weaknesses, I have a few more concerns:\n\n1. What are the time and space complexities of the proposed method?\n2. Does erasing multiple concepts lead to cumulative negative effects, such as a decline in generation quality as the number of erased concepts increases? While I understand that there is no free lunch and that concept erasure will inevitably affect the original generation quality, I still hope the authors can provide a quantitative evaluation of this trade-off.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper focuses on the problem of concept erasure in  VAR text-to-image models, addressing a significant gap in current research.\n2. Experiments cover various erasure tasks and multiple datasets, with visualizations further demonstrating the effectiveness of the proposed method.", "weaknesses": "1. All experiments in this paper are conducted based on Infinity-2B. It lacks validation on other autoregressive architectures, which may limit the generalization of the results.\n2. The authors aim to use $L_{FCE}$ to promote erasure and $L_{Pre}$ to preserve the model’s alignment with the original distribution. However, these two objectives are inherently conflicting, which may lead to gradient interference. The authors should clarify how the balance between them is achieved.\n3. It remains unclear whether the proposed method can preserve the invariance of irrelevant factors in the original image during the erasure process. For example, after erasing nudity, can the method ensure that the person’s identity in the image remains unchanged?\n4. The competitors are mostly from 2024. To my knowledge, several new SOTA methods have emerged in 2025. It is recommended that the authors include comparisons with them. If such methods cannot be adapted to autoregressive architectures, this limitation should be explicitly discussed in the Related Work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a0cAI23IQL", "reviewer": "Reviewer_9kz9", "summary": "This paper identifies a safety gap in modern Visual Autoregressive (VAR) text-to-image models, noting that existing concept erasure (CE) methods designed for diffusion models fail when applied to VARs due to their different (next-scale token prediction) architecture. The authors propose VARE, a foundational framework that enables stable erasure by using auxiliary tokens, and S-VARE, a specific method built upon it. S-VARE introduces two novel components: a \"filtered cross entropy loss\" (LFCE) specifically designed for the quantization-based token prediction of VARs, and a \"preservation loss\" (LPre) to prevent the fine-tuning process from degrading unrelated concepts or reducing image diversity.", "questions": "1. What was the range of concepts used for the erasure experiments? Did you test the method on simple objects, artistic styles, specific identities, and more abstract concepts?\n\n2. How did you balance the weights of the filtered erasure loss (LFCE) and the preservation loss (LPre)? Is there a risk that the preservation loss could \"protect\" parts of an unwanted concept, making erasure more difficult?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The introduction clearly diagnoses why diffusion-based CE methods fail—they are not designed for the \"visual token\" prediction paradigm of VARs and lead to \"cumulative errors\" and \"severe degradation.\"\n\n2. The proposed solution appears well-reasoned and specifically tailored to the VAR architecture. Instead of retrofitting a Mean Squared Error (MSE) loss (for noise), it proposes a cross-entropy-based loss (for tokens/probabilities), which seems far more appropriate for the described model.\n\n3. The inclusion of a \"preservation loss\" (LPre) is a strong point. It shows the authors are not just focused on the primary goal (erasure) but also on the critical side effects of fine-tuning, such as \"language drift\" and loss of diversity, which are common failings of naive erasure methods.\n\n4. The headline metrics (97% erasure with <2% degradation in CLIP score) are impressive and, if validated, would represent a significant achievement in closing the safety gap.", "weaknesses": "1. The justification for the LFCE loss is heavily tied to the \"binary spherical quantization (BSQ)\" used in the Infinity model. It's not clear how generalizable this method is to other or future VAR models that might not use this specific quantization scheme.\n\n2. The paper discusses erasing \"sensitive\" or \"undesirable\" concepts. This is vague. The method's difficulty would vary greatly between erasing a simple object (e.g., a specific logo) versus a complex, abstract, or compositional concept (e.g., \"a violent action\").", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FAXhjvT5Wv", "reviewer": "Reviewer_dr9A", "summary": "This manuscript addresses a critical gap in the safety of Visual Autoregressive (VAR) models for text-to-image generation: existing Concept Erasure (CE) techniques, primarily designed for diffusion models, fail to generalize to VARs due to their next-scale token prediction paradigm. VAR models, which offer advantages in image quality and generation speed, remain vulnerable to generating sensitive content (NSFW, copyright-sensitive material) post-training, and retraining for new undesirable concepts is computationally prohibitive.\nTo solve this, the authors propose two core solutions: (1) VARE (VAR Erasure framework), which leverages auxiliary visual tokens to reduce fine-tuning intensity and mitigate cumulative errors across scales— a key limitation of directly adapting diffusion-based CE; (2) S-VARE, an enhanced method built on VARE that incorporates a filtered cross-entropy loss (L_FCE) (to precisely identify and minimally adjust unsafe tokens) and a preservation loss (L_Pre) (to maintain semantic fidelity and avoid language drift/diversity reduction). Extensive experiments show the approach erases 97% of sensitive concepts with <2% CLIP score degradation, closing the safety gap for VAR models.", "questions": "1.\tRegarding Weakness 1: For semantically distant c/c⁎ pairs (e.g., \"a violent car crash\" vs. \"a peaceful park\"). How do auxiliary visual tokens avoid misaligning non-target concepts (e.g., cannot distinguish target \"violent\" and non-target \"car crash\" )? Is there quantitative data on the model’s ability to distinguish target vs. non-target tokens in such cases?\n2.\tAddressing Weakness 2: Have you tested S-VARE on VAR models with non-BSQ quantization (e.g., traditional VQ-VAE)? \n3.\tThe manuscript sets the filter ratio α in L_FCE to 25% based on Infinity’s self-correction range. For VAR models with different self-correction thresholds (e.g., 10% or 40%), what adaptive strategy would you recommend for α?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tTargeted Solution to a Framework-Specific Gap: The manuscript clearly identifies why diffusion-based CE methods fail for VARs and designs VARE/S-VARE to align with VAR’s intrinsic dynamics. The use of auxiliary visual tokens to counter scale-wise error accumulation directly addresses a core technical barrier.\n2.\tLoss Function Design Aligns with VAR Characteristics: The filtered cross-entropy loss (L_FCE) is tailored to VAR’s binary spherical quantization (BSQ) by measuring bit-wise discrepancies, avoiding the instability of diffusion-style MSE loss on discrete token spaces. The preservation loss (L_Pre), which uses KL divergence to align pre-trained and fine-tuned model outputs, effectively mitigates common fine-tuning side effects. \n3.\tComprehensive Experimental Coverage: The authors test S-VARE across three critical CE tasks and validate robustness using adversarial datasets and real-user prompts —a rare focus on real-world deployment risks. Quantitative results and qualitative visualizations (Figure 4) provide clear evidence of the method’s effectiveness.", "weaknesses": "1.\tAmbiguity in Auxiliary Token Selection Logic: The manuscript states VARE uses tokens from prompts c (neutral) and c⁎ (with target concept) as auxiliary inputs but provides no details on how to handle scenarios where c and c⁎ have large semantic gaps (e.g., \"a violent car crash\" vs. \"a peaceful park\").\n2.\tLimited Generalization Across VAR Models and Data: The authors only use Infinity-2B as the base VAR model, with no validation on other VAR variants (e.g., models with different scale prediction strategies or non-BSQ quantization).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3iZkf4WrXM", "reviewer": "Reviewer_yGxY", "summary": "This paper presents the VARE framework and its variant, S-VARE, for concept erasure in visual autoregressive models. The approach stabilizes training and achieves concept erasure through a filtered cross-entropy loss.", "questions": "1.In line 188, the statement \"optimization with Eq. (6) introduces errors as illustrated in the left column of Figure 2\" means that the VAR model trained with Equation 6 fails to generate images properly, as shown in the left column of Figure 2. Furthermore, please provide a detailed description of the diffusion-based CE method, including how text is embedded into the VAR framework and whether a pre-trained model is used. Additionally, clarify in the main text the specific implementations and differences between the methods shown in Figure 2.\n\n2.How does Figure 3 support the conclusion that \"VAR maintains consistent optimization objectives across scales\"? Please further clarify the content presented in the figure.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. This paper introduces concept erasure into visual autoregressive models, employing auxiliary visual tokens for stable training, filtered cross-entropy for precise erasure, and retention loss to preserve image quality.\n\n2.The proposed method demonstrates effectiveness for sensitive concepts/objects/styles, with sufficient experimental support and clear comparisons against baselines, presenting a comprehensive analysis.", "weaknesses": "Major Weakness\n\n1.In line 59, the differential prompt method should be supported with an appropriate citation. In addition, it is recommended to include comparative experiments to demonstrate the superiority of the proposed VARE framework, which leverages auxiliary target tokens, over the differential prompt method, supported by quantitative results.\n\n2.In lines 211–215, the statement \"This fundamental mismatch leads to degrading the fidelity of the main subject\" lacks citation or experimental evidence. Moreover, the claim that \" it is not directly applicable to autoregressive models\" is not reasonable, as Equation 7 is an MSE-based loss and can also be applied to VAR models. Comparative experiments between the VARE framework trained with Equation 7 and the S-VARE method should be included to demonstrate the superiority of S-VARE.\n\n3.Please further clarify how Equation 7 improves upon Equation 6. Specifically, describe the internal structure of r^{ori} _{<i}, including how tokens interact with the VAR model.\n\n4.The relationship between the cross-attention mentioned in line 203 and c^{*} should be explicitly illustrated in Figure 1.\n\n5.Line 234 mentions the \"characteristics of infinity\", followed by a proposed solution, but the concept (\"characteristics of infinity\") is not clearly defined or explained in the text. Relevant descriptions should be provided.\n\n6.Please describe the novel modifications (structure) in the VARE framework within the Methods section. For example, explain how the auxiliary target tokens are embedded into the VAR model and provide the corresponding equations.\n\nMinor weakness\n\n1.In lines 195–202, the symbols in the main text are inconsistent with those in Equation (7) and Figure 1 and need to be corrected. For example, r_{ori}^{*} and r_{ori} mentioned in line 167 do not appear in the equation, and their meaning is unclear. In line 202, the text refers to \"see left of Figure 1\", but r^{ori} is not labeled in Figure 1, and the symbol r is not explained.\n\n2.What does the subscript in r_{<I}​ represent in Equation 6? Specifically, what is the meaning of r_{<I}?\n\n3.Figure 1 presents the overall framework, the text in line 155 should be revised to \"the overall framework is shown in Figure 1\".\n\n4.Based on the text, \"VARE\" refers to a framework and \"S-VARE\" to a method. The phrase \"erasure method VARE\" in lines 78–80 should be corrected to \"erasure method S-VARE\".\n\n5.In Figure 1, replace -log1/2 with γ.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5DlOzF2TaG", "reviewer": "Reviewer_TYDT", "summary": "This paper proposes a new concept erasure framework for visual autoregressive models (VAR), called VARE, and a fine-grained method S-VARE. The authors point out that existing concept erasure methods designed for diffusion models do not work well in autoregressive models due to error accumulation across scales. To solve this, they use auxiliary visual tokens during training to stabilize optimization and introduce two new loss functions: a filtered cross-entropy loss to remove only harmful tokens and a preservation loss to keep unrelated content unchanged. Experiments show strong results on NSFW, object, and style erasure tasks.", "questions": "1. Can the method be extended to erase multiple concepts at the same time?\n\n2. Can this method be applied to other autoregressive models beyond Infinity?\n\n3. I’m unsure how the filtered cross-entropy loss functions in the context of erasing artistic styles. For object erasure, it makes intuitive sense to optimize the \"object-related\" visual tokens, as they directly correspond to the target object in the image. However, when it comes to styles, the token-wise loss seems to be spread across the entire image, as shown in the first row of Figure 3, where the concept of \"blue\" appears throughout the image. In such cases, how does the filtered cross-entropy loss effectively work?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper first proposes a concept erasure method specifically for VAR models.\n\n2.The use of auxiliary visual tokens during training is a practical and effective way to reduce error accumulation in autoregressive generation.", "weaknesses": "1. The method is designed for single-concept erasure and does not support removing multiple concepts at once, which limits its applicability in real-world scenarios where multiple harmful or unwanted concepts may co-occur.\n\n2. The paper lacks a token-level interpretability analysis. It is unclear which tokens are being erased during training, at which scales, or how the filtering mechanism behaves across different types of concepts.\n\n3. The token filtering mechanism uses a fixed threshold for all tokens without considering their semantic relevance to the target concept. A more adaptive approach could improve precision by focusing only on tokens that are highly correlated with the concept to be erased.\n\n4. Missing related works: \n\n[1] Concept corrector: Erase concepts on the fly for text-to-image diffusion models\n\n[2] Dark miner: Defend against unsafe generation for text-to-image diffusion models\n\n[3] One Image is Worth a Thousand Words: A Usability Preservable Text-Image Collaborative Erasing Framework\n\n[4] Erasing More Than Intended? How Concept Erasure Degrades the Generation of Non-Target Concepts", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)