Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment
Authors: Kaixun Jiang
Zhaoyu Chen
HaiJing Guo
Jinglun Li
Jiyuan Fu
Pinxue Guo
Hao Tang
Bo Li
Wenqiang Zhang
Venue: NeurIPS 2025 poster ()
Keywords: unrestricted adversarial attack
preference alignment
Primary area: applications

Abstract: Preference alignment in diffusion models has primarily focused on benign human preferences (e.g., aesthetic). In this paper, we propose a novel perspective: framing unrestricted adversarial example generation as a problem of aligning with adversary preferences. Unlike benign alignment, adversarial alignment involves two inherently conflicting preferences: visual consistency and attack effectiveness, which often lead to unstable optimization and reward hacking (e.g., reducing visual quality to improve attack success). To address this, we propose APA (Adversary Preferences Alignment), a two-stage framework that decouples conflicting preferences and optimizes each with differentiable rewards. In the first stage, APA fine-tunes LoRA to improve visual consistency using rule-based similarity reward. In the second stage, APA updates either the image latent or prompt embedding based on feedback from a substitute classifier, guided by trajectory-level and step-wise rewards. To enhance black-box transferability, we further incorporate a diffusion augmentation strategy. Experiments demonstrate that APA achieves significantly better attack transferability while maintaining high visual consistency, inspiring further research to approach adversarial attacks from an alignment perspective.

Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment
Kaixun Jiang', Zhaoyu Chen 1M, Haijing Guo?, G Li', Jiyuan Fu?,
Pinxue Guo Hao Tang', Bo Li Wenqiang Zhang 1.2+
CCollege of Intelligent Robotics and Advanced Manufacturing, Fudan University
SSanggai Key Lab of Intelligent Information Processing,
College of Computer Science and Artificial Intelligence, Fudan University
3 School of Computer Science, Peking University
'vivo Mobile Communication Co., Ltd.
Abstract
Preference alignment diffusion models has primarily focused on benign human preferences (e.g., aesthetic). In this paper, we propose novel perspective: framing unrestricted adversarial example generation as problem of aligning with adversary preferences. Unlike benign alignment, adversarial alignment involves two inher- ently conflicting preferences: visual consistency and attack effectiveness, which often lead to unstable optimization and reward hacking (e.g.. reducing visual quality to improve attack success). To address this, we propose APA (Adversary Prefer- ences Alignment), two-stage framework that decouples conflicting preferences and optimizes each with differentiable rewards. In the first stage, APA fine-tunes LoRA improve visual consistency using rule-based similarity reward. In the second stage, APA updates either the image latent or prompt embedding based feedback from substitute classifier, guided by trajectory-level and step-wise rewards. To enhance black-box transferability, we further incorporate diffusion augmentation strategy. Experiments demonstrate that APA achieves significantly better attack transferability while maintaining high visual consistency, inspiring further research to approach adversarial attacks from an alignment perspective. Code is available at https://github. wwwpt/uww..o///pp..ee.eee/eeee.e KatteepPpkiixuuAAA.....
Introduction
Preference alignment, adapting pre-trained dif- fusion models (60) for diverse human prefer- ences, is increasingly prominent in image gener- ation. This typically involves modeling human preferences with explicit reward models pair- wise data [69], then updating model policies via reinforcement learning 3 21 backprop-
(6) Human Preferenze Aligament
Diffission Model
ovr Tou Prange
Klga
(b) Adversary Preferesces Aligmment
agation with differential reward 12
[57 411.
Viell Cebini tin
However, current research largely centers benign human preferences like aesthetics and text-image alignment (Figure a{)),, malicious adversary preferences alignment, where security researchers use diffusion models create unre- stricted adversarial examples [10] has received limited attention. These examples are vital for
Diffission Model
Irage Froncy
Figure 1: Comparison of Human Preference Align- ment and Adversary Preferences Alignment.
indicates the corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

assessing the adversarial robustness of deep learning models. Adversaries, as depicted in Figure TTb). primarily seek two preferences: 1) Visual consistency: Ensuring that generated images have mini- mal, semantically negligible differences from the original. 2) Attack effectiveness: Achieving high transferable attack performance, where adversarial examples generated from a surrogate model fool black-box target models [16]77].
Aligning with such adversarial preferences presents two major challenges. First, preference data is unavailable. Existing diffusion-based attacks [0] 7 build on the idea of traditional Lp attacks [6 adapting the strategy of adding optimized perturbations from the pixel space to the latent space. However, latent spaces in diffusion models are highly sensitive-ewe in severe semantic drift. This makes it infeasible obtain stable, pneference-consistiee examples for pairwise data collection, rendering traditional preference optimization techniques like DPO 69 or unified reward modeling inapplicable. Second, these preferences are inherently in conflict. Joint optimization with reward weighting often results in reward hacking [22], (e.g.. one shorteut improving attack success to reduce visual consistency), leading to unstable or degenerate solutions (Figure b))).
slight perturbations can result adversarial
To address this, we introduce APA (Adversary Preferences Alignment), novel two-stage framework that separates and sequentially optimizes the adversary preferences using direct backpropagation with differentiable rewards. Specifically: 1) Visual Consistency Alignment: We use differentiable visual similarity metric as a rule-based reward and perform policy updates by fine-tuning the diffusion model's Low-Rank Adaptation (LoRA) parameters (29]. This stage encodes the input image's structure into the model's generation space, forming a visually stable foundation for downstream attack optimization. 2) Attack Effectiveness Alignment: We optimize either the image latent or the prompt embedding based on feedback from a white-box surrogate classifier. This process uses dual-path attack guidance (both trajectory-level and step-wise dense rewards) to align with the adversary's 's attack preference. To prevent overfitting to the surrogate, we introduce a diffusion augmentation strategy that aggregates gradients from intermediate steps to increase diversity, thereby improving black-box transferability. Our framework explicitly decouples these conflicting preferences to mitigate reward hacking, enabling more controllable and scalable adversary preferences alignment. Our contributions are summarized follows:
We are the first to transform unrestricted adversarial attacks into adversary preferences alignment (APA) and propose an effective two-stage APA framework.
Our APA framework decouples adversary preferences into sequential stages which include LoRA-based visual consistency alignment using a rule-based visual similarity reward and attack effectiveness alignment guided by dual-path attack guidance and diffusion augmentation.
APA achieves state-of-the-an transferability against both standard and defense-equeee receene--euupped models while preserving high visual consistency. Our framework is flexible and scalable, supporting various diffusion models, optimization parameters, update strategies, and downstream tasks.
2 Related Work
Unrestricted Adversarial Attacks. Unrestricted adversarial attacks address key limitations of tradi- tional Lp attacks, which apply pixel-level perturbations that are often perceptible due to distribution shifts from clean images [33] and are increasingly countered by existing defenses [53] 64]. Instead,
unrestricted attacks generate more natural examples by subtly modifying the semantic content of the original images. Early approaches focused on single-type semantic perturbations, including shape [75 D. texture (8 36 and color [00] manipulations. Shape-based methods use deformation fields to induce structural changes; texture-based methods modify image texture or style-for exam- plc, DidIPGD [79 adds Lp perturbations in pixel space followed by diffusion- based translation, thus falling into this category. Color-based attacks (c.g., SAE (28), ReColorAdy [35), ACE (83) adjust hue, saturation, or channels to improve visual naturalness, often at the cost of transferability. However, these methods typically optimize single semantic factor, limiting their generality and expressiveness. Recent efforts leverage the latent space of generative models to produce more flexible adversarial examples. In particular, diffusion models have been adapted for this purpose [00 7 36 [3 9.. For instance, ACA (10] employs DDIM inversion and skip gradients to optimize latent representations. Nonetheless, due to the sensitivity of latent space manipulations, existing approaches often strug- gle to preserve the visual semantics of the original input. In contrast, our method the first to approach unrestricted adversarial attacks from a preference alignment perspective. The key idea lies
2

---

in alignment- driven modeling: we reframe adversarial attack generation as preference alignment problem and propose two-stage framework that decouples the conflicting objectives of visual consistency and attack effectiveness, enabling more stable and controllable generation.
Alignment of Diffusion Models. Human preference alignment for diffusion models aims optimize the pretrained diffusion model based on human reward. Current methods adapted from large language models (LLMs) include reinforcement learning (RL) 3 21 371. direct preference optimization
(DPO) 699, and direct backpropagation using differentiable rewards (DR) 12 57 41. RL approaches
frame the denoising process as multi-step pecision-making process, often using proximal policy optimization (PPO) [22] for fine-tuning. Although flexible, RL is inefficient and unstable when managing conflicting rewards like visual consistency and attack effectiveness. DPO, which avoids reward models by ranking outputs with the Bradley-Tere Terry model, faces challenges in adapting to adversarial preference alignment (APA), given the difficulty of obtaining high-quality adversarial examples for ranking. DR is efficient, using gradient-based optimization with differentiable rewards. To ensure flexibility and stability, we propose two-stage APA framework built on DR. By decoupling confficting objectives into differentiable rewards, it effectively addresses the challenges of multi- preferences alignment.
Preliminary
Unrestricted Adversarial Example (UAE). Given a clean image æ. considering both visual consis- tency and attack effectiveness, the optimization objective for UAE Nade can be expressed as:
max fo' (Iadu) + y, s.t. Fedu naturally similar to I, Fadv
(I)
where y denotes the label of x, and for (+) represents target models for which gradicnts are inaccessible for direct optimization. Since natural similarity cannot be enforced via Lp norms in perturbation- based attacks [62) unrestricted adversarial attacks must search for optimal adversarial examples in both conflicting optimization spaces.
Latent Diffusion Model (LDM). LDM 60 is a latent variable generative model trained on large- scale image-text pairs, relying on an iterative denoising mechanism. During training, the denoising model oa (.) is trained by minimizing the variational lower bound loss function, typically using a UNet to predict the noise added to the original data:
min E |k cy
(2)
where t denotes the timestep. T denotes the total number of timesteps, and c is the random noise sampled from (0, I). The latent variable 2, is generated by adding noise to 2n over t steps, where 20 is the latent representation of the original input x obtained through encoder E()), i.e., E(x) zo. The diffusion process is defined as y(=x|=o) = votz 2g + vI- at - where va, is a hyperparameter that controls the level of noise added at each timestep / 127 denotes the conditioning text. During inference, we typically sample 2T from (rom)((,)) ( I). and then use the DDIM denoising [65] to iteratively denoise ZT. The iterative denoising process can be expressed as:
2e-1
Var-1 (1 vac
ai- t,),
(3)
After T steps of DDIM denoising, the resulting ăo is decoded into pixel space via a decoder D(-), generating an image that matches the condition For tasks with given reference image (e.g. image editing), 2r is typically not sampled from random noise, instead, it is obtained through DDIM Inversion [65] based on the reference image 2::
2
/1100-1-toaae 1o00 + vI- as eo (f,)). Vor-t
(4)
where initial 2n denotes the latent of reference image x. Through whole DDIM inversion, i.e., iteratively using 7 times, we obtain z7, which preserves information of x.
4 Adversary Preferences Alignment Framework
Existing works 10 leverage the natural image generation capabilities of diffusion models to generate unrestricted adversarial examples, the adversary optimizes 27 (obtained via DDIM Inversion) instead
3

---

Vasual Consisiency Aligamcet
R
Rewward Rs
Sepplind
Class
Cneese
Argantatle
-
"hoseboll player"
Attack Guldonce Denobe
Reward R
(b) Attack Esflectheness Alignment
xTT-T
DDIM
Attak Guidance
Demise
Coutied berkiig
58 Gredirs
Figure 2: Overview of our APA framework. APA first optimizes the LoRA parameters with a visual consistency reward, storing input image information in LoRA. Then, the input image undergoes DDIM inversion obtain 2T. After DDIM denoising and attack guidance denoising, APA generates trajectory-lers Ep and diffusion augmentation output z6. mixing them using Eq, and passing the substitute classifier calculate Ra. Finally, 2r is iteratively optimized using skip gradient (APA-SG) or gradient checkpointing (APA-GC).
of directly optimizing the pixel space. The mapping from 2T to Zade is defined as Tady (20)) where Eg is obtained by applying I' steps of DDIM denoising: Eo De o ... (e(2T). This
sequential process defines denoising trajectory. However, these methods attempt solve Eq. T by jointly optimizing 27- Due to the sensitivity of the latent space to noise (shown in Figure 3 and the mutual exclusivity of the two optimization objectives (shown in (ggure[]bb), the generated adversarial examples often fall into a suboptimal trade-off between the conflicting objectives.
To address this, we propose two-stage adversary preferences alignment framework (APA): first, we reframe unrestricted adversarial attacks as multi-preference alignment problem and decouple visual consistency and attack effectiveness to independent reward models. Then we strengthen visual consistency via LoRA fine-tuning in the first stage and focus on attack effectiveness through dual-path attack guidance and diffusion augmentation in the second stage. Our APA separates visual consistency and attack effectiveness by independently modeling and aligning preference rewards. then maximizes attack performance within the optimal solution space of visual consistency, achieving closer Pareto optimality. Figure 2 and Alogrithim present the overall framework of APA.
4.1 Visual Consistency Alignment
Diffusion models derive their capabilitics from training on extensive images [60 51),. meaning that even minor changes to the latent prompt can result in substantially different generations. As shown in Figure [ without perturbations to the 27 obtained
Nillo State
KZ
RJ
611
VLA
DDIM
Inversion, the model nearly reconstructs the clean image after 7 denoising steps. However, minor noise, particularly adversarial noise, can cause the generated image to lose vi- sual consistency with the original. To pre- serve visual consistency during adversarial op- timization, we aim to strengthen the diffusion model's 's retention of the input image z. straightforward approach is to fine-tune the UNet to overfit the input image, but this risks is inefficient. Previous research in customized generation [24]]
VCA
Figure 3: Impact of adversarial and random noise on in generated images. VCA denotes visual con- sistency alignment. Our LoRA-based VCA, demon- strates improved noise robustness.
catastrophic forgetting, degrades image quality, and
suggests that LoRA [29] efficiently encodes high-dimensional image semantics into the low-rank parameter space. Thus, we adopt LoRA A0 the policy model during the visual consistency alignment stage. Then, need to
81
73 74

---

determine reward model or reward function (.) to optimize AG. The straightforward approach is compute the visual similarity S(-) between the original input and the output of the diffusion model, as follows:
max (20) a)) 29
where Eg represents the T-step denoised output, requiring T computations of Eq. [] To reduce this, first shift the similarity metric to the latent space, caleulating ( We then approximate trajectory-level similarity by accumulating similarity across all steps. Thus, inspired by Eq. 2 Rg (2()
reformulated as:
R. (20) 11- - ||e E0+ (ze, (,
(6)
Since Eq. 6 is differentiable, we can update A0 via the direct backpropagation 12 57 to maximize the reward, as follows: A A + a Ao R,, where represents the learning rate. Finally, AO is integrated into <g. enabling the model to generate visually consistent outputs whether regular noise or adversarial noise is applied 27, as illustrated in Figure
4.2 Attack Effectiveness Alignment
In this stage, We use 27 obtained via DDIM inversion as the optimization variable (optional prompt embedding discussed in Section 5.5) Next, need to model the reward R, for attack effectivenesss alignment. First, using f in Eq. directly the reward model results in sparse rewards of only or 0, significantly increasing optimization difficulty. To address this, we draw inspiration from traditional transfer attacks [17 [6) and choose a differentiable surrogate model f.(.) as the reward model. This allows optimization via direct backpropagation based on gradients [12] (2) Additionally, to mitigate the gap between the surrogate model and the target model, we propose diffusion augmentation to enhance generalization and alleviate potential reward hacking [22]. Thus, the attack effectiveness
reward is formulated as: Ra (2r) ) = L(fo (Radv). ), y). L(-) denotes cross-entropy loss.
Dual-path Attack Guidance. We refer to the generation of Zadv through T-step denoising from 2T as the generation trajectory. Additionally, to enhance gradient consistency, following ACA [10), we use momentum-basee gradient update to optimize ZT, encouraging the model to higher R. output. Thus, our trajectory-Ievel attack optimization can be expressed as:
gtr Ra
mir
mir
Itr
2T
II (2T + P sgn sgn(m.))):
(7)
where Itr denotes the trajectory-level gradient, mi, denotes the momentum of 7th trajectory-Ievel attack iteration, te, keeps 27 remain within the ea ea-ball centered at the original latent 2/. sgn(-) denotes sign function.
Since solving Itr requires computing the gradient across the entire trajectory. direct calculation would require extensive memory. One solution is skip gradient [10), which approximates Itr as
p- V 2g R₂ (x y) [1O), avoiding memory use for Ttstep denoising. The other gradient checkpointing [8) which reduces memory during backpropagation by selectively storing intermediate activations, enabling direct computation of gtr-
Both skip gradient and gradient checkpointing focus on optimizing zr from global trajectory level, where each denoising step uses the same Rgg However, different steps contribute uniquely to the final output: larger timesteps affect structure, while smaller ones refine details [43]. As result, using the same R2 for attack guidance may cause misalignment, reducing attack effectiveness.
To address this issue, we incorporate step-level attack guidance into the denoising steps. Motivated by class -uidedd generation [15]. we introduce the attack reward R. during each denoising step to guide the noise optimization:
60+20(2,,t,c) = 60+20(20, - o. (f R.
(8)
Each denoising step adjusts the generation direction based the current step's R. (f gradually aligning the final image with higher R.,
Since 2, is intermediate denoising result, directly inputting it to the classifier biases reward calculation, as classifiers are typically trained on clean samples. A noise-robust classifier could reduce this bias (15), but it would raise training costs and may introduce inconsistencies between
5

---

substitute and target classifiers, affecting attack performance. To address this, we first replace 2; with the intermediate result zj generated by DDIM, which represents the estimated trajectory-level 2 based on the current step:
2 v1 - 04 (2,,t,c
VOx,
Furthermore, given that =f has a bias that increases with larger 1, we further refine =j by interpolating between the original image's 's latent 20 and the predicted z6. as follows:
2/n v 20 (1 vI ai) 2/-
(10)
where v og decreases L decreases, allowing 2g to take on higher weight at larger / values, making the sample input oe = D(z ) to the classifier progressively cleaner, enhancing reward accuracy at each denoising step. Additionally, inspired by the trajectory-level momentum update method, we propose a step-level momentum accumulation:
Ist Ra (f m'a m
gst
(11)
11
where Ost and m'y denotes step-level gradient and momentum. We replace V., (D)((()))) in Eq. [8 with sgn(m")). Finally, by combining trajectory-level and step-level dual-path attack guidance, the generated images are fully aligned with attack effectiveness preference.
Diffusion Augmentation. Our dual-path attack optimization is based on direct backpropagation with differentiable reward. Studies HPA [47] have shown that direct backpropagation often leads to the diffusion model over-optimizing for the reward model. Similarly, in APA, this causes overfitting to the substitute classifier, limiting transfer attack performance. To address this, we propose diffusion augmentation which uses step-level outputs as data augmentation to enhance the generalization the trajectory-Iever gradient gtr- Specifically, collect the step-level z generated during the denoising using Eq. 9 and mix them with the trajectory-level final output žy:
i e((((=)) + D(2)))/2).
(12)
where o(-) denotes differentiable data augmentation including random padding, resizing, and bright- ness adjustment. Appendix [F3] further shows that stronger data transformations (e.g.. [72] used in Lp attacks) can further boost performance, underscoring the scalability of our method. Finally, the trajectory-level gradient gr, in Eq. is enhanced to gtr = PT + E Ra (fs(())) v).
We collectively refer to step-level attack guidance and diffusion augmentation as the attack guidance denoise process, as shown in Figure To balance time efficiency and image quality, we apply attack guidance denoise only in the final Ta steps. Overall, our framework is implemented as a clean and modular two-stage pipeline. The visual consistency alignment is achieved through lightweight LoRA fine-tuning, and after merging the LoRA parameters into the UNet, no extra parameters are introduced in the subsequent attack alignment stage. The attack alignment is seamlessly integrated into the denoising process by augmenting it with attack guidance. Moreover, our APA framework is highly flexible, allowing for different gradient propagation strategies (APA-SG for skip gradient and APA-GC for gradient checkpointing) and different optimization params (APA-GC-P for text optimization). More details are provided below.
5 Experiments
5.1 Experimental Settings
Datasets and Models. We choose the widely used ImageNet-compatible Dataset (34], consisting of 1,000 images from ImageNet's validation set [4]. Following [00, we select convolutional neural networks (CNNs) and 4 vision transformers (ViTs) as target models for the attack.
Attack Methods. We compare with unrestricted attacks including SAE 28] cAdv 22, tAdy [
ColorFool (63] and NCF [80, as well as diffusion- based methods ACA 100. DiffAttack 7 and
DiffPGD 79 in T Following
[0], we use attack success rate (ASR, %), the percentage of misclassified images- as the evaluation metric, reporting both white-box and black-box ASR. Addi-
tional comparisons under different settings are presented in the Appendix: 1) AdyDiffuser 9
and AdyDiff [13] in Appendix E.3 2) Lp attacks [16. 79 17 45) in Appendix E.2

---

Table 1: Attack performance comparison normally trained CNNs and ViTs. We report attack success rates ASR (%) each method (**" means white-box ASR), Avg. ASR refers to the average attack success rate non-substitute models (black-box ASR)
Models
CNNs
Transfomers
Subotitutc Model
Inc-c3
Dens-161
MobViT-s
(58)
61
66
26
50
22
[7
53]
[S]
71
Clean
12.1
4,8
7.0
6.3
5.6
8.7
7.8
8.9
3.5
3.6
6.83
42.7
cAde
41.9
25.4
33.2
31.2
28.2
34.7
84.3
32.6
22.7
22.0
30.21
tAdv
33.6
18.8
22.1
18.7
18.7
15.8
97.4"
15.3
11.2
13.7
18.66
ColorFool
NCF
67.7
31.2
60.3
41.8
$2.2
32.2
74.3
39.1
20.8
23,
40.93
MobVITss
DiMPGD-MI
59.9
42.4
48.1
44.5
38.5
38.1
95.7"
29.0
24.9
42.5
40.89
ACA
66.2
60.6
58.1
55.9
55.5
89.8"
52.7
58.1
56.90
DiTAAtck
79.2
62.1
25.5
22.2
72.1
66.0
99.8
39.1
64.6
23.7
20.02
APA-SG(OurS)
X1.3
66.3
73.88
71.5
68.9
65.0
98
$1.6
46.1
68.2
65 X5
APA-GC(Our)
88.3
77.1
86.6
81.2
81.2
78.4
99.4
59.3
61.9
83.4
77.48
22.5
38.0
26.9
32.87
cAde
96.6*
26.8
39.6
35.9
29.9
32.7
41.99
33.1
20.6
19.7
30.91
tAdv
99.9
27.2
31.5
24.33
24.5
22.4
40.55
16.1
15.9
15.1
24.17
ColorFool
NCF
93,29
33.6
65.9
43.55
$6.3
33.0
52.6
35.8
21.2
20.6
40.28
MIN-22
DiMPGD-MI
97.4"
54.1
68.2
57.8
$6.6
52.1
68.0
28.7
22.9
41.8
50.02
ACA
93.1
56.8
55.7
$1.0
59.6
48.7
48.6
50.4
54.38
DiffArtack
APA-SG(Ours)
99.8
80.4
88.1
83.0
81.7
78.8
78.5
55.9
39.5
63.4
72.
APA-GC(Ours
100"
91.4
97.7
95.5
95.0
91.8
93.2
74.3
59.0
85.2
87.01
28.8
45.9
19.6
cAde
44.2
25.3
97.2
36.8
37.0
34,9
-00.1
30.6
19.3
20.2
32.04
tAdv
43.4
27.0
99.0
28.8
30.2
21.6
35.99
16.5
15.2
15.
25.97
ColorFool
NCF
71.2
33.6
91.4
48.5
50.5
32.4
$2.6
36.8
19.8
21.7
41.90
RN-50
DiAPGD-MI
75.2
60.6
96.8
75.0
78.9
55.3
67.5
30.3
26.5
48.5
57.53
ACA
69.3
61.6
88.3"
61.9
60.3
62.6
$2.9
53.2
59.49
DiffArtack
97.2-
APA-SG(Ours)
89.0
83.4
99.69
89.6
90.1
71.3
76.7
58.5
45.7
67.6
75
APA-GC(Ours
97.6
93.5
99.74
97.6
98.4
91.1
90.9
75.6
63.8
83.7
88.02
26.9
49.7
30.4
19.9
cAde
31.4
27.0
26.1
22.5
19.9
26.1
32.9
96.5
18.4
16.9
24.58
tAdv
39,5
22.8
25.8
23.2
22.3
20.8
34.1
93.5*
16.3
15.3
24.46
ColorFool
NCF
55.9
25.3
50.6
34.8
42.3
29.9
40.6
81.00
20.0
19.1
35.39
WTBBB
DiMPGD-MI
59.5
40.9
44.2
41.9
41.3
41.3
52.2
95.4*
42.1
33.8
44.13
ACA
64.6
58.8
58.1
58.1
$7.1
87.7*
53.5
54.9
58
DiffArtack
APA-SG(Ours)
69.3
67.6
62.5
66.8
65.4
20.0
67.6
99.2*
62.5
59.2
66.21
APA-GC(Ours)
77,0
74.8
75,4
75.9
75.4
76.8
74.5
98.4
73.4
70.2
74.82
Implementation Details. We set attack guidance step T, 10, attack iterations N = 10. attack
scale c, = 0.4, and attack step size = 0.04. APA-SG adopts the entire inversion step of T = 50. APA-GC adopts T 10 to improve efficiency. Our work is based Stable Diffusion V1.5 [60]. During the visual consistency alignment phase, we fine-tune only the projection matrices Q. K, and in the attention modules of the UNet with each clean image. With the LoRA rank set 8, train for 200 steps. Experimental results indicate that APA-GC delivers strong attack performance. Thus, add visual consistency constraints APA-GC's R, without concerns about impacting attack performance, setting R, = Ra A|| oo 12 with 10.
5.2 Attack Performance Comparison
To evaluate the performance our APA framework, we select 10 models target models, including both CNN and transformer architectures. The attack performance comparison is shown in Table |
Non-diffussio--aal Attacks. Texture-based tAdv achieves higher white-box ASR but lower black-box ASR than color-based methods such as NCF and SAE. NCF demonstrates the highest trans- ferability, achieving average ASR of 39.6% across four models. Our APA, which modifies multiple input semantics simultaneously, surpasses memaeicemmatic attacks, with notable improvements and 30.2% (APA-SG) and 42.2% (APA-GC) in black-box ASR across four models than NCF.
Diffusion-based Attacks. DiffPGD-MI generates unrestricted adversarial examples by first applying Lp norm perturbations, and then performing diffusion-based image translation. ACA and DiffAttack

---

Table 2: Attack performance adversarial defense methods, ViT-B as the substitute model.
Attack
HGD
R&P
NIPS-r3
JPBG
Bit- Red
DittPure
Inc-
Inc- Vieuat
IncRes- Vaar
Res-De
Shape- Res
ViT-B- CvSi
Avg. ASR (%)
6.2
2.6
SAE
21.4
19.0
25.2
25.7
43.5
39.8
25.7
29.6
20.0
35.1
49.6
38.9
31.13
cAdv
12.2
14.0
17.7
11.1
33.9
14.6
16.2
25.3
2016
20.13
tAdv
17.8
12.5
17.38
ColotFool
9.1
9.6
15.3
18.0
37.9
33.8
17.8
21.3
10.5
20.3
35.01
31.2
21:65
NCP
22.8
21.1
25.8
26.8
43.9
39.6
27.4
31.9
21.8
34.4
47.5
35.8
31.57
DIIGGD-
ACA
59.7
$3.6
55.6
$7.47
DifAllack
33.3
34.3
33.2
37.9
47.0
38.3
38.0
42.6
35.7
40.6
45.5
19.7
37.17
APA-SG(Ours)
61.5
61.0
63.8
66.7
71.0
66.8
67.4
63.1
64.5
68.7
45.6
63 59
APA-GC(Ours)
72.4
71.4
directly optimize the input image's latent space, generally achieving higher black-box transferability. especially across architectures. Our method incorporates dual-path attack guidance and diffusion augmentation, enabling APA-SG (with the same gradient backpropagation ACA) to improve black-box ASR by 12.5%, 10.0% while APA-GC improves black-box performance by 24.4%, 21.9% over ACA and DiffAttack across four models.
Overall. Our method outperforms existing unrestricted adversarial attacks in terms of black-box transferability, whether within CNN or transformer architectures or in cross-architecture attacks. Ad- ditionally, due its more precise gradient-guie -uided attack, APA-GC achieves an average performance improvement of 11.9% over APA-SG.
5.3 Attacks on Adversarial Defense
To evaluate unrestricted adversarial attacks against existing defenses, select adversarially trained models (Inc-v3ens3, Inc-v3ens4, Inc-v2ens (68].
ViT-B-CvS((644) and preprocessing defenses (HGD 44 R&P 766] NIPS-r3, JPEG 25], J. Bit-Red [78. DuffPure (55). Additionally, shape-texture ResNet (Shape-Res) (2] are selected to counter unrestricted adversarial examples, shown in Table We use ViT-B as the substitute model and Inc-v3ens3 as the target model for input preprocessing defenses. Since existing defenses mainly address Lp attacks, they remain ineffective against unrestricted adversarial attacks. With advanced APA framework, our APA-GC achieves a 12.7% improvement on Avg. ASR over SOTA method.
debiased models [ResNec50-Debiased (Res- De) 42
5.4 Visual Quality Comparison
We compare the visual performance of the top six attack methods based on attack per- formance in Table I using RN-50 the sub- stitute model.
Table 3: Quantitative comparison of image quality. VCA denotes only using visual consistency alignment. APA-GC-P denotes prompt- based optimization.
Allack
LPIPS)
SSIM
Sooref
NIMA
IQAT
Aug.
We use reference-based metrics (LPIPS, SSIM. CLIP Score (59]) to evaluate visual similarity in terms of distribution, structure, and se- mantics, and no-reference aesthetic metrics (NIMA-AVA [54), CNN-IQA (5)) to assess aesthetic quality, as shown in Table 1) Our visual consistency alignment (VCA) main-
Quantitative Comparison.
Clean
0.00
1,00
1.00
4.99
0.58
6,83
DDIM Inversion
0.9
0.73
0.80
5.03
0.63
22.10
VCA
0.06
0.85
0.97
5.13
0.63
7,60
SAE
043
0.79
0.51
5.00
0.53
36.38
cAdy
0.5
0.96
0.88
4.85
0.55
32.04
NCF
0.57
DIMPGDD-M
0.20
0.61
57.55
DeftAmack
0.14
APA-SG(Ours)
0.25
0.67
0.36
3.29
0.62
75.32
APA-GCCOursi
0.23
0.09
0.33
5.39
0.67
88.02
APA-GC-P(Oarr)
0.09
0.52
0.91
5.22
0.63
62.08
tains strong visual consistency while achiev- ing a higher aesthetic score compared to clean images and original DDIM Inversion. 2) Our method achieves higher aesthetic scores compared to non-diffusion-baat attacks and DifIPGD owing to stable diffusion's 's strong generative capabilities. 3) Our APA achieves comparable visual similarity and aesthetic scores to ACA and DiffAttack. 4) Within framework, APA-GC-P which optimizes prompt instead of latent (see Section 5.5) has the best visual consistency.
Qualitative Comparison. Due to the inability of quantitative metrics to fully measure visual consistency, we perform a qualitative analysis in Figure SAE with NCF alters the original style, DifiPGD-MI introduces noticeable perturbations, and cAdv affects authenticity by changing colors.

---

DINRCD-M
DiIrwwock
Figure 4: Qualitative comparison of image quality.
ACA disrupts the original structure, while DiffAttack makes certain parts of the main subject appear overly sharp and unnatural. In contrast, our APA preserves structure and color, making only subtle, natural adjustments mainly in the background, resulting in a more visually consistent effect.
5.5
Ablation Studies
The previous section has discussed the ablation of visual consistency alignment (Figure B and gradient backpropagation (Table []. Here, focus analyzing the remaining key modules and design. All experiments utilize RN-50 as the substitute model. Time analysis is discussed Appendix A
Key Modules. Rows 1 and 2 of Ta- ble show that the dual-path attack guidance module improves black-box attack performance by 6.6% compared to only trajectory-level guidance. To further validate the superiority of our attack guidance, we re-implement class- guided [13] and Upainting [40] by ap-
Table 4: Ablation studies on key modules. L denotes latent- based optimization, P denotes prompt-
-bsed optimization.
Optimized
Dual-path
Diffusion
Backpro pogation
White-bux ASR(S)
Black-box ASR(S5)
Parims
Guiddnce
Angmentation
SG
96.8
48.55
99.7
54,88
62.58
75.32
99.7
GC
99.5
62.08
plying R. for adversary preferences alignment. Figure shows improved attack performance with our method, which benefits from more accurate attack reward guidance through clear 2f,, in Eq. 10 and step-level momentum accumulation in Eq, TT] Rows 1 and 3 of Table demoostrattts that diffusion augmentation mitigates the limitations of direct backpropagation overfitting to the substitute model, improving black-box performance by 14.1 % with only a slight decrease in white-box performance. Rows 3 and 4 in 4 show that diffusion augmentation combined with dual-path attack guidance effectively improves black-box attack performance.
Two-stage vs. One-stage Alignment. To validate the advantages of our two-stage alignment, we adapt APA into single-stage alignment (APA*): replacing LoRA-based visual alignment and incorporating joint optimization in the second stage, i.e., Rg = R. A||2, žoll2. Experimental results in Figure/5]bb demonstrate: 1) One-stage alignment (both APA * and ACA) suffer from reward hacking due to conflicting objectives during joint optimization (as À increases, Avg. ASR decreases while SSIM increases). 2) Our two-stage APA maximizes attack performance within the optimal solution space of visual consistency, achieving closer Pareto optimality.
Optimized Parameters. Row in Table shows the attack performance with prompt-based optimization (APA-GC-P), which optimizes text features TT(C) with gradient checkpointing. Com- pared to the direct correspondence between the latent and image spaces, prompt-based optimization indirectly guides image generation through 69+ s0 (2a, t, c), resulting in lower attack efficacy than APA-GC. However, as shown in Table APA-GC-P demonstrates improved visual consistency, offering attackers greater flexibility depending on the application scenario.
Scalability. To demonstrate the flexibility and scalability of our framework, we extend it to various diffusion models (e.g.. ControlNet (39) Appendix [.T and different tasks including targeted attacks, visual question answering, and object detection in Appendix F22
[an

---

(bbrre m1:5y 20 MENC gadaron
Dit ENCC
3
410
(2)
Figure 5: (a) Comparison with different opti- mization guidance. (b) Comparison of our two- stage APA-SG and one-stage alignment under X-controlled visual consistency.
Figure 6: (a) denotes hhper--paametteee parameters tuning on T. (b) and (C) denotes syper-parameterd tuning on Ta. RN-50 as the substitute model.
5.6
tuning
Guidance Step To. Figure O (b), (c) show the impact of T, on performance. As Ta increases, attack performance improves, time cost rises, and image quality deteriorates. Considering these factors, we choose T. 10.
Inversion Step T. APA-GC employs gradient checkpoint- ing to save memory at the cost of additional time. To improve efficiency, we investigate the impact of reducing inversion steps 7 on performance. Figure a) shows that setting below Ta reduces attack performance due to insufficient guidance, while exceeding Ta also degrades attack performance due bias introduced by overly deep gradient chains. Thus, we set T T. 10.
effcct el stack forottons on Alook Perfemennc
Attack Iterations N. We analyze the effect of the number of attack iterations on performance, as shown in Figure 7 The attack performance our APA improves rapidly with increasing iterations, where APA-GC, benefiting from more accurate gradient computation, exhibits we observe that APA-SG
Figure 7: Effect of attack iterations on attack performance.
faster improvement and earlier convergence. Together with Table [ surpasses ACA within only six iterations, while APA-GC achieves this just four.
6 Conclusion
In this paper, we broaden the application of preference alignment, reformulating unrestricted adver- sarial example generation
adversary preferences alignment problem. However, the inherently conflicting objectives of visual consistency and attack effectiveness significantly increase the difficulty of alignment. To address this challenge, we propose Adversary Preferences Alignment (APA), two-stage framework that first establishes visual consistency through LoRA-baser alignment guided by a rule-based similarity reward, and then enhances attack effectiveness via dual-pal -ath attack guidance and diffusion- based augmentation. Experimental results demonstrate that APA achieves superior black-box transferability while preserving high visual consistency. We hope work serves as a bridge between preference alignment and adversarial attacks, and inspires further research on adversarial robustness from an alignment perspective.
Broader Impacts. Our work is centered around unrestricted adversarial examples, aiming to deepen the understanding of model vulnerability and enhance the robustness and reliability of deep learning models. The proposed methodology and research results are intended to be used only for academic and ethical purposes, such enhancing security protection capabilities well as facilitating the safe implementation of Al technologies.
Ackowledgements. This work was supported by National Natural Science Foundation of China (No.62576 09, 62072112).
10

---

References
[1] Rima Alaifari, Giovanni S. Alberti, and Tandri Gauksson. Adef: an iterative algorithm to construct adversarial deformations. In ICLR, 2019.
[2] Anand Bhattad, Min Jin Chong, Kaizhao Liang. B. Li, and David A. Forsyth. Unrestricted adversarial examples via semantic manipulation. In ICLR, 2019.
[3] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In ICLR, 2024.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[5] Chaofeng Chen and Jiadi Mo. IQA-PyTorch: Pytorch toolbox for image quality assessment. [Online]. Available: https://githie htt.:hhb..ooomeeeeeoommmmooeeeee wtt.cccocccccaeeee BataaofengeeIQA-PPTooree. 2022.
[6] Chaofeng Chen, Annan Wang. Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi
Lin. Enhancing diffusion models with text-encoder reinforcement learning. In ECCV, 2024.
[7] Jianqi Chen, Hao Chen, Keyan Chen, Yilan Zhang, Zhengxia Zou, and Zhenwei Shi. Diffusion models for imperceptible and transferable adversarial attack. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[8] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear 2016.
memory cost. arXiv preprint
[9] Xinquan Chen, Xitong Gao, Juanjuan Zhao, Kejiang Ye, and Cheng-Zhong Xu. Advdiffuser: Natural adversarial example synthesis with diffusion models. In ICCV, 2023.
[10] Zhaoyu Chen, Bo Li, Shuang Wu, Kaixun Jiang, Shouhong Ding, and Wenqiang Zhang. Content-basee based unrestricted adversarial attack. NeurlPS, 2024.
[11] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. The Eleventh Interna- tional Conference on Learning Representations, 2025.
[12] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In ICLR, 2024.
[13] Xuelong Dai, Kaisheng Liang, and Bin Xiao. Advdiff: Generating unrestricted adversarial examples using diffusion models. In ECCV, 2024.
[14] Jia Deng, Alex Berg, Sanjeev Satheesh, H Su, Aditya Khosla, and Li Fei-Fei. Imagenet large scale visual recognition competition 2012 (ilsvre2012). See net. org/challenges/LLSVC., 41, 2012.
[15] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurlPS, 2021.
[16] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In CVPR, 2018.
[17] Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by canslation-invariant attacks. In CVPR, 2019.
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICIR, 2021.
[19] Maayan Ehrenberg, Roy Ganz, and Nir Rosenfeld. Adversaries with incentives: A strategic alternative to adversarial robustness. In The Thirteenth International Conference on Learning Representations, 2025.
11

---

[20] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- ers for high-resolution image synthesis. In ICLR, 2024.
[21] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to-image diffusion models. In NeurIPS, 2023.
[22] Robert Geirhos, Jörn-Hentt Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning deep neural networks. Nature Machine Intelligence, 2020.
[23] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019.
[24] Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. NeurIPS, 2024.
[25] Chuan Guo, Mayank Rana, Moustapha Cissé, and Laurens van der Maaten. Countering adversarial images using input transformations. In ICLR, 2018.
[26] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[27] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
[28] Hossein Hosseini and Radha Poovendran. Semantic adversarial examples. CVPRW, 2018.
[29] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
[30] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, pages 2261-226222222 22699 IEEE Computer Society, 2017.
[31] Kaixun Jiang, Zhaoyu Chen, Jiyuan Fu, Lingyi Hong, Jinglun Li, and Wenqiang Zhang. Videopure: Diffusion-based adversarial purification for video recognition. IEEE Transactions on Circuits and Systems for Video Technology, 2025.
[32] Kaixun Jiang, Zhaoyu Chen, Hao Huang, Jiafeng Wang. Dingkang Yang. Bo Li, Yan Wang, and Wenqiang Zhang. Efficient decision-base black-box patch attacks on video recognition. In ICCV, pages 4379-4389, 2023.
[33] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In ECCV, 2016.
[34] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security, pages 912 Chapman and Hal//CRC, 2018.
[35] Cassidy Laidlaw and Soheil Feizi. Functional adversarial attacks. In NeurIPS, 2019.
[36] Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. Perceptual adversarial robustness: Defense against unseen threat models. In ICLR, 2021.
[37] Seung Hyun Lee, Yinxiao Li, Junjie Ke, Innfarn Yoo, Han Zhang, Jiahui Yu, Qifei Wang, Fei Deng, Glenn Entis, Junfeng He, et Parrot: Pareto-optimal multi-reward reinforcement learning framework for text-to-image generation. In ECCV, 2024.
[38] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint 201. 2023.
12

---

[39] Ming Li, Taojiannan Yang, Huafeng Kuang, Jie Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. Controlnet++: Improving conditional controls with efficient consistency feedback. In ECCV, 2024.
[40] Wei Li, Xue Xu, Xinyan Xiao, Jiachen Liu, Hu Yang. Guohao Li, Zhanpeng Wang. Zhifan Feng, Qiaoqiao She, Yajuan Lyu, et al. Upainting: Unified text-to-image diffusion generation with cross- moossl guidance. arXiv preprint arXiv:2210.1600 2022.
[41] Yanyu Li, Xian Liu, Anil Kag, Ju Hu, Yerlan Idelbayev. Dhritiman Sagar, Yanzhi Wang. Sergey Tulyakov, and Jian Ren. Textcraftor: Your text encoder can be image quality controller. In CVPR, 2024.
[42] Yingwei Li, Qihang Yu, Mingxing Tan, Jieru Mei, Peng Tang, Wei Shen, Alan L. Yuille, and Cihang Xie. Shape-texture debiased neural network training. In ICLR, 2021.
[43] Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang. Ji Li, and Liang Zheng. Step-aware preference optimization: Aligning preference with denoising performance each step. arXiv preprint arXiv:2406.043 2024.
[44] Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In CVPR, 2018.
[45] Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E Hopcroft. Nesterov accelerated gradient and scale invariance for adversarial attacks. arXiv preprint arXiv:1908.062. 2019.
[46] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:22 2022.
[47] Buhua Liu, Shitong Shao, Bao Li, Lichen Bai, Haoyi Xiong, James Kwok, Sumi Helal, and Zeke Xie. Alignment of diffusion models: Fundamentals, challenges, and future. arXiv preprint 24X.022409.07253, 2024.
[48] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[49] Zhihang Liu, Chen-Wei Xie, Pandeng Li, Liming Zhao, Longxiang Tang, Yun Zheng, Chuanbin Liu, and Hongtao Xie. Hybrid-level instruction injection for video token compression in multi- modal large language models. In Proceedings of the Computer Vision and Patter Recognition Conference, pages 8568 8578, 2025.
[50] Zhihang Liu, Chen Wei Xie, Bin Wen, Feiwu Yu, Jixuan Chen, Boqiang Zhang, Nianzu Yang, Pandeng Li, Yinglu Li, Zuan Gao, et al. Capability: comprehensive visual caption benchmark for evaluating both correctness and thoroughness. arXiv preprini 259. 2025.
[51] Baorui Ma. Huachen Gao, Haoge Deng, Zhengxiong Luo, Tiejun Huang, Lulu Tang, and Xinlong Wang. You see it, you got it: Learning 3d creation pose-free videos at scale. In CVPR, pages 2016-2029, 2025.
[52] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[53] Sachin Mchta and Mohammad Rastegari. Mobilevit: Light-weight, general purroos,, and mobile-friendly vision transformer. In ICLR, 2022.
[54] Naila Murray, Luca Marchesotti, and Florent Perronnin. Ava: large-scale database for aesthetic visual analysis. In CVPR, 2012.
[55] Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Animashree Anand- kumar. Diffusion models for adversarial purification. In ICML, 2022.
[56] Zihao Pan, Weibin Wu, Yuhang Cao, and Zibin Zheng. Sca: Highly efficient semantie-consistent unrestricted adversarial attack. arXiv preprint arXiv:2410.022400 2024.
13

---

[57] Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text- to-image diffusion models with reward backpropagation. arXiv preprint arXlv:2310.037.9, 2023.
[58] Haonan Qiu, Chaowei Xiao, Lei Yang, Xinchen Yan, Honglak Lee, and Bo Li. Semanticadv: Generating adversarial examples via attribute- conditioonc image editing. In ECCV, 2020.
[59] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML,, 2021.
[60] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[61] Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
[62] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.066: 2017.
[63] Ali Shahin Shamsabadi, Ricardo Sánchez-Matilla, and Andrea Cavallaro. Colorfool: Semantic adversarial colorization. In CVPR, pages 1 114--1157, 2020.
[64] Naman Deep Singh, Francesco Croce, and Matthias Hein. Revisiting adversarial training for imagenet: Architectures, training and generalization across threat models. In NeurlPS, 2023.
[65] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[66] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
[67] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. ICML, 2019.
[68] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, lan Goodfellow, Dan Boneh, and Patrick D. McDaniel. Ensemble adversarial training: Attacks and defenses. In ICLR, 2018.
[69] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong. Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. CVPR, 2024.
[70] Kunyu Wang, Xuanran He, Wenxuan Wang. and Xiaosen Wang. Boosting adversarial transfer- ability by block shuffle and rotation. CVPR, 2024.
[71] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. PVT v2: Improved baselines with pyramid vision transformer. Comput. Vis. Media, 2022.
[72] Xiaosen Wang, Zeliang Zhang, and Jianping Zhang. Structure invariant transformation for better adversarial transfcrability. In ICCV, 2023.
[73] Yujie Wei, Shiwei Zhang, Zhiwu Qing. Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan. Dreamvideo: Composing your dream videos with cus- tomized subject and motion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6537-6549, 2024.
[74] Yujie Wei, Shiwei Zhang, Hangjic Yuan, Xiang Wang, Haonan Qiu, Rui Zhao, Yutong Feng, Feng Liu, Zhizhong Huang, Jiaxin Ye, et al. Dreamvideo-2: Zero-shot subject-driven video customization with precise motion control. arXiv preprint 410iv:2410.1388 2024.
[75] Chaowei Xiao, Jun-Ya Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In ICLR, 2018.
14

---

[76] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan L. Yuille. Mitigating adversarial effects through randomization. In ICLR, 2018.
[77] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan Yuille. Improving transferability of adversarial examples with input diversity. In CVPR, 2019,
[78] Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. In 25th Annual Network and Distributed System Security Symposium, 2018.
[79] Haotian Xue, Alexandre Araujo, Bin Hu, and Yongxin Chen. Diffusion- adversarial sample generation for improved stealthiness and controllability. In NeurIPS, 2023.
[80] Shengming Yuan, Qilong Zhang, Lianli Gao. Yaya Cheng, and Jingkuan Song. Natural color fool: Towards boosting black-box unrestricted attacks. In NeurlPS, 2022.
[81] Kaiwen Zhang, Yifan Zhou, Xudong Xu, Bo Dai, and Xingang Pan. Diffmorpher: Unleashing the capability of diffusion models for image morphing. In CVPR, 2024.
[82] Yunqing Zhao, Tianyu Pang, Chao Du Xiao Yang, Chongxuan Li, Ngai-Man Man Cheung, and Min Lin. On evaluating adversarial robustness of large vision-language models. NeurIPS, 2024.
[83] Zhengyu Zhao, Zhuoran Liu, and Martha A. Larson. Adversarial color enhancement: Generating unrestricted adversarial images by optimizing color filter. In BMVC, 2020.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly articulate the main contributions: re- framing unrestricted adversarial attacks
adversary preference alignment problem, proposing the two-stage APA framework, and highlighting its superior performance vali- dated through experiments. These claims consistent with the detailed methodology and results presented in the subsequent sections of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide limitations in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper is primarily an empirical work proposing a novel framework and algorithmic approach. does not introduce new theoretical results, theorems, or mathematical derivations requiring formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed experimental settings and pseudocode in the main text. In addition, the source code is included in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
17

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the code in the supplementary material. The datasets used (e.g., ImageNet-compatible dataset) are publicly available, as referenced in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://mips.ca
Ttaiii/gguidess/odeeuieee oww. for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. //nips.ccppubltt wwtpccpwwcpcpdddeeseeppppppppppe http://uuddddeoooouuubmmmiiiaa sondesuumisssioPoll11e
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed implementation details Section 5.1 as well as hyperpa- rameter tuning experiments in Section 5.6
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: While do not explicitly report error bars in the figures tables, have tested our method under different random seeds and across different GPU environments, and observed negligible performance variation.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We discuss it in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss it in the main body.
Guidelines:
19

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release new large-scale pretrained models or datasets that would inherently pose a high risk for misuse. The primary contribution is algorithmic.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We properly credit the original creators of existing assets by citing the relevant publications.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswipphaeeeooccaaaapp thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: material.
We release the source code and model implementation as supplementary
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research presented in this paper does not involve crowdsourcing experi- ments or direct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The research does not involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: method does not involve the use of large language models (LLMs).
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
22

---

Appendix
This appendix is structured as follows:
In Appendix
A
we provide time analysis of our APA.
In Appendix
B
we provide limitations.
In Appendix
we provide pseudo code of our APA.
Appendix D
we provide more diagnostic experiments.
In Appendix we provide more attack methods compared with our method, including unre-
stricted attacks (ADef [T]. ACE [83]. ReColorAdy [35, PPGD [36), LPA (36), AdvDiff (I3)
and Lp attacks (MI [16]. NI 45), DIM (7),, TIM [77.
In Appendix F we conduct extensive experiments to evaluate our method across various diffusion models (e.g., ControlNet (391) and downstream tasks, including targeted attacks, visual question answering. and object detection. Moreover, we further show that stronger data transformations (e.g., [70] 72 used in Ly attacks) further boost performance, underscoring the scalability of our method.
In Appendix G
we provide more visualizations.
A Time Analysis
Most unrestricted attacks, well as method, fall under the image-specific framework. To further demonstrate the advantages of our proposed APA framework, we conduct a comprehensive time efficiency analysis. Our empirical evaluation on an NVIDIA A100 GPU shows that visual consistency alignment requires 38 seconds, while attack effectiveness alignment (APA-SG) takes 58.5 seconds. When combined, our complete APA framework requires total execution time of 96.5 seconds. This represents a significant 16% improvement in computational efficiency compared to the ACA method, which requires 1 14.8 seconds on an NVIDIA A100 GPU.
B Discussion & Limitations
Adversarial examples have been a long-stad standitin research hotspot in the field of artificial intelligence. Previous works have explored various directions to design stronger adversarial examples, such as gradient-based optimization [52], heuristic optimization [32] ddifiusiioon-bbee based methods [10 [I 35] and 19), etc. In contrast, our method is the first to approach unrestricted adversarial attacks from preference alignment perspective. The key idea lies alignment-driven modeling, where we define and quantify malicious preferences (visual consistency and attack effectiveness), select suitable align- ment strategies (e.g., DPO, RI., or direct backpropagation) and build a stable alignment framework (joint decoupled). A two-stage alignment framework is proposed to decouple inherently conflicting preferences- maximizing attack performance under visual consistency constraints apppritts--ppppcchiinn the Pareto frontier. To further stabilize training, dual-path optimization and diffusion augmentation are introduced to mitigate overfitting in direct backpropagation. Extensive experiments demonstrate the effectiveness, fiexibility, and robustness of our approach.
We recognize two primary limitations of our work: 1) While APA achieves greater time efficiency than previous diffusion-based based attacks (e.g., ACA), it is still more computationally expensive than conventional T,p attacks. One of the costs arises from DDIM sampling: thus, integrating faster samplers could further enhance efficiency. 2) This work is limited to diffusion models, and does not explore the applicability of our method to other generative frameworks such as rectified flows [46 20 We plan to investigate this in future work.
C Pseudo Code of APA
We provide pseudo code of our APA framework in Algorithm I and Symbol Table in Table 5
23

---

Algorithm 1:
Our APA Framework
Input: input image æ; prompt c, label y, substitute classifier fo: denoising model co, encoder &, decoder D, inversion step T, attack guidance denoising step Tax Mar 0. Output: unrestricted adversarial example Vade
1: 66+ = CCaallatCoonnnnnnnnee (eaissency_Aigmmme((( C, Eg).
2: 20 =
3: Generate 2r using DDIM Inversion, 24 = 2T.
4: for in [1, N + do
0. V = {}- 6: for in [T, [2.1] do
5:
7: if > Т, then
8:
Calculate 24-1 using DDIM Denoising. else
9:
Calculate z/n using Eq. 11.
11:
9st V (D) (2,))))).))
12:
Mat = m2t + 9ue
c0+20(=t,t,0 T1
13:
VI a4 sgn(mst).
=/ 31 siroT-arrlaaaaai
14:
vāa
= V (z/)
16:
21-1 var- + VI 04 150110000 (2,1,)
17:
end if
18:
end for
19:
r= O.
20:
for =0 in V do
21:
z = r=rt R. (f
+ D((a))))))))
22:
23:
end for
24:
if Skip Gradient then
25:
9hr V Vi To mar = mer + gir l I
26:
else
Ptr 9tr V t Tr,mi mer mer + To
27:
end if
28:
2T +o, (eT jr. sgn(mer)).
29:
end for
30:
return D(20)
Table 5: Symbol Table
Symbol
latent updated Deseription
2y.
iteration, initialized xy
Latent representation obtained by applying DDIM inversion to the original image Intent for T steps
20
VAE-encoded latent of the original image
30
Reconstructed ay obtained by DDIM denoising for T stops
26
Reconstructed latent based on 2s, as defined Eq.
80
<
Interpolation between and
Str
Gradicnt at trajectory level
Momentum at the trajectory level Gradient step
Gat
Momentum al the step level
D More Diagnostic Experiments
D.I. LoRA Rank
Regarding the LoRA insertion strategy, we adhered to the default configuration for LoRA fine-tuning on UNet. Additionally, we investigated the impact of the LoRA rank on VCA performance. Using 50 randomly sampled images (source model: RN-50), we evaluated two aspects: (1) the effect of rank on reconstruction quality in aov-adversarial setting, and (2) the effect of rank on performance under attack (see Table 6). Overall, the results are consistent with intuition: higher ranks improve visual

---

Table 6: The effect of LoRA rank on APA.
w/o Attack
w/ Attack
Rank
CLIP Score
Avg ASR
CLIP Score
Avg ASR
0.88
21.1
0.62
96.2
4
0.97
7.2
0,87
79.2
0.97
6.0
0.89
74.5
16
0.97
5.0
0.91
71.2
Table More attack performance comparison on normally trained CNNs and ViTs. We report attack success rates ASR (%) of each method ("*" means white-box ASR), Avg. ASR refers to the average attack success rate on non-substitute models (black-box ASR).
Medels
Substitte
Model
Attack
CNNs
TranSommers
MN-22
Inc-v3
RN-50
Dense-161
RN-152
EF-b7
MobViT-s
VIT-B
Swin-B
PVT-22
ADef
14.5
6.6
9.0
8.0
7.1
9.8
50.8
9.7
5.1
4.6
8.27
ReColorAdy
37.4
14.7
26.7
22.4
21.0
20.8
96,17
21.5
16.3
10.7
21.94
VobViT-s
PPGD
15.7
7.0
9.4
8.8
15.5
7.2
10.5
100.0*
100.0*
12.5
9.6
5.8
5.5
8.83
APA-SC(Ours)
66.3
21.5
65.0
51.6
APA-GC(Ours)
$8.3
77.1
$6.6
81.2
81.2
78.4
99.47
59.3
61.9
$3.4
77.48
ADer
$6.6
7.6
8.4
7.7
7.1
10.9
11.7
9.5
4,5
4.5
7.99
ReColorAdy
97.7-
18.6
33.7
24.7
26.4
20.7
31.8
17.7
12.2
12.6
22.014
MN-22
PPGD
100.0
99.9-
10.4
11.9
13.5
99.1-
9.5
17.9
12.4
12.6
11.7
12.1
APA-SG(Ours)
99.8
80.4
$3.1
83:0
81.7
78.8
78.5
55.9
39.5
63..4
72.14
APA-GC(Ours)
100
91.4
97.7
95.5
95.0
91.8
93.2
74.3
50.0
85.2
N7.01
ADer
15.5
7.7
55.7*
8.4
7.8
11.4
12.3
9.2
4.6
4.9
9.09
RaColorAdv
33.3
LPA
24.0
34.4
38.0
22.0
29.2
13.5
12.2
25,022
ACE
32.8
9.4
99.19
16.1
15..2
12.7
20.5
13.1
6.1
5.3
14.58
APA-SG(Ours)
80.0
81.4
99.6*
89.5
90.1
22.3
76.7
58.5
45.7
67.6
75.32
APA-GC(Ouss)
97.6
93.5
99.7"
97.6
98.4
91.1
90.9
75.6
63.88
83.7
88,02
15.3
8.3
8.4
7.5
12.0
12.4
$1.5*
5.3
VIT-B
PRGD
LPAA
21.4
10.4
7.5
13.9
12.5
8.3
11.6
7.9
10.3
14.5
16.6
100,0*
99.77
9.1
7.8
13.09
ACE
30.9
11.4
22.0
15.5
15.2
13.0
17.0
95.6"
6.5
6.3
15.31
APA-SG(Ours)
69.3
67.6
67.5
66.8
65.4
20.0
67.6
99.2*
62.5
59.2
66.21
APA-GC(Ours)
77.0
74,8
75.4
75.9
75.4
76.8
74,5
98.4*
73,4
70.2
74,82
fidelity but reduce attack success rates and increase both computational and training costs due to larger number of tunable parameters. Considering this trade-off, we find that a rank achieves a favorable balance.
D.2 Visual Consistency Alignment
Figure qualitatively illustrates the importance of VCA in maintaining visual consistency. The first three rows of debonsteto that VCA improves the reconstruction quality clean samples. We also evaluated the image quality when only attack effectiveness alignment was applied (i.e., without VCA) in Table 7 The noticeable degradation in visual quality in this setting underscores the essential role of VCA in preserving visual consistency during attack alignment.
Table 7: Performance comparison of APA-SG with and without VCA.
Methed
LPIPS
SSIM
CLIP Score
Avg. ASR
who VCA 0.55
0.46
0,62
95.43
VCA
0.25
0.67
0.86
75.32
25

---

Attack Performance Comparison with attacks
RN-50
92
WT-3
01
75.32
66.21
Average
63.64
58.10
54.44
37.11
32
NI
DIM
TIM
APA-SG
APA-GC
Figure 8: Attack performance comparison with Lp attacks. RN-50 and ViT-B as substitute models.
Average ASR refers to the average attack success rate on non-substitute models.
E
More Attack Performance Comparison
E.1
More Unrestricted Attacks
As a supplement to Table the main body, we include a comparison of the attack performance of four additional methods in Table 8 ADef (1). ACE [83]. ReColorAdy (35), PPGD (36], and LPA [36]. against our APA. Our findings demonstrate that, regardless of whether a CNN or ViT model used the substitute model, the transfer attack performance of method significantly surpasses these four methods.
E.2 Ip Attacks.
To further validate the robustness of our method, we compare our method with classical transfer attacks with Ly attacks (Loc 16/ /255): gradient optimization attacks (MI [16). NI (45)) and input transformations attacks (DIM (77). and TIM [17]), as shown in Figure 8 Although this comparison inherently unfair for ours (as unrestricted adversarial examples are more natural), our methods consistently achieve superior black-box transferabilityy especially when using ViT as the substitute model.
E.3 AdyDiff
AdvDiff explores non-traditional form of unrestricted adversarial examples, which cannot specify reference image and instead generates adversarial examples starting from random noise. (AdvDiffuser g adopts a similar approach; however, AdvDiff has demonstrated superior perfor- mance compared to AdyDiffuser, and since the code for AdvDiffuser is not publicly available, we include comparisons exclusively with AdvDiff, considering both the AdvDiff and AdvDiff-Une Untargett versions [3).) To highlight the advantages of our method, we follow the experimental setup of AdvDiff for comparison. Specifically, we first run AdvDiff without adversarial optimization to generate clean, original images. These images then used reference images. The results presented in Table 9 We find that our method demonstrates robust attack performance regardless of whether RN-50 or VIT is used the substitute model. In contrast, AdvDiff shows significant performance discrepancies between RN-50 and VIT. Overall, our method achieves superior attack performance.
Figure / shows the visualizations including our method and AdvDiff. We observe that while AdvDiff- Untarget shows improved attack performance compared to AdvDiff, its image generation quality is significantly lower. In contrast, our method achieves superior performance in both image generation quality and attack effectiveness.
26

---

AdvDiff- Untarget
Clean
DdvD
APA-SG
APA-GC
Figure 9; Visual quality comparison with AdvDiff using RN-50 as the substitute model.
Table 9: Attack performance comparison with AdvDiff and AdvDiff- AdvDiff-Untagee.
Substiture
Model
Allack
ASR (15
Avg.
MN-v2
Inc.>3
RN-50
Demse-16 161
RN-152
BF-87
MobViT-s
VIT-B
Swin-D
PVT-v2
Clean
6.2
7.6
7.3
6.5
6.5
7.2
7.4
6.2
5.7
4.8
6.54
AdvDifl
9.3
100,0*
9.3
8.8
10.5
9.0
9.6
6.5
7.2
8.9
RN-50
MovDiff-Uttrrge
73.1
74.9"
64.9
68.0
AGA-SG
09.2
33.6
49.5
09.22
AdvDif
8.7
9.5
8.1
6.7
10.0
9.3
100.0
8.1
6.9
8.33
VIF-B
AdvDiff Untarget
15.3
17.5
16.1
13.4
15.6
19.3
21.3
53.7
22.1
17.6
18.02
APA-SGIOUrSi A APA-G((Ours)
69.7
52.9
$1.1
52.2
$1.6
36.0
54.2
98.9"
59.1
41.2
$2.86
56.3
80.7
39.5
60.4
39.3
63.5
62.0
80.30
60.0
56.9
60662
F Flexibility and Scalability
F1 ControlNet
To evaluate the flcxibility of our framework, we utilize the Canny and Hed versions of [31 As shown in Figure T0, under the constraints imposed by ControlNet, attack optimization primarily targets texture information outside the contour regions, leaving the overall outline intact. Additionally, Table highhliights that adversarial examples generated using diffusion models with ControlNet maintain strong transfer attack performance.
F.2 Extend to Other Tasks
To assess the flexibility of our method, we evaluate APA's performance across various tasks. Thanks to its high adaptability, transferring APA to new tasks requires only adjusting the corresponding Rg-
Targeted Attacks. We evaluate the feasibility of targeted attacks by specifying target class during attack optimization. Our findings indicate that the explicit designation of the target class in the optimization process leads the generated unrestricted adversarial examples to subtly incorporate features of the target class. For instance, as shown in the right panel of Figure IT Iaa), when the target class set to "hen." the generated image preserves overall structure of the original input (ensured
by our visual consistency alignment) while discreetly embedding patterns resembling hen feathers into the texture of rock. Overall, the targeted adversarial examples generated by our method bear resemblance camouflaged representations of the target class.
27

---

Table 10: Attack performance ControlNet, RN-50 as the substitute model. APA-GC-C denotes that APA utilizes a diffusion model equipped with ControlNet, with gradient propagation implemented through gradient checkpointing.
Models
CNNs
Trunsformers
ASR (%%
MN-v2
Ino-3
RN-50
Dense
RN-152
BF-b7
MobWiT-s
VTF-
Swin-8
PVT-v2
APA-OC-C
96-4
92.0
100.0
97.3
97.9
88.6
88.3
71.8
59.6
81.8
85.96
APA-GC-C (Canny)
Adversarial (Hed)
Adversarial (Canny)
Clean
Hed
Canny
2
Figure 10: Visualization of adversarial examples generated by ControlNet++. Adversarial examples generated under the constraints of ControlNet can preserve the complex structures of clean images, such text.
Object Detection. We further evaluate the attack effectiveness on object detection tasks, as illustrated in Figure b]), using R, defined as the loss function from 44. Our results demonstrate that unrestricted adversarial examples can effectively compromise object detectors by either completely preventing object detection or inducing misclassification of detected objects.
Visual Question Answering. We also investigate targeted attacks on Vision-Language Models (VLMs) 50 49
The goal of this task is to ensure that, given specified target text, the model generates the content of the target text when presented with adversarial examples. Figure (Tc) presents examples of unrestricted adversarial images for VL.Ms generated using APA. The design of R2 based on [82]. where an image first synthesized from the target text, and the cosine similarity between the features of the synthesized image and the adversarial example is computed using the BLIP-2 image encoder [38]. Our findings indicate that, consistent with the observations for targeted attacks, the generated unrestricted adversarial examples guide the diffusion model to subtly incorporate visual features corresponding to the target text into non-salient regions of the image. This enables the VLM to output the target text while disregarding the primary original objects in the image.
F.3 Integration with different e(-)
We further investigate whether stronger data augmentation strategies can enhance the performance of our diffusion augmentation. Specifically, we adopt the data transformation methods used in SIA 772 as augmentation strategies e(-), while keeping all other components unchanged. As shown in Table the results consistent with our findings under Lp attacks: stronger data transformations further
28

---

(a) Targeted Attack
giant panda (b) Attack Object Detector
Target: sca lion
marmot
Target: hen
Clean
Adversarial
Clean
Adversarial
(c) Attack VL.M
Target: coffee cup with heart shape in the foam
Target: two butterflies thistle plant
belt with buckle with a flower on it
coffee cup with heart design on
bald eagle with its wings spread
two butterflies are sitting on a flower
Figure 11: (a) shows targeted attacks on the classification task based on RN-50 model. (b) shows untargeted attacks on the object detect task based on DETR. (c) shows targeted attacks on the visual question answering task based on VLM model BLIP-2.
Table 11: Attack performance on different o(.). RN-50 as the substitute model.
Models
Attack
CNNs
Transformers
Avg.
(8)
MN-v2
Inc-v3
RN-50
Denso-161
RN-152
EF-87
MobViT-s
VIT-B
Swin-B
PVT-22
APA-SG
89.0
83.4
99.6*
89.6
90.1
77.3
76.7
58.5
45.7
67.6
75.32
APA-SG-SIA
90.8
83.4
99.99
91.2
92.6
80.3
82.9
62.9
52.7
72.9
78.86
improve the transferability of our framework. This demonstrates the scalability and extensibility of our framework.
G Visualization
We select the top six attack methods based on their performance in Table Bnd Table 1 of the main body for further visual comparison experiments, as shown in Figure T2
29

---

Clann
VCA
SAE
cAdr
MCF
DillAtcck
ACA
AZA-SG
APA-CC
APA-OC-
Figure 12: Qualitative comparison of image quality. Images generated by RN-50.
30


Reviews:
[{"review_id": "ffco2XaTjV", "reviewer": "Reviewer_a1PE", "summary": "This paper proposes a two-stage framework APA (Adversary Preferences Alignment) to craft unrestricted adversarial examples using diffusion models. For an original image, APA first employs LoRA to fine-tune the UNet in the LDM to improve the reconstruction quality of DDIM inversion to ensure the visual consistency of the adversarial example after perturbing the latent embeddings. Then, during the denoising process of LDM, APA performs adversarial optimization on the latent embeddings through a dual-path strategy at both the trajectory-level and step-level. Additionally, APA introduces Diffusion Augmentation to further enhance the transferability.", "questions": "1. Except for the results in Figure  3, are there any quantitative results or theoretical analyses to support the effectiveness of Visual Consistency Alignment?\n2. Considering the image quality of APA-SG in Figure 4 appears better than that of ACA and DiffPGD-MI, yet their quantitative metrics are comparable, and some results of APA-SG in Appendix H exhibit noticeable artifacts, are the visualization images specially selected? More visualization results of randomly sampled images should be included. Moreover, it is observed that APA-GC-P, which achieves better image quality according to quantitative metrics, suffers a significant drop in average ASR, but its detailed results are not included in Tables 1 and 2.\n3. A comparison with the state-of-the-art LDM-based unrestricted adversarial attack DiffAttack [1] is missing. ([1] Chen J, Chen H, Chen K, et al. Diffusion models for imperceptible and transferable adversarial attack[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.)\n4. It should be clarified whether the APA variant compared with AdvDiff in Appendix E.3 remains under an untargeted attack setting, to avoid potential unfair comparisons, or how APA can be extended to targeted attack scenarios.\n5. Why does the optimization process of APA appear more complex and involve more computation steps than ACA, yet APA runs faster according to Appendix B? The authors should explain this result and provide a corresponding analysis for the computational complexity of APA.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The methodology presented in this paper demonstrates technical novelty. The APA approach is built upon the existing ACA method but incorporates several significant improvements. For example, it cleverly fine-tunes the UNet model parameters to enhance the visual consistency of Unrestricted Adversarial Examples. Additionally, it introduces both trajectory-level and step-level optimization of latent embeddings and diffusion augmentation to improve attack effectiveness.\n\nHowever, the authors seem to overclaim the connection between APA and the concept of Performance Alignment in diffusion models. APA still follows the conventional paradigm of adversarial attacks, where each adversarial example must be generated through a complete gradient-based optimization process (for both UNet and latent embeddings in APA) with a given original image, rather than directly aligning the model once for solving this problem. Moreover, finding better solutions that simultaneously achieve high image quality and strong attack performance has long been a key objective in adversarial attacks. Although certain components of APA are defined as \"rewards\", they are essentially equivalent to the loss functions commonly used in existing adversarial attacks for computing gradients (e.g., cross-entropy loss from surrogate classification models).", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their constructive feedback and for recognizing the technical contributions of our work.\nBelow, we provide point-by-point responses to each comment. \n\n---\n\n**Weakness 1:**  \n*1. APA still follows per-sample gradient-based optimization and does not perform true model-level alignment.*\n\n*2. finding better solutions that simultaneously achieve high image quality and strong attack performance has long been a key objective in adversarial attacks.*\n\n*3. The so-called attack reward are essentially standard loss functions like cross-entropy.*  \n\nThank you for your comments. We would like to clarify that the key difference between APA and conventional attack optimization lies in its **alignment-driven design**. Our framework is motivated by how to:\n- define malicious preferences (*visual consistency* & *attack effectiveness*),\n- quantify them (*rule-based* or *learned rewards*),\n- select alignment strategies (*DPO*, *RL*, or *direct backpropagation*),\n- design task-specific modules (*dual-path optimization* & *diffusion augmentation*).\n\nThese considerations lead to our proposed **two-stage alignment framework**. Below, we address the three concerns you raised in detail:\n\n1. **Instance-level alignment**:  \n   You're right, we perform *per-instance alignment*, not model-level alignment, as a practical choice due to the difficulty of generating unrestricted adversarial examples under conflicting preferences. This approach aligns with *test-time preference alignment* [1].\n\n    [1] Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback, ICML,2025.\n\n2. **No novelty claim on objective formulation**:  \n   We do **not claim** that our objective design is novel. Instead, we extract preferences from adversarial attack literature. Our key contribution is to **decouple** the optimization of these preferences, showing clear advantages over joint optimization.\n\n3. **Reward function is not our contribution**:  \n   Under the direct backpropagation setting, it is indeed mathematically valid to formalize a reward function as a loss. Actually, reward design is not our contribution (discussed in Section 4.2). Our key contributions under this setting lie in **dual-path attack optimization** and **diffusion augmentation**, which are specifically designed to address known issues with direct backpropagation (such as overfitting to the reward model).\n\nIn summary, we believe the **technical similarity between alignment via backpropagation** and **gradient-based optimization in adversarial attacks** may have led to some misunderstanding regarding the novelty of our work. However, the core innovation of APA is the decoupled alignment of two preferences (instead of joint optimization) and the design of task-specific modules that effectively mitigate overfitting issues in direct backpropagation settings. We further support our claims through extensive experiments demonstrating the superiority, flexibility, and robustness of our method.\n\n---\n\n **Question 1:**  \n*Except for the results in Figure 3, are there any quantitative results or theoretical analyses to support the effectiveness of Visual Consistency Alignment?*  \n\nWe sincerely appreciate the reviewer’s careful analysis. To further clarify the role of our proposed **Visual Consistency Alignment (VCA)**, we have conducted two additional ablation studies.\n\n1. To evaluate whether VCA improves the inherent reconstruction ability of diffusion models, we compare DDIM with and without VCA under a non-attacking setting.\n\n    | Method | LPIPS↓ | SSIM↑ | CLIP Score↑ | NIMA AVA↑ | CNNIQA↑ | Avg ASR↓ |\n    |--------|----------|---------|----------------|-------------|-----------|------------|\n    | DDIM| 0.19| 0.73| 0.89| 5.03| 0.63| 22.10|\n    |DDIM+VCA| 0.05| 0.85| 0.97| 5.13| 0.63| 7.60|\n\nThese results suggest that **VCA clearly enhances the reconstruct quality** of diffusion model.\n\n2. We also evaluated the quality of images generated when only the attack effectiveness alignment (i.e., without VCA):\n    | Method | LPIPS↓ | SSIM↑ | CLIP Score↑ | \n    |--------------------|----------|---------|----------------|\n    | APA-SG w/o VCA | 0.55  | 0.46  | 0.62   | \n    | APA-SG  | 0.25 | 0.67 | 0.86 | \n\nThe noticeable degradation in visual quality upon the removal of VCA highlights **its essential contribution to maintaining visual consistency during attack alignment.**\nWe will incorporate all the above findings into the revised paper to better communicate the role and advantages of VCA.\n\n---\n\n **Question 2.1:**  \n*APA-SG’s images in Figure 4 look better than ACA and DiffPGD-MI despite similar metrics, yet Appendix H shows artifacts. Were the visuals selectively chosen? Please provide more random samples.*  \n\n\nThank you for your careful observation. First, we would like to clarify that **we did not deliberately select qualitative examples**. Due to the constraints, we could not include the full set of test results. However, you can reproduce them using the provided code in the supplementary materials. We will release all results, including those of the baselines, upon acceptance of the paper.\n\nIn addition, **we believe that our quantitative analysis is consistent with the qualitative findings.** For instance, DiffPGD-MI introduces noticeable background perturbations while preserving image structure, leading to high SSIM but low aesthetics. ACA's low CLIP score reflects its tendency to alter image semantics. In contrast, APA-SG mainly modifies the background while preserving the subject, producing more natural outputs and thus a higher aesthetic score.\n\n---\n\n **Question2.2 and Question3:**  \n*1. Could the authors clarify why the detailed results of APA-GC-P, which appear to offer better image quality but lower ASR, are not included in Tables 1 and 2?*\n\n*2. Comparison with the state-of-the-art LDM-based attack DiffAttack [1] seems to be missing.\"*  \n\nThank you for your valuable suggestions. We have added detailed results for **APA-GC-P** and **DiffAttack**.  \nFor (a) and (b), we report the **Avg. ASR** due to space limitations. The complete results will be included in the final version.\n\n**a. Attack Performance Comparison (Table 1)**\n\n| Source model  | DiffAttack | APA-SG | APA-GC | APA-GC-P |\n| --- | --- | --- | --- | --- |\n| MobViT-s | 70.02 | 65.85 | 77.48 | 52.56 |\n| MN-v2 | 59.05 | 72.14 | 87.01 | 56.67 |\n| RN-50 | 65.10 | 75.32 | 88.02 | 62.08 |\n| ViT-B | 45.06 | 66.21 | 74.82 | 45.90 |\n| Average | 59.79 | 69.88 | 81.83 | 54.30 |\n\n**b. Attacks on Adversarial Defense (Table 2)**\n| Attacks | Avg ASR |\n| --- | --- |\n| DiffAttack | 37.17 |\n| APA-SG | 63.59 |\n| APA-GC | 70.20 |\n| APA-GC-P | 39.26 |\n\n**c.  Visual Quality Comparison (Table 3)**\n\n| Method |  LPIPS↓ | SSIM↑ | CLIP Score↑ | NIMA AVA↑ | CNNIQA↑ | Avg ASR↑ |\n| --- | --- | --- | --- | --- | --- | --- | \n| DiffAttack | 0.14 | 0.68 | 0.87 | 5.17 | 0.66 | 65.10 | \n| APA-SG | 0.25 | 0.67 | 0.86 | 5.29 | 0.62 | 75.32 | \n| APA-GC | 0.23 | 0.69 | 0.83 | 5.39 | 0.67 | 88.02 | \n| APA-GC-P | 0.09 | 0.82 | 0.91 | 5.22 | 0.63 | 62.08 | \n\n**Conclusion:**\n\n1. Our APA-SG and APA-GC **achieve 10.09 and 22.04 higher attack performance than DiffAttack, respectively.**\n\n2. Under defense scenarios, **APA-GC, APA-SG, and APA-GC-P all demonstrate better robustness compared to DiffAttack.**\n\n3. APA-GC achieves comparable visual similarity and aesthetic scores to DiffAttack. Moreover, **APA-GC-P exhibits significantly better visual quality than DiffAttack with comparable attack performance.** We will include qualitative examples in the main paper as well (we observed that images generated by DiffAttack often have overly sharp object boundaries).\n\n4. **These results further demonstrate the superiority of our two-stage design in APA over the joint optimization used in DiffAttack,** enabling more effective and controllable generation.\n\n5. As noted, APA-GC-P offers superior visual quality but moderate attack effectiveness. We include it to showcase the flexibility of our framework in supporting task-specific optimization choices.\n\n---\n\n**Question4:**  \n*It should be clarified whether the APA variant compared with AdvDiff in Appendix E.3 remains under an untargeted attack setting, to avoid potential unfair comparisons, or how APA can be extended to targeted attack scenarios.*  \n\nThank you for your insightful comments. As stated in its original paper('Evaluation metrics' on page 10), both AdvDiff and AdvDiff-untarget (which enhances transferability) are intended for evaluating **untargeted attacks.** Accordingly, **all comparisons were performed under the untargeted setting, with fairness across all baselines strictly maintained.** Moreover, extending APA to targeted attacks is straightforward—one only needs to replace the optimization objective accordingly. Please refer to lines 612–619 for implementation details.\n\n---\n\n**Question5:**  \n\n*APA appears more complex and involves more computation steps than ACA, yet runs faster according to Appendix B*  \n\nThank you for your suggestion. We identified two contributing factors. First, **our VCA optimization is significantly faster than APA’s null text optimization (e.g., 38 s vs. 55 s)**. Second, our Stage-2 alignment is implemented on a lightweight codebase based on Diffusers, whereas ACA relies on a heavier framework, which introduces slight overhead. You may verify this using the code provided in the supplementary zip. **Our codebase is clean and modular—easily integrable into Diffusers, with flexible parameter selection and customizable optimization strategies.**\n\nRegarding the complexity analysis:\nStage 1 has a time complexity of **O(n)** (number of training steps),\nwhile Stage 2 is **O(N·T)** (attack iterations × denoising steps).\n\nOnce again, thank you for your time. We sincerely hope our response can address your concerns."}, {"review_id": "kLimRalBN1", "reviewer": "Reviewer_sWRo", "summary": "This paper introduces the Adversary Preferences Alignment (APA), an innovative two-stage framework designed to enhance diffusion-based unrestricted adversarial attacks. Unlike typical benign preference alignments that optimize for human-preferred attributes, APA addresses adversarial preferences by balancing two conflicting goals: visual consistency and attack effectiveness. The authors propose decoupling these conflicting objectives, initially optimizing visual consistency through LoRA fine-tuning with a rule-based visual similarity reward, followed by optimizing attack effectiveness using dual-path attack guidance and diffusion augmentation.", "questions": "1. In the paper, you conducted evaluation on existing defenses which originally focusing on Lp attacks, but are you also considering defenses method on unrestricted adversarial attacks?\n\n2. In Eq.7, it introduces momentum-based gradients, but $\\Pi z^0_T + \\epsilon_a$, what this refer to? Can you provide more details?\n\n3. Line 164 ~166, you mentioned regarding the shift of similarity metric from the pixel/image space (Eq. 5) to the latent space (Eq. 6) for computational efficiency. However, the paper does not provide sufficient empirical comparison to support the claim that latent similarity adequately preserves visual consistency, which is critical to the proposed method. Can you possibly explain more on this or provide more context?\n\n4.  The “skip gradient” and “gradient checkpointing” strategies are described briefly; their impact on optimization stability and convergence, I think is not well justified or analyzed. Are there convergence guarantees ?", "limitations": "The authors have provided the limitations in Appendix C.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n\n- Concept of aligning diffusion models to adversary preferences rather than benign human preferences\n- I appreciate that the author conduct comprehensive evaluation across multiple CNNs and Vision Transformers\n- APA significantly outperforms prior methods in maintaining visual consistency and naturalness, supported by rigorous quantitative  (LPIPS, SSIM, CLIP Score) and qualitative assessments.\n- The authors provided the discussion regarding the ablation of visual consistency alignment  and gradient backpropagation.\n\n## Weakness:\n- Although the 2-stage method is novel, multi-component optimization process may introduce complexity in real-world implementation. Possibly further simplification or integration techniques could be beneficial.\n- The paper frequently uses $z_T$, $z_0$, $\\bar{z}_0$, $\\hat{z}_0$, $z^0_t$, $z^{in}_t$, etc., sometimes with and sometimes without bars, hats, or superscripts. It's easy for the readers to lose track of what each variable specifically represents—particularly the difference between “trajectory-level” and “step-level” variables. Maybe you could provide a summary table of symbols.\n- And next on equations. The reader is left piecing together the order and purpose of updates without a clear, step-by-step guide. For example: \n   - In Eq. 6, the switch from image similarity to latent similarity is abrupt; justification for why latent-space similarity is a good surrogate for visual consistency is not well articulated.\n- Regarding the explanation of methods and strategies, I believe you need to provide more clarification. For example, details on LoRa Fine-Tuning, to be more precise, how LoRa modules are inserted.", "was_revised": true, "final_justification": "- **Focus on My Original Concerns:**  \n  My main concerns were:  \n  1) Practical deployment complexity due to the multi-component structure.  \n  2) The need for stronger empirical justification for latent-space similarity as a surrogate for visual consistency.  \n  3) Improving clarity of notation and optimization flow.  \n\n  The authors’ rebuttal addressed these well: they committed to restructuring the methodology section, adding a concise pipeline overview, and distinguishing core vs. optional components to reduce perceived complexity. They also reiterated the rationale for latent-space similarity and linked it to visual consistency, while promising clearer integration in the final version. These commitments address the core of my concerns, although the improvements will only be realized in the camera-ready.\n\n- **Consideration of Other Reviewers’ Concerns:**  \n  I have reviewed the other reviewers’ comments and the authors’ responses. Key issues raised — such as organization, necessity of components, computational efficiency, and the framing of the method — were all answered with detailed clarifications. The authors not only justified their design choices (e.g., direct backpropagation vs. RL/DPO) but also provided quantitative evidence for efficiency trade-offs and acknowledged the inverse-problem perspective.\n\n- **Assessment of Authors’ Responsiveness:**  \n  The rebuttal was comprehensive, technically precise, and showed a willingness to integrate feedback from all reviewers. The level of detail and empirical support provided in their clarifications gives me confidence that the final version will substantially improve in clarity and presentation.\n\n- **Remaining Issues:**  \n  The dense presentation of the original submission remains a limitation until the promised revisions are implemented. I also believe more explicit empirical connection between latent-space similarity and visual consistency would further strengthen the paper.\n\n- **Recommendation Rationale:**  \n  Given the strong technical contributions, thorough experimental results, and the authors’ demonstrated ability to address feedback from multiple reviewers, I recommend acceptance while keeping my original score to reflect that the current submission still relies on future revisions for full clarity.", "rebuttal": "We sincerely thank the reviewer for their constructive feedback and for recognizing the novelty of our method and the thoroughness of our experiments. Below, we provide point-by-point responses to each comment. \n\n---\n\n**Weakness 1:**  \n*Although the 2-stage method is novel, multi-component optimization process may introduce complexity in real-world implementation. Possibly further simplification or integration techniques could be beneficial.*  \n\nThank you for your valuable comment. We understand the concern regarding potential complexity in real-world implementation due to the multi-component design. We would like to clarify that our method is straightforward to implement and highly modular, as outlined below:\n\n**Simple and Modular Implementation:**\nUnlike prior approaches that tightly couple objectives into a single optimization loop, our method is implemented as a clean two-stage pipeline. The visual consistency alignment is based on **lightweight** LoRA fine-tuning, and after merging the LoRA parameters into the UNet, **no additional parameters are introduced for the subsequent attack alignment.** The attack alignment is seamlessly integrated into the denoising process via **plug-and-play modules** (i.e., dual-path attack optimization + diffusion augmentation). The entire pipeline is fully **compatible with the widely-used Diffusers library, requiring only a few lines of code to integrate with existing workflows.**\n\n**Low Integration Cost:**\nDespite its modularity, the default configuration works well across various settings, **requiring minimal hyperparameter tuning.** For most use cases, users only need to specify the gradient computation method (e.g., skip-step or gradient checkpointing), with the rest handled by default wrappers. **The full method runs efficiently on a single RTX 3090 GPU without special hardware or distributed training.**\n\nWe will revise the paper accordingly to make the simplicity of our implementation more transparent. Specifically, **we will add a structural overview and an integration guide in the Appendix**.\n\n---\n\n**Weakness 2:**  \n*provide a summary table of symbols*  \n \nThank you for your comment. We have provided a detailed symbol table to aid your understanding.\n\n### Symbol Table\n\n| Symbol       | Description |\n|--------------|-----------------------------------------------------------------------------|\n| $z_T$        | The latent updated at each iteration, initialized as $z_T^0$               |\n| $z_T^0$      | Latent representation obtained by applying DDIM inversion to the original image latent for T steps |\n| $z_0$        | VAE-encoded latent of the original image                                    |\n| $\\bar{z}_0$  | Reconstructed $z_0$ obtained by DDIM denoising for T steps                  |\n| $z_0^t$      | Reconstructed latent based on $z_t$, as defined in Equation (9); $z_0^0 = \\bar{z}_0$ |\n| $z_{in}^t$ | Interpolation between $z_0$ and $z_0^t$                                 |\n| $g_{tr}$ | Gradient at the trajectory level                                         |\n| $m_{tr}$ | Momentum at the trajectory level                                         |\n| $g_{st}$ | Gradient at the step level                                               |\n| $m_{st}$ | Momentum at the step level                                               |\n\nA more detailed description is provided in Appendix D, which can be best understood together with the accompanying pseudocode.\n\n---\n\n**Weakness 3 & Question 3:**  \n*In Eq. 6, the switch from image similarity to latent similarity is abrupt; justification for why latent-space similarity is a good surrogate for visual consistency is not well articulated.*  \n\nRegarding the shift from pixel-space similarity to latent-space similarity, we appreciate your observation. **Our design follows the motivation outlined in the LDM paper[1]**, particularly the concept of perceptual compression—that is, most pixel-level information is redundant, and perceptually equivalent representations can be obtained in the latent space through a trained autoencoder (see Sec. 3.2 and Fig. 2 in the original paper). Therefore, we perform consistency optimization in the latent space, which preserves high-level semantics while improving training efficiency and stability. Although this substitution is not strictly equivalent, **it is a common practice in the current image generation field.**\n\nTo further assess the perceptual loss incurred by using latent representations, we conducted the following experiment:\n\n**VAE:** Reconstruction quality of the input image using only the VAE encoder-decoder.\n\n**VAE + DDIM:** Standard reconstruction quality using the full diffusion model pipeline.\n\n**VAE + DDIM + VCA:** Reconstruction quality after incorporating our Visual Consistency Alignment (VCA) module.\n\n| Method |  CLIP Score↑ | NIMA AVA↑ | Avg ASR↓ |\n|--------|----------|---------|----------------|\n|   VAE    |  0.97           | 5.16        | 7.00       |\n| VAE+DDIM  | 0.89              | 5.03        |  22.10      |\n|   VAE+DDIM+VCA     | 0.97           | 5.13          | 7.60       |\n\nWe found that the VAE itself introduces only a negligible perceptual degradation (with a 0.03 drop in CLIP score and a 0.17% increase in Avg ASR). Furthermore, we observed that **incorporating our VCA effectively compensates for the perceptual loss caused by denoising, reaching a level comparable to that of VAE compression.**\n\nMoreover, **another advantage of compressing into the latent space is that optimizing the latent-to-pixel mapping does not introduce noticeable noisy perturbations**, whereas direct optimization in the pixel space often does (as seen with DiffPGD in Figure 4).\n\n[1] Rombach R. et al., \"High-resolution image synthesis with latent diffusion models,\" CVPR, 2022.\n\n---\n**Weakness 4**  \n*Details on LoRa Fine-Tuning, to be more precise, how LoRa modules are inserted.*  \n\nThank you for your rigorous review. As stated in Appendix F (lines 597–599), during the visual consistency alignment phase, we fine-tune only the projection matrices Q, K, and V in the UNet’s attention modules. We set the LoRA rank to 8 and train for 200 steps using the AdamW optimizer with a learning rate of 1×10⁻⁴. Before performing attack effectiveness alignment, we simply merge the LoRA parameters into the UNet, introducing no additional parameter overhead. For more implementation details, please refer to the source code included in the supplementary zip file.\n\n---\n**Question 1**  \n*In the paper, you conducted evaluation on existing defenses which originally focusing on Lp attacks, but are you also considering defenses method on unrestricted adversarial attacks?*  \n\nThank you for your careful feedback. The defense methods **Res-De (a model trained to reduce shape and texture bias) and Shape-Res (a model trained to increase shape bias for improved robustness)** in Table 2 are specifically designed for unrestricted adversarial examples, as introduced in line 273. Compared to prior work, our defense evaluation is more comprehensive, covering adversarial training (for both CNNs and ViTs), input preprocessing defenses, diffusion-based defenses, and defenses tailored to unrestricted attacks. However, as shown in Table 2, existing methods still struggle to defend against our strong adversarial examples, **which highlights a clear gap and motivates future research in this direction.**\n\n---\n**Question 2**  \n*In Eq.7, it introduces momentum-based gradients, but $\\Pi_{z^0_T+\\epsilon_a}$ , what this refer to? Can you provide more details?*  \n\nIn the adversarial attack domain, $\\Pi_{z^0_T+\\epsilon_a}$ is commonly defined as restricting the modification range of $z^0_T$ within $[ z^0_T-\\epsilon_a, z^0_T+\\epsilon_a ]$. Its purpose is to prevent the optimized $z_T$ from deviating too far from the original $z_T^0$. We will restate this more explicitly in the final version. \n\n---\n**Question 4**  \n*The “skip gradient” and “gradient checkpointing” strategies are described briefly; their impact on optimization stability and convergence, I think is not well justified or analyzed. Are there convergence guarantees ?*  \n\nThank you for your rigorous review. To ensure a fair comparison with prior work, we fixed the attack iteration number \\( N \\) to 10 steps and therefore did not include an explicit convergence analysis initially. Theoretically, gradient checkpointing yields more accurate gradient estimates than skip-gradient methods, which is supported by our experimental results showing that APA-GC outperforms APA-SG in attack effectiveness.\n\nWe  supplement  a convergence analysis as follows:  \nFirst, we use the white-box ASR as an indicator of convergence. In adversarial attacks, accurate gradients should result in a steadily increasing white-box ASR, reflecting effective optimization.  \nSecond, we randomly sampled 100 images and recorded the reward values over 15 iterations to observe convergence behavior. The results are summarized in the table below:\n\n| Method  | 2    | 4    | 6    | 8    | 10   | 12   | 14   | White-box ASR (%) |\n|---------|-------|-------|-------|-------|-------|-------|-------|-------------------|\n| APA-SG  | 5.32  | 7.19  | 9.10  | 10.86 | 12.27 | 12.94 | 13.16 | 99.6 |\n| APA-GC  | 6.67  | 10.27 | 13.39 | 16.20 | 18.81 | 19.51 | 20.31 | 99.7  |\n\nWe observe a rapid increase in reward from steps 2 to 8, followed by slower growth from steps 10 to 14. Together with the high white-box ASR at step 10, this suggests the optimization essentially converges within 10 steps. Moreover, APA-GC shows a clearly faster reward increase than APA-SG. Due to certain methodological constraints, we cannot provide a clearer convergence curve at this stage, but we will include it in the final version of the paper.\n\nOnce again, thank you for your time. We sincerely hope our response can address your concerns."}, {"review_id": "SWsVAuC8gh", "reviewer": "Reviewer_ELVx", "summary": "This paper proposes APA, a novel framework reframing unrestricted adversarial example generation in diffusion models as alignment with adversary preferences. Addressing the conflicting objectives of visual consistency and attack effectiveness – which often cause unstable optimization and reward hacking – APA employs a two-stage approach: first fine-tuning LoRA for visual quality using rule-based rewards, then optimizing image latents or prompt embeddings using substitute classifier feedback with trajectory-level and step-wise rewards. Enhanced by diffusion augmentation for transferability, APA achieves significantly better black-box attack success while maintaining high visual consistency.", "questions": "1. While the approach presents an interesting exploration of unrestricted attacks, the connection to adversary preference alignment appears tenuous. Consequently, the paper's organization makes it difficult to grasp the necessity of the numerous proposed techniques. The authors are adviced to **significantly improve the presentation clarity**. For reference, my understanding is that the attack comprises two core modules:\n\n  - Optimizing $z_T$ for attack effectiveness. Since diffusion model is instable near $z_T$, LoRA-based training is used to enhance visual consistency.\n\n  - Solving an inverse problem (maximizing adv. loss) via denoising path guidance, similar to DPS[1]. \n\n2. Although APA-GC demonstrates the best performance, the authors provide no analysis of its computational efficiency. It is crucial to compare the performance of different attack algorithms under equivalent computational budgets. For example, the authors can include results using standardized metrics, such as ASR plotted against computation time (e.g., wall-clock time or FLOPs), with time/compute on the x-axis and ASR on the y-axis.\n\n[1] Diffusion Posterior Sampling for General Noisy Inverse Problems. (https://arxiv.org/abs/2209.14687)", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. This paper presents APA, an effective framework addressing the fundamental trade-off between visual consistency and attack effectiveness in adversarial example generation.\n2. The experiments are thorough and well-designed, providing convincing evidence for the algorithm’s effectiveness and superiority.\n\nWeaknesses：\n\n1. The paper’s organization is challenging to follow, with an abundance of complex design choices that obscure the core methodology and hinder readability.\n2. The framework’s high time overhead significantly limits its real-world applicability.", "was_revised": true, "final_justification": "The clarity of the presentation and the time overhead of the methods are not ideal, but this is not a fatal flaw.", "rebuttal": "We sincerely thank the reviewer for their constructive feedback and for recognizing the effectiveness of our method and the thoroughness of our experiments. Below, we provide point-by-point responses to each comment. \n\n---\n\n**Weakness 1 & Question 1.2:**  \n*The paper’s organization is challenging to follow, with an abundance of complex design choices that obscure the core methodology and hinder readability. Consequently, the paper's organization makes it difficult to grasp the necessity of the numerous proposed techniques. The authors are adviced to significantly improve the presentation clarity.*  \n \nWe thank the reviewer for the feedback. We will revise the manuscript to emphasize how each serves a distinct role in the alignment process, and de-emphasize optional parts to reduce perceived complexity, primarily in the following aspects:\n\n1. **Clarifying the core methodology**  \nWe will restructure Section 4 to clearly present our two-stage alignment framework at the outset and add a concise overview summarizing the core pipeline.  \n- **Stage 1** aligns visual consistency by encoding the input image into LoRA parameters.  \n- **Stage 2** enhances attack transferability through dual-path optimization and diffusion augmentation.\n\n2. **Reducing perceived complexity**  \nWhile our framework includes several components, we will clearly distinguish between core modules (e.g., alignment stages, dual-path attack optimization) and optional extensions (e.g., different optimization parameters, gradient strategies). To improve readability, we will include a brief summary of these in the paragraph starting at line 230.\n\n3. **Highlighting simplicity and extensibility**  \nOur framework is easy to implement with minimal modifications to Hugging Face’s Diffusers library. It is decoupled from specific models and tasks, making it generalizable to a wide range of downstream applications.\n\nWe believe these revisions will significantly improve the paper’s clarity and accessibility. \n\n---\n\n**Weakness 2:**  \n*The framework’s high time overhead significantly limits its real-world applicability.*  \n\n\nWe thank the reviewer for the concern regarding time cost, which we have acknowledged as a limitation in our paper. However, we would like to clarify two key points. First, our method offers a relative efficiency advantage over existing approaches. Second, in current applications of unrestricted adversarial examples, the primary focus lies in attack performance and image quality, with efficiency being a secondary yet desirable attribute. We further elaborate on this issue from four perspectives:\n\n**1. Competitive Efficiency Compared to SOTA**\n\nAs reported in Appendix B, our method achieves approximately **16% speedup** over the SOTA method ACA on a single A100 GPU, while significantly outperforming ACA in terms of **attack success rate and visual quality** (see Tables 1,2,3). This demonstrates a favorable trade-off between efficiency and performance.\n\n\n**2. Real-World Applicability Is Primarily Offline**  \n\nUnrestricted adversarial examples serve two key real-world purposes: **(1) as a tool for adversarial robustness evaluation, particularly under transfer-based attacks;** and **(2) as hard samples for data augmentation（recent studies[1,2] have shown that hard samples generated by diffusion models can substantially improve downstream model performance when used for training)**. Importantly, both applications are conducted offline and thus impose no strict real-time constraints on the generation process.\n\nIn summary, the practical value of unrestricted adversarial examples lies in their ability to stress-test and enhance models. Their generation time is largely irrelevant in these contexts, as performance—not speed—is the primary concern.\n\n\n[1]ASAM: Boosting Segment Anything Model with Adversarial Tuning, CVPR,2024\n\n[2]Synthesizing Near-Boundary OOD Samples for Out-of-Distribution Detection, ICCV,2025\n\n**3. Acceleration Strategies**\n\nThe main overhead stems from attack effectiveness alignment, particularly the attack iterations and denoising steps during diffusion sampling. Thanks to the strong performance of our method, we can further reduce the number of attack iterations to improve efficiency (see Question 2 for details). Moreover, more efficient sampling strategies may offer further acceleration opportunities.\n\nWe have implemented a parallel version of our pipeline to support **batch-wise optimization** for attack alignment, substantially improving the efficiency of generating batched adversarial examples. The updated code will be released upon paper acceptance.\n\n\n\n\n---\n\n**Question 1.1:**  \n*While the approach presents an interesting exploration of unrestricted attacks, the connection to adversary preference alignment appears tenuous.*\n\nThank you for your comments. We would like to clarify that the key distinction between our APA and prior unrestricted attacks lies in our **alignment-driven modeling**. Specifically, our framework is motivated by the need to:\n\n- define malicious preferences (*visual consistency* and *attack effectiveness*),  \n- quantify these preferences via *rule-based* or *learned* reward functions,  \n- select appropriate alignment strategies (*DPO*, *reinforcement learning*, or *direct backpropagation*), \n- define stable alignment framework (*joint* or *decouple*) \n- design task-specific modules such as *dual-path optimization* and *diffusion augmentation*.\n\nThese components are unified in our proposed two-stage alignment framework, where the central novelty lies in **decoupling the alignment of inherently conflicting preferences**—i.e., maximizing attack performance within the solution space constrained by visual consistency, thereby approaching closer to the Pareto frontier. Furthermore, *dual-path optimization* and *diffusion augmentation* are specifically introduced to mitigate overfitting issues commonly observed in direct backpropagation. We validate the effectiveness, flexibility, and robustness of our method through extensive experiments.\n\n---\n\n**Question 1.3:**  \n*Solving an inverse problem (maximizing adv. loss) via denoising path guidance, similar to DPS.*  \n\nThank you for your suggestion. We carefully compared DPS with our method and found they address fundamentally different goals. DPS solves an inverse problem focused on reconstruction quality, whereas our attack alignment targets the generation of transferable adversarial examples, where reconstruction quality is already ensured by visual consistency alignment.\n\nThe only loose connection lies in using gradient-based guidance: DPS leverages reconstruction loss gradients, while our dual-path guidance is tailored for adversarial alignment. **Notably, such guidance originates from classifier-guided diffusion, which we already compare against—along with its stronger variant, Upainting—both being conceptually closer to our method than DPS.**\n\nAs shown in Figure 5(b), our dual-path guidance, incorporating a cleaner latent reference ($z_{in}$ in Eq. 10) and momentum accumulation, yields significantly better attack performance. We will include a discussion on DPS in the final version. \n\n---\n\n**Question 2:**  \n*Although APA-GC demonstrates the best performance, the authors provide no analysis of its computational efficiency. It is crucial to compare the performance of different attack algorithms under equivalent computational budgets. The authors can include results using standardized metrics, such as ASR plotted against computation time (e.g., wall-clock time or FLOPs), with time/compute on the x-axis and ASR on the y-axis.*  \n\n\nThank you for your suggestion. First, we have already analyzed the time efficiency of our method compared to the state-of-the-art ACA in Appendix B. Under the same number of attack iterations and denoising steps, our method achieves **higher efficiency** and **significantly better performance** than ACA.\n\nWe fully agree with your point. As discussed earlier, the primary time cost of alignment-based attacks stems from the attack optimization and the diffusion denoising process. **Since the number of attack iterations and denoising steps directly determines the number of forward/backward passes through the UNet, they serve as reliable proxies for overall compute cost.** We therefore analyze both components separately to explore whether efficiency can be further improved without compromising performance.\n\n**1. Attack Iteration**\n\nAlthough we are unable to include images and figures, we report the transfer success rate of our method across different numbers of attack iterations (see the table below). For reference, ACA achieves **59.49 Avg.ASR**. \n| Iterations| 1| 2| 3| 4| 5| 6| 7| 8| 9| 10|\n|----------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| APA-SG| 27.96| 37.73| 44.62| 51.26| 57.42| 62.25| 66.03| 69.53| 72.48| 75.32|\n| APA-GC|  25.56|  40.22|    51.82|   61.09|    68.86|   74.88|   79.83|   83.76|   87.01|    88.02\n\nOur **APA-SG** surpasses this benchmark within just **6 iterations**, and **APA-GC** within only **4**. This demonstrates that, thanks to the superiority of our approach, we can significantly reduce the number of iterations while still achieving state-of-the-art attack performance. **In the final version, we will convert the above table into a curve plot.**\n\n**2. Denoising Process**\n\nAs shown in Figure 6(a) and 6(c) of the main paper, the number of denoising steps can also be further reduced. For example, using only **5 denoising steps** (Figure 6a), our method achieves a **1.7× speedup** while still reaching a success rate of **87.14**, which remains substantially higher than ACA.\n\nOnce again, thank you for your time. We sincerely hope our response can address your concerns."}, {"review_id": "1aOO0t0WLP", "reviewer": "Reviewer_KeyC", "summary": "The paper introduces APA (Adversary Preferences Alignment), a novel two-stage framework for generating unrestricted adversarial examples using diffusion models. The key contributions are: \n1. Reformulating adversarial attacks as a preference alignment problem with conflicting objectives (visual consistency vs. attack effectiveness); \n2. A decoupled approach: Stage 1 uses LoRA fine-tuning with rule-based rewards for visual consistency, Stage 2 uses dual-path attack guidance with diffusion augmentation for attack effectiveness.", "questions": "- How do we get equation 6 (page 4 line 168), what is the definition of \n$E_{t, \\epsilon}$? I think the loss function should be \n\n$$\\mathbb{E}_{t \\sim [1, T], \\epsilon \\sim \\mathcal{N}(0, \\mathbf{I})} \\left( - \\| \\epsilon - \\epsilon_{\\theta + \\Delta \\theta}(z_t, t, c)\\|^2 \\right).$$\n\nIt is a little ambiguous to $(E_{t,\\epsilon}) - (\\| \\epsilon - \\epsilon_{\\theta + \\Delta \\theta}(z_t, t, c)\\|^2)$.\n- In equation (7), why do we use the L1 norm on $g_{tr}$? As the same norm is used in ACA, I am wandering what is the purpose of using L1 norm instead of L2 norm.\n- The manuscript claims that the proposed method can achieve closer Pareto optimality, but the detail discussion is missing. Please provide more details.\n- Since it is not clear whether one image is applied or a batch of images is applied to the purposed method. Various thanks for providing the code. After reading the code provided in the attachment, it seems that you fine-tune one LoRA for each Image, and attack one image using the pretrained LoRA on that Image. Does my understanding correct? I think this is horrible. \n  - It is super computational expensive on getting one adversarial example. \n  - It is not clear whether the fine-tuned LoRA on one image can work on other images.\n  - I am not sure weather the good performance is due to something like over-fitting, and the model is highly optimized for one specific image.\n- The manuscript had discussed the one-stage and two-stage alignment in page 9 line 320 to 326. It is not clear the performance of the alignment without LoRA. Please consider providing more details.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n- Novel on consider both human preference alignment and adversarial preference alignment.\n- Rich experiments on various attacks and models.\n\n**Weaknesses:**\n- Heavy computation cost on per-image LoRA fine-tuning.\n- Limited discussion on why the proposed method can achieve closer Pareto optimality.\n- Some core operations are from references papers. For example, equation (7) is following ACA, and skip gradient also mentioned in ACA, and gradient checkpointing is widely used in network tuning. Although these are correctly cited, it would raise the concern of novelty.", "was_revised": true, "final_justification": "All my questions are clearly answered.", "rebuttal": "We sincerely thank the reviewer for their constructive feedback. Below, we provide point-by-point responses to each of the comments.\n\n---\n\n**Weakness 1 & Question 4:**  \n*1. Heavy computation cost on per-image LoRA fine-tuning.*\n\n*2. It is not clear whether the fine-tuned LoRA on one image can work on other images.*\n\n*3. I am not sure weather the good performance is due to something like over-fitting, and the model is highly optimized for one specific image.*  \n\n\nThank you for the reviewer’s detailed questions. **We would like to emphasize that our ultimate goal is to generate adversarial examples with strong transferability and high visual consistency. The use of LoRA serves merely as a means to achieve visual consistency alignment.** To address your concerns comprehensively, we provide clarifications from the following four perspectives:\n\n**1). Regarding the computational cost of per-image LoRA fine-tuning**  \na. Visual consistency alignment (VCA) is relatively lightweight, we update only a very small number of LoRA parameters (<1% of full parameters ), while keeping the backbone model frozen.  \n\nb. On an A100 GPU, fine-tuning for a single image takes approximately **38 seconds**. In comparison, the null-text optimization used in the previous state-of-the-art **ACA** method takes around **55 seconds**. \n\nc. VCA only requires around **6GB** of GPU memory, making it a lightweight module with minimal resource demands. \n\nOverall, we believe that the computational cost is acceptable given the significant performance VCA gains (please see R#a1PE Question 1).\n\n**2). Regarding whether the fine-tuned LoRA can transfer to other images**  \nOur method is fundamentally an **image-specific adversarial attack framework**, which optimizes a fine-tuning module (LoRA) for a given input image to preserve its high-level semantics.  As such, we do **not** aim for generalization across entirely different images.\n\nUnder this setting, LoRA is inherently **tailored** to each image and is **not intended to be transferable** to arbitrary inputs — which aligns with our design objective.\n\nNevertheless, we believe that good transferability should exist among **semantically similar images**. To validate this, we conducted the following experiment:\n\n- We randomly selected 5 video clips from **UCF101** video dataset.\n- From each clip, we extracted **16 consecutive frames**.\n- We trained a LoRA on the **first frame** of each clip using our **VCA**.\n- Then we applied **the same LoRA** to subsequent frames to evaluate its reconstruction ability on semantically similar images.\n\nThe results are shown below:\n\n| Type| LPIPS↓ | SSIM↑ | CLIP Score↑ |\n|-------------------------------------|--------|-------|--------------|\n| First frame w/ first frame VCA  | 0.05   | 0.91  | 0.96 |\n| Whole video w/ first frame VCA | 0.06   | 0.90  | 0.94|\n| Whole video w/o first frame VCA | 0.21   | 0.79  | 0.81|\n\nWe observe that the LoRA fine-tuned on the first frame generalizes well to the subsequent frames, suggesting that our VCA module exhibits good generalization across images with similar semantics. This observation indicates a potential future optimization: grouping batch data by semantic similarity and performing VCA once per group, rather than per image.\n\n\n**3) Regarding concerns about “overfitting”**\n\nWe believe this concern stems from a misunderstanding: that overfitting the LoRA parameters to a single image would compromise the generalization of the attack.\n\nIn fact, our method fine-tunes the LoRA module in an image-specific manner to preserve semantic consistency with the input. The generalization, however, arises from the adversarial image itself, not from the LoRA parameters.\n\nSpecifically, we first align visual semantics via LoRA fine-tuning, then apply an alignment loss targeting the victim model to improve transferability. This two-stage strategy ensures both semantic preservation and strong cross-model attack success (see Table 1). Thus, while the optimization is per-image, the resulting adversarial image generalizes well across models.\n\n\n\n**4) On the advantages and justification of image-specific optimization**\n\n**Image-specific optimization** is a widely adopted strategy in both adversarial attacks and personalized generation. For instance, many diffusion-based attack methods and personalized generation[1] fine-tune models per image or subject to ensure high fidelity. \n\nCompared to dataset-level optimization, image-specific tuning offers key advantages:\n\n- **Flexibility:** A single alignment can generalize across multiple downstream tasks (see Appendix G.2).  \n- **Higher performance ceiling:** Tailoring to each image enables better use of its semantic features.  \n- **Faster convergence:** More stable and efficient without large-scale data.  \n- **Low adaptation cost:** Less dependent on task/model configurations than dataset-level methods.\n\n[1]DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, CVPR, 2023.\n\n---\n\n**Weakness 2 & Question 3:**  \n*Limited discussion on why the proposed method can achieve closer Pareto optimality.*  \n\n\nThank you for your insightful comment. In multi-objective optimization, the Pareto optimal set consists of solutions where no objective can be improved without sacrificing another. Since the true Pareto frontier is intractable in our task, we construct the analysis shown in **Figure 5(b)** to approximate it. In this figure, the x-axis measures **visual consistency**, and the y-axis represents **attack effectiveness**. Ideally, Pareto-optimal solutions lie toward the **top-right corner**, where both objectives are maximized.\n\nOur proposed algorithm, **APA-SG** (red points), first approaches the optimal solution space for visual consistency and then searches locally for solutions with higher attack effectiveness. Compared to other joint optimization methods (blue points), our approach yields results that are **closer to the Pareto-optimal frontier** (i.e., the upper-right region of the plot).\n\nWe will provide a more detailed discussion of this insight in the **final version**, specifically in the analysis of **Section 5.5**.\n\n---\n\n**Weakness 3:**  \n*Some core operations are from references papers. For example, equation (7) is following ACA, and skip gradient also mentioned in ACA, and gradient checkpointing is widely used in network tuning. Although these are correctly cited, it would raise the concern of novelty.*  \n \n\nThank you for your comments. We would like to clarify that we never claimed these two gradient computation methods as our own contributions. These are standard components in attack optimization, much like how backpropagation is standard in model training. **Our intention in describing these methods is to highlight the flexibility of our framework, which can readily incorporate different gradient computation strategies as needed.**\n\nWhile previous works have adopted these optimization techniques, none has systematically examined how different gradient computation strategies affect the attack performance — a gap our work addresses.\n\nMost importantly, our main innovations are as follows:\n\n(1) For the first time, we interpret the unrestricted adversarial example generation problem from the perspective of preference alignment.\n\n(2) We decouple  the conflicting preferences and propose a two-stage alignment framework that is highly flexible and adaptable to different tasks, diffusion models, optimization parameters.\n\n(3) To address the unique needs of attack preference (i.e., strong attack generalization), we introduce dual-path attack optimization and diffusion augmentation, which improve attack preference alignment and prevent reward hacking.\n\nIn summary, the core novelty of our work lies in the **new perspective**, **the general and flexible optimization framework**, and **the newly designed components for attack preference alignment**—not in the specific gradient calculation methods themselves. \n\n---\n\n**Question 1:**  \n*Correction of Eq. 6.*\n\nThank you for your insightful comment. You are absolutely right — since a similar form was already introduced in Equation 2, we applied a simplified notation in Equation 6.  As per your suggestion, we will revise the equation accordingly. We sincerely appreciate your feedback.\n\n---\n\n**Question 2:**  \n*In Equation (7), why do we use the L1 norm*\n\nThank you for your careful observation. Both ACA and our method follow the classical work in adversarial attacks — MI [1], which first introduced momentum into adversarial example generation. As stated in the original paper (last paragraph of Section 3.1), any distance metric can be used for normalization, with L1 norm being the default. Therefore, using the L2 norm is also valid.\n\n[1] Dong et al., *Boosting Adversarial Attacks with Momentum*, CVPR 2018.\n\n---\n\n**Question 5:**  \n*The manuscript had discussed the one-stage and two-stage alignment in page 9 line 320 to 326. It is not clear the performance of the alignment without LoRA. Please consider providing more details.*\n\nThank you for your valuable comment.  Following your suggestion, we have now included the corresponding quantitative results, shown in the table below, ResNet50 as the source model:\n\n| Method| LPIPS↓ | SSIM↑ | CLIP Score↑ |  Avg. ASR\n|---------------------|--------|--------|----------------| ----------------|   \n| APA-SG w/o VCA | 0.55| 0.46| 0.62|  95.43 |\n| APA-SG| 0.25| 0.67| 0.86|75.32|\n\nThese results show that aligning only with attack effectiveness significantly boosts attack performance, but severely degrades image quality (CLIP Score = 0.62), making the outputs practically unusable. We will add these results to Figure 5(b) and discuss them on page 9, lines 320–326.\n\nOnce again, thank you for your time. If these clarifications resolve your concerns, we would be grateful if you could consider raising your score."}]

Author rebuttal:

Meta-review: This paper proposes a diffusion-based method for crafting unrestricted adversarial attacks that aims to balance adversarial performance and consistency. Reviewers found the paper to make a meaningful technical contribution to the field. The discussion was very fruitful and helped identify several clarity issues in the current draft - acceptance is conditioned on the authors incorporating all points raised by the reviewers in their next revision. One question raised was wether the connection to preference alignment is needed or useful. Here the authors should decide weather to make it less pronounce, or to adopt it fully (in which case the related works section should be updated, e.g. see [1]). The authors should pay special attentions to the concerns raised regarding ambiguities, overclaims, and portraying the method's limitations.

[1] Adversaries With Incentives: A Strategic Alternative to Adversarial Robustness, ICLR 2025.
Decision: Accept (poster)