Title: Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment
Venue: NeurIPS 2025
Accepted: yes
Keywords: unrestricted adversarial attack

preference alignment
Abstract: Preference alignment in diffusion models has primarily focused on benign human preferences (e.g., aesthetic). In this paper, we propose a novel perspective: framing unrestricted adversarial example generation as a problem of aligning with adversary preferences. Unlike benign alignment, adversarial alignment involves two inherently conflicting preferences: visual consistency and attack effectiveness, which often lead to unstable optimization and reward hacking (e.g., reducing visual quality to improve attack success). To address this, we propose APA (Adversary Preferences Alignment), a two-stage framework that decouples conflicting preferences and optimizes each with differentiable rewards. In the first stage, APA fine-tunes LoRA to improve visual consistency using rule-based similarity reward. In the second stage, APA updates either the image latent or prompt embedding based on feedback from a substitute classifier, guided by trajectory-level and step-wise rewards. To enhance black-box transferability, we further incorporate a diffusion augmentation strategy. Experiments demonstrate that APA achieves significantly better attack transferability while maintaining high visual consistency, inspiring further research to approach adversarial attacks from an alignment perspective.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces APA (Adversary Preferences Alignment), a novel two-stage framework for generating unrestricted adversarial examples using diffusion models. The key contributions are: \n1. Reformulating adversarial attacks as a preference alignment problem with conflicting objectives (visual consistency vs. attack effectiveness); \n2. A decoupled approach: Stage 1 uses LoRA fine-tuning with rule-based rewards for visual consistency, Stage 2 uses dual-path attack guidance with diffusion augmentation for attack effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- Novel on consider both human preference alignment and adversarial preference alignment.\n- Rich experiments on various attacks and models.\n\n**Weaknesses:**\n- Heavy computation cost on per-image LoRA fine-tuning.\n- Limited discussion on why the proposed method can achieve closer Pareto optimality.\n- Some core operations are from references papers. For example, equation (7) is following ACA, and skip gradient also mentioned in ACA, and gradient checkpointing is widely used in network tuning. Although these are correctly cited, it would raise the concern of novelty.\n\nQUESTIONS:\n- How do we get equation 6 (page 4 line 168), what is the definition of \n$E_{t, \\epsilon}$? I think the loss function should be \n\n$$\\mathbb{E}_{t \\sim [1, T], \\epsilon \\sim \\mathcal{N}(0, \\mathbf{I})} \\left( - \\| \\epsilon - \\epsilon_{\\theta + \\Delta \\theta}(z_t, t, c)\\|^2 \\right).$$\n\nIt is a little ambiguous to $(E_{t,\\epsilon}) - (\\| \\epsilon - \\epsilon_{\\theta + \\Delta \\theta}(z_t, t, c)\\|^2)$.\n- In equation (7), why do we use the L1 norm on $g_{tr}$? As the same norm is used in ACA, I am wandering what is the purpose of using L1 norm instead of L2 norm.\n- The manuscript claims that the proposed method can achieve closer Pareto optimality, but the detail discussion is missing. Please provide more details.\n- Since it is not clear whether one image is applied or a batch of images is applied to the purposed method. Various thanks for providing the code. After reading the code provided in the attachment, it seems that you fine-tune one LoRA for each Image, and attack one image using the pretrained LoRA on that Image. Does my understanding correct? I think this is horrible. \n  - It is super computational expensive on getting one adversarial example. \n  - It is not clear whether the fine-tuned LoRA on one image can work on other images.\n  - I am not sure weather the good performance is due to something like over-fitting, and the model is highly optimized for one specific image.\n- The manuscript had discussed the one-stage and two-stage alignment in page 9 line 320 to 326. It is not clear the performance of the alignment without LoRA. Please consider providing more details.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes APA, a novel framework reframing unrestricted adversarial example generation in diffusion models as alignment with adversary preferences. Addressing the conflicting objectives of visual consistency and attack effectiveness – which often cause unstable optimization and reward hacking – APA employs a two-stage approach: first fine-tuning LoRA for visual quality using rule-based rewards, then optimizing image latents or prompt embeddings using substitute classifier feedback with trajectory-level and step-wise rewards. Enhanced by diffusion augmentation for transferability, APA achieves significantly better black-box attack success while maintaining high visual consistency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This paper presents APA, an effective framework addressing the fundamental trade-off between visual consistency and attack effectiveness in adversarial example generation.\n2. The experiments are thorough and well-designed, providing convincing evidence for the algorithm’s effectiveness and superiority.\n\nWeaknesses：\n\n1. The paper’s organization is challenging to follow, with an abundance of complex design choices that obscure the core methodology and hinder readability.\n2. The framework’s high time overhead significantly limits its real-world applicability.\n\nQUESTIONS:\n1. While the approach presents an interesting exploration of unrestricted attacks, the connection to adversary preference alignment appears tenuous. Consequently, the paper's organization makes it difficult to grasp the necessity of the numerous proposed techniques. The authors are adviced to **significantly improve the presentation clarity**. For reference, my understanding is that the attack comprises two core modules:\n\n  - Optimizing $z_T$ for attack effectiveness. Since diffusion model is instable near $z_T$, LoRA-based training is used to enhance visual consistency.\n\n  - Solving an inverse problem (maximizing adv. loss) via denoising path guidance, similar to DPS[1]. \n\n2. Although APA-GC demonstrates the best performance, the authors provide no analysis of its computational efficiency. It is crucial to compare the performance of different attack algorithms under equivalent computational budgets. For example, the authors can include results using standardized metrics, such as ASR plotted against computation time (e.g., wall-clock time or FLOPs), with time/compute on the x-axis and ASR on the y-axis.\n\n[1] Diffusion Posterior Sampling for General Noisy Inverse Problems. (https://arxiv.org/abs/2209.14687)\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Adversary Preferences Alignment (APA), an innovative two-stage framework designed to enhance diffusion-based unrestricted adversarial attacks. Unlike typical benign preference alignments that optimize for human-preferred attributes, APA addresses adversarial preferences by balancing two conflicting goals: visual consistency and attack effectiveness. The authors propose decoupling these conflicting objectives, initially optimizing visual consistency through LoRA fine-tuning with a rule-based visual similarity reward, followed by optimizing attack effectiveness using dual-path attack guidance and diffusion augmentation.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n- Concept of aligning diffusion models to adversary preferences rather than benign human preferences\n- I appreciate that the author conduct comprehensive evaluation across multiple CNNs and Vision Transformers\n- APA significantly outperforms prior methods in maintaining visual consistency and naturalness, supported by rigorous quantitative  (LPIPS, SSIM, CLIP Score) and qualitative assessments.\n- The authors provided the discussion regarding the ablation of visual consistency alignment  and gradient backpropagation.\n\n## Weakness:\n- Although the 2-stage method is novel, multi-component optimization process may introduce complexity in real-world implementation. Possibly further simplification or integration techniques could be beneficial.\n- The paper frequently uses $z_T$, $z_0$, $\\bar{z}_0$, $\\hat{z}_0$, $z^0_t$, $z^{in}_t$, etc., sometimes with and sometimes without bars, hats, or superscripts. It's easy for the readers to lose track of what each variable specifically represents—particularly the difference between “trajectory-level” and “step-level” variables. Maybe you could provide a summary table of symbols.\n- And next on equations. The reader is left piecing together the order and purpose of updates without a clear, step-by-step guide. For example: \n   - In Eq. 6, the switch from image similarity to latent similarity is abrupt; justification for why latent-space similarity is a good surrogate for visual consistency is not well articulated.\n- Regarding the explanation of methods and strategies, I believe you need to provide more clarification. For example, details on LoRa Fine-Tuning, to be more precise, how LoRa modules are inserted.\n\nQUESTIONS:\n1. In the paper, you conducted evaluation on existing defenses which originally focusing on Lp attacks, but are you also considering defenses method on unrestricted adversarial attacks?\n\n2. In Eq.7, it introduces momentum-based gradients, but $\\Pi z^0_T + \\epsilon_a$, what this refer to? Can you provide more details?\n\n3. Line 164 ~166, you mentioned regarding the shift of similarity metric from the pixel/image space (Eq. 5) to the latent space (Eq. 6) for computational efficiency. However, the paper does not provide sufficient empirical comparison to support the claim that latent similarity adequately preserves visual consistency, which is critical to the proposed method. Can you possibly explain more on this or provide more context?\n\n4.  The “skip gradient” and “gradient checkpointing” strategies are described briefly; their impact on optimization stability and convergence, I think is not well justified or analyzed. Are there convergence guarantees ?\n\nLIMITATIONS:\nThe authors have provided the limitations in Appendix C.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a two-stage framework APA (Adversary Preferences Alignment) to craft unrestricted adversarial examples using diffusion models. For an original image, APA first employs LoRA to fine-tune the UNet in the LDM to improve the reconstruction quality of DDIM inversion to ensure the visual consistency of the adversarial example after perturbing the latent embeddings. Then, during the denoising process of LDM, APA performs adversarial optimization on the latent embeddings through a dual-path strategy at both the trajectory-level and step-level. Additionally, APA introduces Diffusion Augmentation to further enhance the transferability.\n\nSTRENGTHS AND WEAKNESSES:\nThe methodology presented in this paper demonstrates technical novelty. The APA approach is built upon the existing ACA method but incorporates several significant improvements. For example, it cleverly fine-tunes the UNet model parameters to enhance the visual consistency of Unrestricted Adversarial Examples. Additionally, it introduces both trajectory-level and step-level optimization of latent embeddings and diffusion augmentation to improve attack effectiveness.\n\nHowever, the authors seem to overclaim the connection between APA and the concept of Performance Alignment in diffusion models. APA still follows the conventional paradigm of adversarial attacks, where each adversarial example must be generated through a complete gradient-based optimization process (for both UNet and latent embeddings in APA) with a given original image, rather than directly aligning the model once for solving this problem. Moreover, finding better solutions that simultaneously achieve high image quality and strong attack performance has long been a key objective in adversarial attacks. Although certain components of APA are defined as \"rewards\", they are essentially equivalent to the loss functions commonly used in existing adversarial attacks for computing gradients (e.g., cross-entropy loss from surrogate classification models).\n\nQUESTIONS:\n1. Except for the results in Figure  3, are there any quantitative results or theoretical analyses to support the effectiveness of Visual Consistency Alignment?\n2. Considering the image quality of APA-SG in Figure 4 appears better than that of ACA and DiffPGD-MI, yet their quantitative metrics are comparable, and some results of APA-SG in Appendix H exhibit noticeable artifacts, are the visualization images specially selected? More visualization results of randomly sampled images should be included. Moreover, it is observed that APA-GC-P, which achieves better image quality according to quantitative metrics, suffers a significant drop in average ASR, but its detailed results are not included in Tables 1 and 2.\n3. A comparison with the state-of-the-art LDM-based unrestricted adversarial attack DiffAttack [1] is missing. ([1] Chen J, Chen H, Chen K, et al. Diffusion models for imperceptible and transferable adversarial attack[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.)\n4. It should be clarified whether the APA variant compared with AdvDiff in Appendix E.3 remains under an untargeted attack setting, to avoid potential unfair comparisons, or how APA can be extended to targeted attack scenarios.\n5. Why does the optimization process of APA appear more complex and involve more computation steps than ACA, yet APA runs faster according to Appendix B? The authors should explain this result and provide a corresponding analysis for the computational complexity of APA.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}