Title: Fit the Distribution: Cross-Image/Prompt Adversarial Attacks on Multimodal Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Large Language Models; Adversarial attack; Evaluation
Abstract: Although Multimodal Large Language Models (MLLMs) have demonstrated remarkable achievements in recent years, they remain vulnerable to adversarial examples that result in harmful responses. Existing attacks typically focus on optimizing adversarial perturbations for a certain multimodal image-prompt pair or fixed training dataset, which often leads to overfitting. Consequently, these perturbations fail to remain malicious once transferred to attack unseen image-prompt pairs, suffering from significant resource costs to cover the diverse multimodal inputs in complicated real-world scenarios. To alleviate this issue, this paper proposes a novel adversarial attack on MLLMs based on distribution approximation theory, which models the potential image-prompt input distribution and adds the same distribution-fitting adversarial perturbation on multimodal input pairs to achieve effective cross-image/prompt transfer attacks. Specifically, we exploit the Laplace approximation to model the Gaussian distribution of the image and prompt inputs for the MLLM, deriving an estimate of the mean and covariance parameters. By sampling from this approximated distribution with Monte Carlo mechanism, we efficiently optimize and fit a single input‑agnostic perturbation over diverse image‑prompt pairs, yielding strong universality and transferability. Extensive experiments are conducted to verify the strong adversarial capabilities of our proposed attack against prevalent MLLMs spanning a spectrum of images/prompts.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a generalizable cross-image/prompt adversarial attack method   from a novel distribution approximation perspective against Multimodal Large Language Models. This attack method first utilizes the Laplace approximation to model the distribution over prompts and images as two Gaussian distributions. It then employs the Monte Carlo mechanism to sample over the image and prompt distributions to optimize the image/prompt agnostic perturbation. Extensive experiments on different MLLMs and datasets validate its superior effectiveness and transferability compared with other attack methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.\tThe paper is well-organized and easy to follow and the findings of this paper is interesting.\n2.\tThe paper provides very detailed mathematical explanation of the procedures of the proposed attack method, thus has good mathematical theoretical support. It offers a novel method for generating effective and transferable adversarial examples against different MLLMs.\n3.\tThe paper provides extensive and detailed experiments results to verify the effectiveness and transferability of the proposed attack method. On different MLLMs and datasets, this attack method can achieve much better cross-sample performance compared with other adversarial attack methods.\nWeakness:\n1.\tThe compared MLLMs are relatively old. The authors should test the effectiveness of the proposed attack method against more latest models such as DeepSeekVL or DeepSeekVL-2. Besides, the number of the tested MLLMs is too small, the authors should increase the number of the tested MLLMs.\n2.\tThe paper mentions that it uses all-MiniLM-L6-v2 model to compute the cosine similarity between embeddings of adversarial output and target text. However, it does not explain the reason for using all-MiniLM-L6-v2. Does the cosine similarity computed by other models is consistent with all-MiniLM-L6-v2?\n3.\tThe datasets used in experiments are limited. More different types of datasets are needed to further make the experiment results more convincing.\n4.\tThe paper only compare Gaussian distribution and fixed distribution to prove the superior performance of Gaussian distribution. More different types of distributions should be considered and discussed.\n\nQUESTIONS:\n1.\tThe compared MLLMs are relatively old. Is the proposed attack method still effective against more latest models such as DeepSeekVL or DeepSeekVL-2? \n2.\tThe paper claims that it uses all-MiniLM-L6-v2 model to compute the cosine similarity between embeddings of adversarial output and target text. Why choose all-MiniLM-L6-v2 instead of other models? Do other different models can get the same results as mentioned in the paper? \n3.\tThe datasets used in experiments are limited. Is the proposed attack method still effective on other different datasets and other vision-language tasks?\n4.\tThe paper uses Gaussian distributions to approximate the prompt and image distributions. Do other different data distributions have similar results?\n\nLIMITATIONS:\nI suggest authors to add discussions on limitations in the paper instead of appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

comments: SUMMARY:
This paper proposes a novel adversarial attack against MLLMs under the cross-image/prompt scenario. Specifically, it models the potential image-prompt input distribution using Laplace approximation and optimizes a universal adversarial perturbation via Monte Carlo sampling from the distribution. Extensive experiments on prevalent MLLMs across multiple datasets demonstrate the strong generalizability and transferability across unseen image-prompt pairs.

STRENGTHS AND WEAKNESSES:
## Strengths 

- **Innovation**: The distribution-driven perspective is innovative, addressing the overfitting issue of existing attacks.
- **Performance**: The experimental results demonstrate that the proposed method outperforms previous attacks.
- **Writing**: The paper is well-structured and easy to follow.

## Weaknesses

- Comparisons with more related adversarial attacks could make it more thorough to validate the superiority. 
- The robustness evaluation only contains two defenses. Analyses on more defenses (e.g. defenses against traditional adversarial attacks) are expected.

**Minor Weakness**:
- The placement of Figure 1 and Tables 2, 3, and 5 is inappropriate, affecting the visual flow of the text.
- Table 4 should be positioned after Table 2 and 3.
- The pipeline in Figure 2 is relatively cluttered. It is recommended to redraw it to clarify the connections between different steps.

QUESTIONS:
Please refer to the Weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

comments: SUMMARY:
This paper proposes a novel cross-image/prompt adversarial attack method against multimodal large language models (MLLMs). The key idea is to model the distribution of image and prompt inputs using Laplace approximation, approximating them as Gaussian distributions. The authors then apply Monte Carlo sampling from these distributions to optimize a universal, input-agnostic adversarial perturbation that generalizes across unseen image-prompt pairs. Extensive experiments on LLaVA, BLIP-2, and MiniGPT-4, across MS-COCO and DALLE-3 datasets, show the proposed method achieves superior cross-image/prompt attack performance compared to existing baselines (e.g., PGD, CroPA, UniAtt).

STRENGTHS AND WEAKNESSES:
## Strengths:
1. Innovative distribution-based attack framework.
2. Strong cross-sample results across multiple datasets and models.
3. Efficiency: generates universal perturbations with reduced computational cost.

## Weaknesses:
1. The entire method and experiments assume full access to model internals, including gradients and architecture. The paper does not explore black-box, gray-box, or query-limited scenarios, which are far more relevant in real-world adversarial contexts where model parameters are typically inaccessible.
2. Experiments only cover MS-COCO and DALLE-3; evaluation on more varied datasets (e.g., Flickr30k) would strengthen generalization claims.

QUESTIONS:
See weaknesses.

LIMITATIONS:
1. The method assumes that the modeled input distributions align well with unseen real-world inputs. In practice, distribution shift may degrade attack transferability.
2. The study focuses on simple input-level defenses; it does not explore more advanced defenses (e.g., adversarial training, multimodal attention masking).
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

comments: SUMMARY:
This paper introduces a new approach to generate cross-image/prompt transferable adversarial examples by approximating the distribution of both image and text input. This method achieves a significant improvement compared to the previous methods.

STRENGTHS AND WEAKNESSES:
## Strength

1. The proposed method achieves a significant improvement compared to the previous methods.
2. Based on the simple intuition of approximating the distribution to improve the generalization/transferability of adversarial attacks.

## Weakness

1. Lack of results on state-of-the-art MLLMs, like Qwen2-VL, InternVL, as well as closed-source models.
2. This method introduces a new computation to fit both the image and text distribution, as well as a sampling computation.

QUESTIONS:
1. I notice the optimization step number is set to 300, while CroPA uses 2000. Will a larger step number lead to better performance?
2. The author can also show the cross-model transferability of the proposed method, and compare the result with methods like [1].
3. Can you provide more details on the efficiency analysis? It generates **all** adversarial examples. But the proposed method generates the cross-image perturbation for all data but CroPA generates adversarial examples for each data sample. Do I understand correctly? And what’s the size of the dataset used?

[1] Gu, Chenhe, et al. "Improving Adversarial Transferability in MLLMs via Dynamic Vision-Language Alignment Attack." arXiv preprint arXiv:2502.19672 (2025).

LIMITATIONS:
1. This approach approximates the distribution of a given prompt/image set, which might be ineffective under a prompt or an image from a different distribution. So it is crucial to sample a good prompt/image set.
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-21