Title: Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Large Language Models

Transferable Adversarial Attacks

Feature Optimal Alignment
Abstract: Multimodal large language models (MLLMs) remain vulnerable to transferable adversarial examples. While existing methods typically achieve targeted attacks by aligning global features—such as CLIP’s [CLS] token—between adversarial and target samples, they often overlook the rich local information encoded in patch tokens. This leads to suboptimal alignment and limited transferability, particularly for closed-source models. To address this limitation, we propose a targeted transferable adversarial attack method based on feature optimal alignment, called FOA-Attack, to improve adversarial transfer capability. Specifically, at the global level, we introduce a global feature loss based on cosine similarity to align the coarse-grained features of adversarial samples with those of target samples. At the local level, given the rich local representations within Transformers, we leverage clustering techniques to extract compact local patterns to alleviate redundant local features. We then formulate local feature alignment between adversarial and target samples as an optimal transport (OT) problem and propose a local clustering optimal transport loss to refine fine-grained feature alignment. Additionally, we propose a dynamic ensemble model weighting strategy to adaptively balance the influence of multiple models during adversarial example generation, thereby further improving transferability.  Extensive experiments across various models demonstrate the superiority of the proposed method, outperforming state-of-the-art methods, especially in transferring to closed-source MLLMs.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to produce more transferrable adversarial attacks against closed-source MLLMS. They observe that existing methods produce adversarial examples by minimizing the contrastive loss between the global features (produced by the CLS tokens)  of the example and target samples. They argue that adversarial samples generated this way have poor transferability due to the weak alignment between the local representations (produced by the patch tokens) and the global representations (produced by the CLS token). To solve this problem, the paper proposes FOA-Attack — a method to jointly align global and local features to achieve better transferability. The method uses a clustering technique to distill compact local patterns and frames the alignment of local features as an optimal transport problem. Furthermore, they use an ensemble of models to improve transferability. Experiments show that the proposed method significantly outperforms prior method for generating adversarial examples.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper is well written and clearly describes the proposed idea\n- The key insight of aligning local features is well motivated\n- The results show strong improvement in transferability across multiple target models.  \n\nWeaknesses\n\n- The proposed FOA-Attack is significantly more computationally intensive than the primary baseline, M-Attack, taking 217 minutes versus M-Attack's 90 minutes. The paper does not provide a compute-normalized comparison to show how M-Attack would perform if allotted a similar computational budget (e.g., by increasing its attack iterations). Such a comparison would help to more clearly isolate the performance gains attributable to the proposed methodology versus those from simply using more computation.\n\n- The paper argues that clustering is necessary to handle redundant local features, but this is not experimentally validated. The ablation study removes the entire local alignment component but does not include a condition where local alignment is performed on all patch tokens without clustering. This makes it difficult to assess the specific contribution and necessity of the K-Means step.\n\nQUESTIONS:\n1. What happens to the attack performance if you perform alignment of local features without using K Means?\n2. Any idea why the attack performance on Claude is worse compared to other models?\n3. I don’t fully understand Table 6. Do the four rows for FOA have different number of cluster centers?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-19"}

comments: SUMMARY:
The paper proposes a novel approach for generating transferable adversarial examples by maximizing both global and local feature alignment between the adversarial and target examples. For global alignment, the authors employ a feature loss based on the cosine similarity between the [CLS] tokens of the adversarial and target examples. For local alignment, they introduce an optimal transport loss that aligns representative image patch embeddings. These representative patches correspond to cluster centers obtained via k-means clustering applied to all patch embeddings. Additionally, the authors propose an aggregated loss that combines the aforementioned losses across multiple encoders. The weights assigned to each encoder are designed to balance contributions from both fast-learning and slow-learning models.

STRENGTHS AND WEAKNESSES:
** Strengths **
* the paper is well written and easy to read.
* the proposed approach is sound and improves upon the baselines


**Weaknesses **
* How expensive is the Sinkhorn algorithm in practice? how many iterations is it run for in average until convergence?
* The approach is incremental (limited novelty)

QUESTIONS:
* How expensive is the Sinkhorn algorithm in practice? how many iterations is it run for in average until convergence? 

* How can such an attack be mitigated?

LIMITATIONS:
The authors include a limitation section.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-24

comments: SUMMARY:
FOA-Attack is a novel targeted adversarial attack framework for MLLMs that enhances transferability by aligning both global and local features in the image encoder. It utilizes a global cosine similarity loss and a local clustering optimal transport loss for coarse-grained and fine-grained feature alignment, respectively. The experiments demonstrate higher ASR on open-sourced and closed-sourced models like GPT, Gemini, and Claude compared to existing targeted attacks.

STRENGTHS AND WEAKNESSES:
## Strengths
- The methodology is sound, using an optimal transport based objective to align local features is elegant and novel
- The empirical results are very compelling. I like that the authors focus on evaluating ASR on closed sourced models.
 

## Weaknesses
- The paper does not consider using targets that may elicit unsafe or harmful captions from the target model. This setting is underexplored, but poses a more real-world risk.
- Both the source and target images are natural images, but it is unclear how similar do the source and target images need to be for attacks to be successful. 
- The set of image encoders in the ensemble are all CLIP models. Increasing the diversity of the ensemble may further improve performance. For example, what happens if we mix CLIP and SigLIP models in the ensemble?
- [Minor] Below Line 180, the optimization problem should have $min_{\pi}$ instead of just $min$ for clarity.

QUESTIONS:
See weaknesses

LIMITATIONS:
addressed
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

comments: SUMMARY:
This paper proposes a new adversarial attack method against closed source MLLMs, which not only use global features but also adopt local features to align the adversarial sample with target sample.

STRENGTHS AND WEAKNESSES:
Strengths:
1.	This paper proposes a targeted transferable adversarial attack method based on both global and local feature-alignment.

2.	Clustering techniques and a local feature alignment method, i.e., a local clustering OT are employed, due to the redundant local features of the image.

3.	A dynamic ensemble model weighting strategy is proposed for improving the adversarial transferability, which is based on the models’ convergence rates.

Weaknesses:

1.	The method use clustering technique to clustering the local features, how to select the cluster number. As the number of clusters also affect the attack performance.

2.	The generation of the final adversarial sample is not given, how do you obtain the adversarial sample based on the global and local alignment?

3.	How many models are used in Eq.(11). And are they only CLIP models? Why not use more different models to improve the transferability.

QUESTIONS:
1.	The method use clustering technique to clustering the local features, how to select the cluster number. As the number of clusters also affect the attack performance.

2.	The generation of the final adversarial sample is not given, how do you obtain the adversarial sample based on the global and local alignment?

3.	How many models are used in Eq.(11). And are they only CLIP models? Why not use more different models to improve the transferability.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03