Adversarial Attacks against Closed-Source MLLMs via Feature Optimal
  Alignment
Authors: name: Xiaojun Jia; name: Sensen Gao; name: Simeng Qin; name: Tianyu Pang; name: Chao Du; name: Yihao Huang; name: Xinfeng Li; name: Yiming Li; name: Bo Li; name: Yang Liu
Published: 2025-05-27 17:56:57

Abstract: Multimodal large language models (MLLMs) remain vulnerable to transferable
adversarial examples. While existing methods typically achieve targeted attacks
by aligning global features-such as CLIP's [CLS] token-between adversarial and
target samples, they often overlook the rich local information encoded in patch
tokens. This leads to suboptimal alignment and limited transferability,
particularly for closed-source models. To address this limitation, we propose a
targeted transferable adversarial attack method based on feature optimal
alignment, called FOA-Attack, to improve adversarial transfer capability.
Specifically, at the global level, we introduce a global feature loss based on
cosine similarity to align the coarse-grained features of adversarial samples
with those of target samples. At the local level, given the rich local
representations within Transformers, we leverage clustering techniques to
extract compact local patterns to alleviate redundant local features. We then
formulate local feature alignment between adversarial and target samples as an
optimal transport (OT) problem and propose a local clustering optimal transport
loss to refine fine-grained feature alignment. Additionally, we propose a
dynamic ensemble model weighting strategy to adaptively balance the influence
of multiple models during adversarial example generation, thereby further
improving transferability. Extensive experiments across various models
demonstrate the superiority of the proposed method, outperforming
state-of-the-art methods, especially in transferring to closed-source MLLMs.
The code is released at https://github.com/jiaxiaojunQAQ/FOA-Attack.