Backdoor Cleaning without External Guidance in MLLM Fine-tuning
Authors: name: Xuankun Rong; name: Wenke Huang; name: Jian Liang; name: Jinhe Bi; name: Xun Xiao; name: Yiming Li; name: Bo Du; name: Mang Ye
Published: 2025-05-22 17:11:58

Abstract: Multimodal Large Language Models (MLLMs) are increasingly deployed in
fine-tuning-as-a-service (FTaaS) settings, where user-submitted datasets adapt
general-purpose models to downstream tasks. This flexibility, however,
introduces serious security risks, as malicious fine-tuning can implant
backdoors into MLLMs with minimal effort. In this paper, we observe that
backdoor triggers systematically disrupt cross-modal processing by causing
abnormal attention concentration on non-semantic regions--a phenomenon we term
attention collapse. Based on this insight, we propose Believe Your Eyes (BYE),
a data filtering framework that leverages attention entropy patterns as
self-supervised signals to identify and filter backdoor samples. BYE operates
via a three-stage pipeline: (1) extracting attention maps using the fine-tuned
model, (2) computing entropy scores and profiling sensitive layers via bimodal
separation, and (3) performing unsupervised clustering to remove suspicious
samples. Unlike prior defenses, BYE equires no clean supervision, auxiliary
labels, or model modifications. Extensive experiments across various datasets,
models, and diverse trigger types validate BYE's effectiveness: it achieves
near-zero attack success rates while maintaining clean-task performance,
offering a robust and generalizable solution against backdoor threats in MLLMs.