Dissecting Adversarial Robustness of Multimodal LM Agents
Date: 2024-06
Affected: GPT-4V web agents, CLIP-based VLM agents
Attack vector: adversarial-input
Category: research
Severity: High

Wu et al. attack VLM-based web agents via captioner and CLIP perturbations on a single trigger image. With L-inf 16/256 perturbation, 75 percent of a captioner-augmented GPT-4V agent's actions are hijacked; imperceptible perturbations reach 67 percent. ICLR 2025.


Mitigations:

Architectural layers (MAESTRO):