Robust CLIP: Unsupervised Adversarial Fine-Tuning Defenses Reveal CLIP Vulnerabilities
Date: 2024-02
Affected: CLIP, LLaVA, OpenFlamingo and downstream VLMs
Attack vector: adversarial-input
Category: research
Severity: High

Schlarmann et al. show that off-the-shelf CLIP-based VLMs (LLaVA, OpenFlamingo) are highly fragile to imperceptible adversarial perturbations on input images, motivating their unsupervised adversarial fine-tuning defense.


Mitigations:

Architectural layers (MAESTRO):