Visual Adversarial Examples Jailbreak Aligned LLMs
Date: 2023-06
Affected: GPT-4V, MiniGPT-4, InstructBLIP, LLaVA
Attack vector: adversarial-input
Category: research
Severity: Critical

Qi et al. (Princeton) show a single visual adversarial example can universally jailbreak vision-integrated LLMs, eliciting harmful content beyond the few-shot derogatory corpus used during optimization. OpenAI confirmed the findings for GPT-4V. AAAI 2024 Oral.


Mitigations:

Architectural layers (MAESTRO):