Title: VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Models
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Vision-language models, Jailbreaking, Red-teaming

Abstract: Vision-Language Models (VLMs) extend large language models with visual reasoning capabilities but remain vulnerable to jailbreak attacks. Existing multimodal red-teaming methods largely rely on brittle templates, operate in single-attack settings, and expose only narrow modes of vulnerability. To address these limitations, we introduce VERA-V, a variational inference framework that recasts multimodal jailbreak discovery as learning a joint posterior distribution over paired text-image prompts. This probabilistic view enables the generation of stealthy, coupled adversarial inputs that bypass model guardrails. We train a lightweight attacker to approximate the posterior, allowing efficient sampling of diverse jailbreaks and providing distributional insights into vulnerabilities. VERA-V further integrates three complementary strategies: (i) typography-based text prompts that embed harmful cues, (ii) diffusion-based image synthesis that introduces adversarial signals, and (iii) structured distractors to fragment VLM attention. Experiments on HarmBench and HADES benchmarks show that VERA-V consistently outperforms state-of-the-art baselines on both open-source and frontier VLMs, improving up to 53.75\% ASR over the best baseline on GPT-4o.