Title: Adversaries Fight Back: Revisiting Test-Time Adversarial Defenders of Vision-Language Model under Adaptive Attacks
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Adversarial Robustness, Vision-Language Model, Computer Vision

Abstract: Recent works have introduced test-time defenses to protect zero-shot classification of vision-language models (VLM) against adversarial attacks. While these methods claim to achieve test-time efficiency and performance comparable to train-time defenses on CLIP, we evaluate 6 state-of-the-art VLM test-time adversarial defenses and find that their robustness significantly overestimated when subjected to defense-specific attacks (adaptive attacks). By simply applying basic adaptive attacks including full-defense gradient and approximated-gradient attacks, we find that the robustness for half of the evaluated methods can be largely reduced. We then further develop a flexible Distributional-Difference-Targeting Adaptive Attack (CLIP-DDT) to enhance the strength of adversarial points for CLIP’s test-time defenses, even when full-defense gradient adaptive attack is hard to compute or ineffective. Specifically, we identify a ubiquitous but vulnerability-inducing key step in these defenses: the zero-shot measurement of a detectable distributional difference between adversarial and clean data. Our method targets this Achilles’ Heel by operating on this distributional difference during attacking the underlying static model when basic adaptive attacks fail or it can be combined with basic adaptive attacks for boosted effect otherwise. Experiments across 13 datasets demonstrate that our method generates strong adversarial samples that further degrade the robustness of various test-time defenses, revealing a false sense of security in CLIP’s zero-shot robustness.

TL;DR: This paper shows the false sense of security in state-of-the-art test-time defenses in CLIP's zero-shot classification and proposes a method to flexibly construct strong adaptive attacks for CLIP's test-time defenses.