Title: Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Models
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Adversarial Robustness, Vision-Language Models, Consistency

Abstract: Pre-trained vision-language models (VLMs) such as CLIP have demonstrated strong zero-shot capabilities across diverse domains, yet remain highly vulnerable to adversarial perturbations that disrupt image-text alignment and compromise reliability. 
Existing defenses typically rely on adversarial fine-tuning with labeled data, limiting their applicability in zero-shot settings. 
In this work, we identify two key weaknesses of current CLIP adversarial attacks—lack of semantic guidance and vulnerability to view variations—collectively termed semantic and viewpoint fragility. To address these challenges, we propose \textsc{Self-Calibrated Consistency (SCC)}, an effective test-time defense. SCC consists of two complementary modules: 
\textit{Semantic consistency}, which leverages soft pseudo-labels from counterattack warm-up and multi-view predictions to regularize cross-modal alignment and separate the target embedding from confusable negatives; and
\textit{Spatial consistency}, aligning perturbed visual predictions via augmented views to stabilize inference under adversarial perturbations.
Together, these modules form a plug-and-play inference strategy. 
Extensive experiments on 22 benchmarks under diverse attack settings show that SCC consistently improves the zero-shot robustness of CLIP while maintaining accuracy, and can be seamlessly integrated with other VLMs for further gains. 
These findings highlight the great potential of establishing an adversarially robust paradigm from CLIP, with implications extending to broader vision-language domains such as BioMedCLIP. Code will be released.

TL;DR: We propose SCC,  a test-time defense that enforces semantic and spatial consistency, boosting zero-shot robustness of VLMs without retraining