Title: Zero-Sacrifice Lifelong Adversarial Defense for Pre-Trained Encoders
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: adversarial robustness, self-supervised learning, pretrained models, security vulnerability

Abstract: The widespread use of publicly available pre-trained encoders from self-supervised learning (SSL) has exposed a critical vulnerability: their susceptibility to downstream-agnostic adversarial examples (DAEs), which are crafted without knowledge of the downstream tasks but capable of misleading downstream models. While several defense methods have been explored recently, they rely primarily on task-specific adversarial fine-tuning, which inevitably limits generalibility and causes catastrophic forgetting and deteriorates benign performance.
Differ with previous works, we propose a more rigorous defense goal that requires only a single tuning for diverse downstream tasks to defend against DAEs and preserve benign performance.
To achieve this defense goal, we introduce **Ze**ro-Sacrifice **L**ifelong **A**dversarial **D**efense (**ZeLAD**), which is inspired by the inherent sensitivity of neural networks to data characteristics. Specifically, ZeLAD is a dual-branch structure, which consists of a Multi-Pattern Adversarial Enhancement Branch (MPAE-Branch) uses two adversarially fine-tuned encoders to strengthen adversarial resistance. The Benign Memory Preservation Branch (BMP-Branch) is trained on local data to ensure adversarial robustness does not compromise benign performance. Surprisingly, we find that ZeLAD can directly detect DAEs by evaluating branch confidence, without introducing any adversarial sample identification task during training. Notably, by enriching feature diversity, our method enables a single adversarial fine-tuning to defend against DAEs across downstream tasks, thereby achieving lifelong robustness. Extensive experiments on 11 SSL methods and 6 datasets validate its effectiveness. In certain cases, it achieves a 29.20\% improvement in benign performance and a 73.86\% gain in adversarial robustness, highlighting its zero-sacrifice property.