Zero-Sacrifice Persistent-Robustness Adversarial Defense for Pre-Trained Encoders
Authors: Zhuxin Lei
Ziyuan Yang
Yi Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: adversarial robustness
self-supervised learning
pretrained models
security vulnerability
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: The widespread use of publicly available pre-trained encoders from self-supervised learning (SSL) has exposed a critical vulnerability: their susceptibility to downstream-agnostic adversarial examples (DAEs), which are crafted without knowledge of the downstream tasks but capable of misleading downstream models. While several defense methods have been explored recently, they rely primarily on task-specific adversarial fine-tuning, which inevitably limits generalizability and causes catastrophic forgetting and deteriorates benign performance. Different with previous works, we propose a more rigorous defense goal that requires only a single tuning for diverse downstream tasks to defend against DAEs and preserve benign performance. To achieve this defense goal, we introduce **Ze**ro-Sacrifice **P**ersistent-Robustness **A**dversarial **D**efense (**ZePAD**), which is inspired by the inherent sensitivity of neural networks to data characteristics. Specifically, ZePAD is a dual-branch structure, which consists of a Multi-Pattern Adversarial Enhancement Branch (MPAE-Branch) that uses two adversarially fine-tuned encoders to strengthen adversarial resistance. The Benign Memory Preservation Branch (BMP-Branch) is trained on local data to ensure adversarial robustness does not compromise benign performance. Surprisingly, we find that ZePAD can directly detect DAEs by evaluating branch confidence, without introducing any adversarial exsample identification task during training. Notably, by enriching feature diversity, our method enables a single adversarial fine-tuning to defend against DAEs across downstream tasks, thereby achieving persistent robustness. Extensive experiments on 11 SSL methods and 6 datasets validate its effectiveness. In certain cases, it achieves a 29.20\% improvement in benign performance and a 73.86\% gain in adversarial robustness, highlighting its zero-sacrifice property.

Published as a conference paper at ICLR 2026
ZERO-SACRIFLE PERSISTENT-ROBUSTNESS ADVER- SARIAL DEFENSE FOR TRAINED ENCODERS
Zhuxin Lei Ziyuan Yang Yi Zhang
a School of Cyber Science and Engineering, Sichuan University
Key Laboratory of Data Protection and Intelligent Management, Ministry of Education, Sichuan University Tianfu Jiangxi Laboratory
202214153006 ssstu sou edu. cn, oziyuanyangggmai il. com
rchang@scu.edu..
ABSTRACT
The widespread use of publicly available pre-trained encoders from self- supervised learning (SSL) has exposed critical vulnerability: their susceptibility to downstream- agnosstic adversarial examples (DAEs), which are crafted without knowledge of the downstream tasks but capable of misleading downstream mod- els. While several defense methods have been explored recently, they rely primar- ily on task-specific adversarial fine-tuning, which inevitably limits generalizability and causes catastrophic forgetting and deteriorates benign performance. Differ- ent with previous works, we propose a more rigorous defense goal that requires only single tuning for diverse downstream tasks to defend against DAEs and preserve benign performance. To achieve this defense goal, we introduce Zero- Sacrifice Persistent- Persistent-Robustnness Robustness Adversarial Defense (ZePAD), which is inspired by the inherent sensitivity of neural networks to data characteristics. Specifically, ZePAD is dual-branch structure, which consists of Multi-Pattern Adversar- ial Enhancement Branch (MPAE-Branch) that uses two adversarially fine-tuned encoders strengthen adversarial resistance. The Benign Memory Preservation Branch (BMP-Branch) Branch) is trained on local data to ensure adversarial robustness does not compromise benign performance. Surprisingly, we find that ZePAD can directly detect DAEs by evaluating branch confidence, without introducing any adversarial exsample identification task in training. Notably, by enriching feature diversity, our method enables a single adversarial fine-tuning to defend against DAEs across downstream tasks, thereby achieving persistent robustness. Exten- sive experiments on 11 SSL methods and datasets validate its effectiveness. In certain cases, it achieves a 29.20% improvement benign performance and a 73.86% gain in adversarial robustness, highlighting its zero-sacrifice property. To facilitate reproducibility, the code is publicly available
INTRODUCTION
Training robust models is resource-intensive. Consequently, using public pre-trained models from self-supervised learning (SSL), such as CLIP (Radford et al. 2021 and GPT (Brown) 2020), is common paradigm for downstream tasks. These encoders learn powerful representations, fostering
advancements in fields like classification (Huang et al. 2023 and segmentation (Liu et al. 2023
However, these encoders are vulnerable ples (DAEs) (Liu et al.] (002)). which are crafted without knowledge of the downstream tasks but capable of misleading downstream models. Even when the model has been task-specific tuned on local data, the attack remains effective. For example, PAP (Ban & Dong. 2022) and AdvEncoder (Zhou al. 2023) can generate effective DAEs without knowledge of the pre-training or downstream data.
Downstream-Agnostie Adversarial Exam-
"Ziyuan Yang and Yi Zhang are the corresponding authors. http://gitub..cmm/ www.a/////aaaiitt///////a

---

Published as a conference paper at ICLR 2026
To address this security concern, researchers have proposed several secure fine-tuning methods against DAEs, but these methods lead to a sigmificant drop in benign performance, making them impractical. For instance, Gen-AF (Zhou et al. 2024) improves robust accuracy from 35.58% to 62.28% but reduces benign accuracy from 81.99% to 68.69%. Meanwhile, these methods require re-tuning for each task. This leads to ask:
"Is it possible to achieve a zero-sacrifice persistent-robustues defense that requires only a single tuning for various downstream tasks, without compromising the performance of benign sam- ples?"
We find that neural networks inherently exhibit higher confidence in inputs that resemble the training data, a behavior attributed to the memorization of the data's characteristics Zhang et al. (2016). Inspired by this tendency, we shift our focus from traditional adversarial fine-tuning to evaluating the confidence levels of different encoders.
This insight motivates us to rethink the defense process, shifting the focus from finding a tradeoff between adversarial robustness and performance decline to achieving a zero vero-sacrifice approach. We propose Zero-Sacrifice Lifelong Adversarial Defense (ZePAD), which uses two branches to enhance robustness while maintaining even improving benign performance: i) Multi-Pattern Adversarial Enhancement Branch (MPAF-Branche) To counter DAFs that exploit specific patterns, this branch adversarially fine-tunes encoders trained with diverse SSL methods. This diversity makes it difficult for attackers to find common vulnerabilities. üi) Benign Memory Preservation Branch (BMP- Branch): To mitigate the performance drop clean data from adversarial fine-tuning, this branch is trained only on benign samples, preserving the model's 's sensitivity to them.
To effectively integrate the different branches and achieve zero-sate sacrisace,, Federal Decision Mechanism (RFDM). Each branch's confidence adapts to the alignment between the input and its training dataset distribution, and RFDM makes a decision based this evalua- tion. Specifically, for benign samples, the BMP-Branch assigns higher confidence than the other branches, as the latter are adversarially fine-tuned and more biased towards adversarial exsamples. Since adversarial exsamples were not encountered during BMP-Branch training, they fall outside its distribution, resulting in lower confidence. Similarly, the MPAE-Bras -Baachh, trained with adversarial examples, exhibits higher confidence when handling such samples.
propose Robust
Existing defense methods typically require task-specific adversarial fine-tuning. In this paper, we show that introducing diversification through multiple heterogeneous encoders enhances robustness beyond single-encoder fine-tuning. a result, ZePAD requires only a single finettuning process to provide strong, versatile representations across all downstream tasks, making it a persistent- robustness defense method. Surprisingly, the aforementioned inherent characteristics of neural net- works allow ZePAD to detect adversarial exsamples based solely confidence scores, without requiring specific training for this purpose. Our main contributions are:
We propose sero-sacrifice, persistent-robustneess adversarial defense method that not only maintains but also improves benign performance, while enhancing adversarial robustness. Besides, our method only requires single-time adversarial tuning to achieve robust per- formance across various downstream tasks.
Although adversarial exsample detection is not the focus of our work, our method di- rectly detect adversarial exsamples, without w requiring any specific training for this purpose.
Extensive experiments across 11 SSL methods and 6 datasets validate ZePAD's effective- ness, showing improvements as high as 29.20% in benign performance and 73.05% in robust performance in some scenarios.
RELATED WORKS
Self-Supervised Learning. SSL pre-trains a generic encoder vast unlabeled data for various downstream tasks. This paradigm overcomes labeled data limitations, allowing users to adapt pow- erful pre-trained encoders through fine-tuning without training from scratch. Common adaptation methods include linear evaluation (Aghajanyan et al. 2020; Kumar et al., 2022), which adjusts only the last layer, and full fine-tuning (Hendryyks et al. 2019 2019 Miller al. 2021). which tunes all layers.
2

---

Published as a conference paper at ICLR 2026
Adversarial Attacks. Deep neural networks (DNNs) are vulnerable to adversarial examples (Good- fellow et al. 2014, Hu et al., 2022), including universal adversarial perturbations (UAPs) Co et al. (2019) where single perturbation affects multiple samples. Attacks are categorized white- box (Naseer et al., 2020), where the attacker has full model access, or black-hox (Lin et al.,
2024)
with access only to inputs and outputs. Recent research (Ban & Dong 2022] /hhou et al.
2023
Zhang et al.) 2022) explores attacks on pre-trained encoders, which are downstream-agnoste and thus pose a significant security risk the pre-trained paradigm.
Adversarial Defenses. Adversarial defense methods include input filtering (Guo et al. 2017) and en- hancing model robustness, often through adversarial training (Zhu & Gupta 201 Zhou al. 2024] Li et al. 2024 Liu et (021). However, it typically harms generalization, reducing performance on clean samples. While recent work (Lamb al. 2019 Zhou al. 2024) aims to balance robust- ness and generalization, no existing method enhances robustness without compromising -or while even improving- generaliiizninnnnn ability. The detailed related works are discussed in Appendix [
3 PROBLEM FORMULATION AND THREAT MODELING
3.1 PROBLEM STATEMENT
Let &n be pre-trained encoder, parameterized by 0. We define D, as the dataset used to pre-train En. where the benign input is represented (x, y) E D,, with x as the input and y as the corresponding label. The output feature of &g is denoted as v = Ex(r), where E V and V represents the feature space. The downstream model is denoted Fes, where è represents its parameters. The input Fs is the output of Eg, and the process can be formulated as (v)...
In adversarial attacks, an attacker aims to exploit vulnerabilities in &g by adding small perturbations 5 to the input The perturbation 8 is constrained by upper bound based on the no-rom,, which ensures the noise remains imperceptible. The adversarial attack can be described as:
Eo(x) + Eo(x + 6), subject to ||01|p S.
The ultimate goal of the attacker is to deceive the downstream model Fø- Specifically, for clas- sification task, the attacker aims to manipulate F& to output different predictions for benign and adversarial examples. This can be formalized as follows:
(E((())) * Fo + 6)), subject to ||01|p S c.
(2)
As defenders, our goal is to create defense mechanism that prevents the adversarial attack from succeeding while preserving improving benign performance. Specifically, aim to ensure that the predictions for both the original input x and the adversarial exsample x + remain the same.
3.2 THREAT MODEL.
This paper investigates the vulnerability of pre-trained encoders in downstream classification tasks. We assume attackers exploit publicly available encoders, wheih can be obtained from the public platforms, to craft adversarial examples that transfer to any downstream model utilizing the encoder. These attacks are non-targeted, universal across downstream tasks, effective even after fine-tuning, and stealthy, requiring perturbations to remain visually imperceptible.
For defenders, they cannot control the pre-training stage and are unaware of the original training data or SSL method used in pre-training. However, they have full access to the released encoder and complete control over downstream fine-tuning, including architectural adaptation.
We propose stronger defender's objective: improving downstream robustness without sacrificing benign performance. Our defender's goals can be summarized as: 1) Zero-Sacrifice Accuracy; Maintain (or even improve) benign performance without degrading clean accuracy; 2) Adversarial Robustness: Resist adversarial examples generated from pre- pre-trained encoders and maintain correct predictions; 3) Resource Efficiency: Avoid reliance external data additional computational costs beyond standard fine-tuning; 4) Persistent Robustness: Enable a single adversarially-tume encoder to remain robust across diverse downstream tasks without task-specific retraining.
3

---

Published as a conference paper at ICLR 2026
Attacker
User
Trankigg
Step E: Encoder Pesporation
Robust Federol Beckion Mechasime
Server 269 Cenretey Risrer
Pres Tratshry
Advensorial Proporation
Leeining
Pre-Prakenz
Bongg Properation
RMMP-Branch
The overview of the proposed method. The red and green arrows denote the attack flow and normal flow the inference stage, respectively.
Figure 1:
3.3 CHALLENGES
Unlike previous works, we aim to enhance adversarial robustness without sacrificing benign per- formance, and even expect to improve Furthermore, we aim to establish a persistent-robustesss paradigm that enables one-time tuning process while ensuring robustness across all downstream tasks. However, just re-tuning the pre-trained encoder may compromise its powerful feature extrac- tion capabilities. Hence, achieveing goal is challenging, which can be outlined as:
(I) Challenge I: Zero-Sacrifice Conflict: Existing adversarial fine-tuning methods inevitably de- grade benign accuracy due the inherent conflict between robustness optimization and clean per- formance. Our work targets the more challenging zero-sac cero-sacrifice objective that achieving robustness improvement without harming benign accuracy, despite limited access to the pre-training process.
(2) Challenge II: Persistent Robustness Across Tasks: Most defenses require task-specific tuning and cannot generalize across downstream tasks. We introduce persistent robustness, aiming for "tune once, defend all" capability, enabling single tuned encoder to remain robust across diverse downstream tasks without re-training or task-specific assumptions.
The detailed threat model and challenges are provided in Appendix B
4 METHODOLOGY
4.1 ZEPAD: A COMPLETE ILLUSTRATION
In this section, we introduce ZePAD, the first zero-sacrifice, specifically designed for pre-trained encoders. The pipeline the proposed method is illustrated in Figure [ and it comprises two primary steps: encoder preparation and RFDM. These steps are elaborated upon in the following subsections.
defense method
4.1.1 STEP 1: ENCODER PREPARATION
The core motivation behind ZePAD in achieving DAEs is leverage the inherent confidence bias within neural networks. Therefore, the first step in ZePAD is encoder preparation. Since the defender can modify the local encoder, the goal of this step build feature extractor with both strong representation capabilities and robust adversarial resilience. Specifically, encoder consists of the SIPAE-Branch and the BMP- Branch, which are designed to enhance adversarial robustness and improve benign performance, respectively.
robusstnessssss zero-sacrifice defense against
DAEs exploit the model's sensitivity to specific patterns and textures, but it is hard to learn common sensitive patterns across multiple encoders, trained with different SSL methods. The diversity in feature representations learned by different encoders makes it difficult for DAEs to pinpoint shared vulnerabilities. This significantly complicates the task of exploiting common weak- nesses across all encoders. Hence, MPAF-Brann consists of two public pre-trained encoders.
4

---

Published as a conference paper at ICLR 2026
To ensure adversarial robustness, both encoders must be fine-tuned to strengthen their resilience to adversarial attacks. However, pre-trained encoders are highly sensitive to fine-tuning, which presents dilemma: how improve adversarial robustness while preserving the original pre-trained knowledge. To address this challenge, we propose hybrid loss that strikes a balance between preserving the pre-trained encoders" knowledge and enhancing the adversarial robustness. The loss function is formulated as follows:
(3) where Le represents the classification loss of adversarial exsamples, and C represents the encoding feature loss of adversarial exsamples. À is a hyperparameter. Then, Le is calculated as follows:
Lo + XCz,
Le 3) Lee (F& (Eg (a ()) 0)).
where & denotes the encoder, and JF represents the downstream classification model. æ and y stand for the data and labels, respectively. To better describe the loss caused by encoding features, we first define the distance between two encoding features. We use cosine distance to measure the feature differences between them. Additionally, we we remove the nearest features to prevent local high-density data from overly influencing the loss value and to better describe the global structure. This process formulated as follows:
(c0, T.)) Paj))
Dij ΣN Ek=1, (2 (cos(1;, xk)) - psa))
(5)
where cos(r;, 2T) denotes the cosine distance between 2; and Tj.- and Pay denotes the minimum distance between vj and all samples in the training dataset. Then, the loss be formulated as:
D2g Dig
1 1-D0 1 Dj
]
f,=
D log
+ +11 D',)) log
(6)
where D and D represent the distances between the adversarial exsample pair( æzy) and their corresponding benign sample pair, respectively. This makes the feature distribution of adversarial exsamples after encoding similar to that of benign samples.
B.MP-Branch: To address this challenge, we introduce benign branch. this step, we train an additional encoder locally using only benign data. By focusing on local data, the benign branch can memorize and capture crucial features that are essential for maintaining strong performance on clean, non-adversarial inputs.
Overall, our encoder & consists of three sub-encoders: Ed,1, Ea,2, and &.- Here, E,, and E, represent the publicly shared and adversarially tuned pre-trained encoders, while E, refers to the locally trained benign encoder. As assumed, the downstream task is a classification problem. The classification heads can be trained based on these sub-encoders using the local data. This process allows us to obtain classification heads Ha,1, Ha,2, and Hs, which are responsible for mapping the features from E, Ea,,, and Es, respectively. A key challenge now is how effectively combine these sub-encoders to achieve the zero-sacrifice goal.
4.1.2 STEP 2: ROBUST FEDERAL DECISION MECHANISM
Our method builds on the observation that neural networks implicitly estimate posterior confidence. Consequently, they assign higher confidence to samples aligned with their training distribution and lower confidence -ittr--distribution adversarial inputs. This inherent behavior leads natural confidence separation between branches for different kinds of data. To support our observation, the theoretical proof can be found in Appendix C
To fully leverage this observation, we utilize two branches trained for different sample types: the MPAF-Braach for adversarial exsamples and the BMP-Branch for benign ones. This design lever- ages the principle that neural networks show higher confidence on inputs consistent with their train- ing data (Carlini et al. 2019). The HPAE-Branch is adversarially fine-tuned for robustness, while the BMP-Branch, trained solely on benign samples, is highly sensitive to clean data.
Our experiments also support our assumption, as illustrated in Figure 2 On benign samples, the EMPP-ranch consistently exhibits higher confidence. Conversely, the BAAA--Banch's 's adversarial training. while enhancing robustness, slightly compromises its generalization on clean data, resulting in lower confidence.
5

---

Published as a conference paper at ICLR 2026
Table 1: BA (%) comparison between the bascline method and ZePAD in the semi-black-bor setting,
where pp D and Da denote the pre-training, adversarialt adversarial training and downstream datasets, respectively.
Dy and D2
Method
W-MSE
BYOL,
NNCUR
SimCLR
MoCr2+
MoCa3
SupCon
RESSI.
DINO
SwAV
VitCrng
AVG
bascline
50.17
54.39
91.77
92.35
95303
$4,21
90.22
92:75
91.23
92.25
92.39
23,35
CIFARIO
93.42
93.49
93.96
95.05
93.79
54.27
92.39
92.87
93.36
93.51
93.12
93.57
+3.25
-0.90
+1..19
+3.49
-1.24
-11.44
-3.53
40.14
+1.93
+1.22
40.73
+0.23
haseline
78,49
83.08
R1.09
$1.94
84:46
$4.00
85.29
81.87
83.54
81.53
61
62.63
STL.O
ours
$1.43
82. 20.30
62.52
84.55
81.52
63.51
82.90
82.10
$1.42
81.87
63-65
$2.51
A
-2.94
0.78
0.53
+2.44
2.94
0.49
2.39
+0.25
2.12
+0.34
+2.01
0.12
50.39
10122
91.22
50.82
3300
53,115
85.25
87.86
35501
89.16
94.20
S5.52
ANIMALSIO
ours
97.35
98.29
97.84
97.92
98.56
98.41
97.29
98.67
97.76
98.07
98.86
98.14
CIFARIO
73.76
+6.995
+6.07
75.09
79.06
66.63
-5. 10
78.61
+10.53
76.37
+10.36
K1.11
+9.56
K1.65
+10.81
76.03
+12.75
30.63
+8.91
85.51
82225
+6.16
+9.62
79.18
GTSRD
95.34
93.58
94.20
94,99
94.78
94.38
96.25
94.12
94,41
95.54
94,09
94,51
+21.58
+17.59
+15.14
+13.27
+12.58
+18.09
+13.78
+10.00
+11.4
+15.34
baselin
05.34
57.74
65.00
5337
05.45
02:31
63,05
ImageNet22
73.83
72.88
73.09
74.05
74.45
74.45
72.54
72.39
72.94
73.41
73.41
+10.44
+10.32
SVII
ours
94,92
68:00
70.56
20.35
71.35
71.20
21:75
08:46
CIFARIO
ours
03.03
95.05
93.07
93.46
93.88
95.02
93.12
A
+36.51
+26.45
+21.01
+32.51
+23.11
+21.51
+22.09
+21.25
+31.30
+24.56
+19.77
+23.6
haelline
51.99
63.37
65.41
66.14
64.78
63.21
64.79
64.89
64.37
62.90
68 65.76
63.69
STLIO
curs
50.08
76.82
81.14
$2,07
81.60
$2.94
79.35
80.22
80.77
80.58
81.35
88.63
A
+28.09
+13.45
+15.75
+35.93
+16.32
+19.75
+14.56
+15.35
+16.40
+17.68
+12.59
116.94
Tesclin:
43.33
6532
65.81
7030
6526
65.21
74365
64:75
62.19
61.75
76.31
65.97
ANIMALS10
CMr
97.28
97.70
95. 30.30
97.86
97.70
96.35
95.55
96.05
98.38
97.73
95.61
98.03
ImageNet
A
+68.75
57.17
+33.88
67785
+29.49
72361
+27.17
07.35
+31.84
67:21
+30.04
74.85
+23.00
7KDS
+33.30
X119
-35.99
60:34
+35.08
72:17
+22.04
77795
+32.33
XLTI
GTSRD
ours
94,56
92.84
94.34
94.98
98.12
94,33
94.87
91.79
94.55
94.76
94.45
94,35
A
+42.39
+25.19
+21.71
+25.62
+26.91
+19.45
+15.82
+23.70
+25.61
+22.59
+15.48
+24.22
baselme
35.04
45.76
36.18
3c.44
32.35
$4.30
47.30
31.29
21.13
47.24
3430
50.1
ImageNer20
ours
72,80
72.75
72.99
73.97
75.34
75.69
71.20
73.16
72.33
71.91
69.81
72,90
A
+37.16
+33.09
+1681
+17.53
+32.09
+21.33
+23.80
+21.67
+21.20
+24.07
+1535
+22.39
Tenelon
50545
6236
68
00:25
02:50
64.31
67:30
63.05
71.31
66:72
69:51
65.55
SVIN
94,16
93.60
94.05
94,92
95.20
94.75
94.36
94.57
94.21
94.03
94.03
94.35
+30.74
+25.07
+22.90
+29.00
On adversarial exsamples, the roles reverse. The BMP-Branch's 's confidence drops significantly as it is vulnerable to the versatile attacks. In contrast, the MPAE-Branch maintains the highest confidence, as its adversarial fine-tuning makes it resilient to these spe- cific inputs.
Ist whiig
te Athamara CC
This observed phenomenon, where the branches ex- hibit complementary strengths, is the foundation of our proposed approach. We leverage this insight to com- bine their outputs, aiming to maximize both adversar- ial robustness and performance on benign data. The decision process formulated as follows:
1/ Reeign wi
66 Anserrnn cappl CI
Figure 2: The confidence distributions of each branch under different conditions. C- indicates fine-tuning on CIFARIO and downstream testing on CIFAR1O; C-S in- dicates fine-tuning on CIFAR10 10 and down- stream testing on STL.10.
C:
W
Σ
where W; represents the confidence score of the i-th branch, while e; represents the confidence value of the i-th branch. Specifically, the calculation method for c is as follows:
(8) where m, represents the maximum value in the probability vector output of i-th branch. However, argue that the confidence value does not change linearly with the maximum probability. Instead, we posit the maximum probability value is positively correlated with the confidence value. Besides, as the maximum probability value approaches 1, the rate which the confidence value increases should accelerate. To help readers follow our work, the algorithm can be found in Appendix D
G m,e3(m
5 EXPERIMENTS
5.1 EXPERIMENTAL SETTING
Attacks. We evaluate the effectiveness of our work using the following five general adversarial attacks: AdvEncoder (Zhou et al. 2023). PAP (Ban & Dong 2022), UAP (Moosavi-Deztooli et al.
2017), UAPGD (Deng & Karam 2020) and SSP (Naseer et al. 2020).

---

Published as a conference paper at ICLR 2026
Table 2: RA (%) comparison between the baseline method and ZePAD in the semi-black-Be setting,
where Dp. Dy and Da denote thepre-training adversarialt adversarial training and downstream
datasets, respectively
Dy and D2
Method
W-MSE
BYOL,
NNCLR
SimCLR
MrCr2+
MoCr3
SupCon
RESSI.
DINO
SwAV
VitCmg
AVG
bascline
10.05
12.47
10.71
47:13
45.01
20.85
1530
12:05
10:36
21.07
14.77
22:41
CIFARIO
56.84
80.18
70.84
74,20
69.70
72.49
77.99
84.58
80.44
74.61
70.62
78.27
5
+76.75
+67.71
460.13
427.57
+34.69
+35.63
+61.99
+72.53
469.98
+42.94
465.55
+55.66
haseline
41.16
46.85
44:05
67.52
65.55
64.28
$6.89
51.69
11:77
59.55
43.07
$2.05
STL,O
ours
70.25
71.30
72.66
77.34
74.01
74.89
75.17
73.74
71.23
72.12
71.18
73.18
A
+29.12
+24.45
+28.61
+9.82
+8.46
+10.61
+18.28
+22.05
+39.96
+12.57
+28.11
+21.99
48.31
49225
40:25
63.45
6303
00.32
35.48
34.92
35.46
65.75
30.15
$4.22
ANIMALSIO
ours
91.94
90.49
M6.71
87.94
88.34
84,58
88.52
19.02
89.01
87.88
K8.65
88,46
CIFARIO
+43.63
10.85
+41.34
12.41
+57.48
11:29
+34.46
34.63
+25.3
27.00
+24.26
27.60
+30.04
1322
+34.10
12.37
+53.55
10.62
+26.13
27.82
+38.46
11:56
+34.34
17.22
GTSRD
75.00
79.25
73.95
74.59
74.15
73,54
72.15
72.47
80.89
76.04
72.18
74.93
+51.15
+66.32
+62.66
+49.96
+37.03
+65.96
+58.95
+60.10
+70.27
+18.2
+60.22
+57.71
baselin
39.24
2339
35.10
21.24
ImageNet22
64.34
61.69
62.60
62.08
63.74
63.55
62.46
55.99
62.48
+33.45
+29.33
922
SVII
ours
91.26
+76.68
11.25
11.78
34.51
16.39
12.05
15.30
CIFARIO
ours
66.69
66.03
78.07
68.01
69:37
71.21
81.77
78.72
86.01
73.37
A
252.22
+58.91
+53.35
+53.76
+51.52
+56.71
+57.11
+20.46
+62.92
+33.57
+45.80
+5.48
hueliine
26.76
33.02
22.42
46.23
40:43
42.37
25.12
41.98
33:68
42.77
40.84
35.97
STLIO
curs
70.35
67.80
64.67
73.25
74.30
76.78
67.00
71.90
70.38
75.18
69.36
78,95
A
+53.59
+34.78
+42.25
+27.05
+33.87
+34.41
+41.38
+29.92
+36.50
+:2.41
+28.52
+35.01
Teeclin:
31.65
45-20
37.41
$3.24
47.71
46.21
37.45
44.26
401.00
41:23
25.36
42.05
ANIMALS10
CMr
67.54
84.00
83.23
$3.63
X6.55
67.31
91.28
83.52
39.49
91.70
$4.92
87.11
ImageNet
A
+5.85
8.78
+40.50
1517
+45.76
1023
+35.39
19:25
+38.84
1571
+41.10
14.35
+53.85
10099
+39.26
17:75
+48.79
10:30
+50.47
21.39
+35.58
17:32
+44.15
14.22
GTSRD
ours
69.38
66.69
60.51
69.33
73.65
76.34
69.11
69.03
67.39
$1.41
68.12
70,09
A
+50.60
+51.52
+50.28
+50.10
+55.02
+h1.98
+58.12
+55.28
+36.89
+59.92
+:0.80
+55.86
baselmn
19.33
25.37
1266
30.00
29.44
23.31
1331
25.47
21.33
21.74
24.31
23.32
ImageNer20
ours
63.06
59.42
60.26
61,07
61.78
64.78
02:46
60.20
65.00
67.42
58.
62.15
A
+43.21
+34.05
+47.61
+30.38
+32.34
+36.44
+46.65
+31.73
+43.05
+45.68
+33.74
+38.63
6:44
6:26
6.72
26.38
1297
11.85
7134
6.55
19:58
15:45
19:42
12.79
SVIN
87.88
85.36
66.21
90.36
86.44
89.36
92.81
84.46
82.31
89.49
91.53
85.84
679.30
463.98
+72.11
Defense. We evaluate the effectiveness of our work using the following four adversarial training mechanisms: Gen-AF (Zhou et al. 2024). PGD-AT (Madry et al. 2018). TRADES (Zhang et al., 2019) and MART (Wang al. 2019).
Datasets and models. We evaluate our method on the six datasets: CIFARIO (rizhevsky et al,
2009). STL10 (Coates et al. (001T) ANIMALS10 (Alessio) 2020). GTSRB (tallkamp et al, 2012). ImageNet20 (Russakovsky et al.) 2015). and SVHN Netzer et al., 2011). We use the
publicly available pre-trained encoders from solo-tearr ?
well-established SSL library, as victim 2023 2024), we select eleven SSL methods (W-MSE (Ermolov)
encoders. Following (Zhou et af.]
cet al., 2021), BYOL (Grill et al.
2020 NNCLR (Dwibedi et al.) 202I). SimCLR (Chen et al.
2020a). MoCo v2+ (Chen et al.
2020b). MoCo v3 (Chen et al. 2021). SupCon (Khosla et al.,
2020 RESSL (Zheng et al. 1021). DINO (Caron et al.] 2021). SWAV (Caron et 20200, and
). All encoders are pre-trained on either CIFARIO (Krizhevsky et al., 2009) or ImageNet (Russakovsky et al. 2015), using ResNet18 as the backbone.
VibCreg (Lee & Aune |02I)
Evaluation Metrics. To evaluate the effectiveness of our proposed method, we examine the model's robustness and generalization capability using the following three metrics.
Benign Test Accuracy: Benign Test Accuracy (BA) denotes the classification accuracy on clean samples. Higher values indicate better generalization ability.
Robust Test Accuracy: Robust Test Aceuracy (RA) measures the classification accuracy on adver- sarial examples. Higher values signify greater robustness.
Attack Success Rate (ASR) is metric from the attacker's perspective, which provides insights into model robustness. It represents the proportion of adversarial examples where the model's prediction differs from that of the corresponding clean sample. Higher values reflect stronger attack performance.
Attack Success Rate:
Implementation details. Adversarial tuning is performed for 20 epochs with a batch size of 256, optimized by the Adam optimizer (Kingmal 2014). The classification head is composed of three fully connected layers. Following the previous work (Zhou et al. 2023), we select CIFAR1O as the
7

---

Published as a conference paper at ICLR 2026
Table 3: BA(%) comparison between the bascline method and ZePAD in the semi-black-be -bxx setting, where Dy and D denote the adversarial training and downstream datasets, respectively.
Supt.on
buseline
$5.21
+4.55
+1.88
-2.36
baseline
90.52
58.05
35.01
94.70
ANIMALSIO S10
91.31
02.37
08.18
92.01
92.64
91.52
91
92.00
92.92
92.13
A
+10.9
+2.11
+3 1555
+1.61
+3.98
+4.59
+3.29
+3.57
+6.53
+2.34
1,78
+3.62
baseline
73.76
73.90
79,05
78.51
76.37
31.11
$1.65
76.03
80053
85.31
K2.25
79.15
CIRARTO
GTSRB
82.50
83.17
83. K3.86
84.36
83.10
$4.49
K2.13
K2.71
82.25
M4.33
K3.41
$3.30
A
68.74
67.18
+4.30
+5:75
+6.73
+3.38
+0.48
e0.68
+1.82
-1.18
+1 61.16
+4.12
baseline
59.01
67.47
65.00 65.94
$7.74
66.22
65:00
58.87
61.
66645
65:01
62.31
63.09
ImageNer20
64.80
66.23
66.86
66.10
66.21
66.49
62.
64.47
65.42
66.63
62.35
66.33
A
+5.83
-1.24
+0.92
+5.36
00.01
+1.49
+3.69
+2.52
-1.03
+3.62
+0.45
-2.24
Baseime
37792
58.59
55.43
04.51
54.07
64.33
00708
87.85
70.36
72:37
73:25
SVIN
DEES
75.32
78.62
90,31
79.43
73.14
77.73
75.39
77.55
77.57
75.88
77.32
77,12
A
+17.30
+20.05
+13.85
-14.62
+8.47
+13.41
+5.51
+9.90
+7.49
+3.31
+4.06
+10.72
baseline
56.32
68.00
72.38
70.56
70.35
72.37
71.35
70.20
11.75
68.46
73.35
69. 09.72
CIFARIO
91.20
91.07
90,92
94,40
91.54
91.24
91.11
90,91
91.06
91.53
90.65
91.42
+38.87
biseline
31.99
63.37
53.43
64.75
63.21
64.79
54.89
6290
63.69
81.19
90.38
$1.19
80.91
79.05
+29.20
+17.01
+15.25
+17.95
+15.45
+17.11
baseline
ANIMALSIO
OLES
99.75
90.29
50.09
99.79
90:11
39.30
+28.25
+21.45
-18.36
+15.14
+25.36
+26.91
+13.06
bsselinc
48.75
54.36
ImageNer20
61.61
60.05
60:09
59.96
59.52
60:71
60.32
59.87
60.00
61.17
57.41
+35.07
+11.90
+7117
+12.92
+8.0 48.38
+13.33
+9,1
82.00
64.31
67.40
6195
71.31
87:11
63.15
SVIIN
DIES
81.67
80.35
$5.15
82,42
80.87
$2.40
79.53
19.19
82.87
81.80
82.29
$1.63
A
+31.22
+17.50
+13.75
-13.17
+18.37
+1..6
+12.13
+17.28
1.5.5
+15.08
+12.78
+16.50
dataset for the attacker and set the perturbation limit to 10/255. The other experimental settings can
be found in Appendix E
The default experimental setting follows a semi-black-bee scenario. When the victim encoder in the MPAE-Branci is not BOL,, we select B BOL as the adversarial auxiliary encoder for the MPAE- Branch. Conversely, if the victim encoder is B YOI,, we choose NNCLR as the adversarial auxiliary encoder (AdvA.-Model). The encoder in the BMP-Branch is trained using SimCLR with local datasets. Besides, we also validate our method under the white-box scenario, and the experimental results can be found in Appendix
5.2 SEMI-BLACK BOx SCENARIO
In this scenario, the attacker knows the SSL method employed by the victim model, as well as the pre-training dataset. We evaluate the performance of ZePAD across 11 SSL methods and 6 datasets, selecting AdvEncoder as the attack method due its status as a SOTA nownstream-agnoste sarial examples method. This method enables effective attacks without knowledge of the pre- traitre dataset or downstream tasks.
adver-
We present comparisons between ZePAD and the baseline in Tables [ and 2 where the adversarial training dataset and downstream task dataset are identical. The results show that ZePAD significantly enhances task perfor- mance on benign samples, with average ac- curacy improvements of 29.20% and 32.03% on the SVHN and ANIMALS10 datasets, re- spectively, when the pre-training dataset Im- ageNet. In contrast, the baseline's accuracy drops significantly on adversarial exsamples, with RA falling below 10% in some experi- mental settings, underscoring the threat posed by adyEncoder-generaaee encoders. As shown in Table 4 ZePAD sig- nificantly improves RA. For instance, when the pre-training dataset is CIFAR 10 and the down- stream task is SVHN, RA increases by 73.86% compared to the baseline, reaching an average RA of 86.97% This demonstrates that ZePAD
(n Denige Sumploc with Tuadine
(hi Advorconal Samppao with Bastine
DAES to pre-trained
BebAD
Figure 3: t-SNE visualization of feature spaces for the baseline and ZePAD under benign and adver- sarial examples. Different colors indicate differ- ent classes.
8

---

Published as a conference paper at ICLR 2026
Table 4: RA (%) comparison between the baseline method and ZePAD in the semi-black-bol boxx setting,
where Dy and D denote the adversarial training and downstream datasets, respectively.
Methed
BYOL.
NNCCR
SuptCon
DINO
Vilcreg
buseline
46.85
51.69
31.77
59.55
52.003
UnES
72.84
77.12
70.54
12.65
73.67
436.45
+28.79
+9.00
+8.17
+13.87
+22.30
+14.49
427.76
+21.16
bascline
48.31
49:25
49.23
63.48
63.03
64.32
35.46
50.19
ANIMAL S10
75.49
72.15
74.21
75.33
60.35
70.76
74.60
77.62
74.71
73.33
75 75.90
73.94
A
+27.18
+22.90
+24.95
-11.75
+5.32
+10.64
+16.12
+22.70
+39.33
+9.58
+25.71
+19.72
baseline
10.35
12:41
11.29
25.33
27.02
27.50
13:22
12:37
10.62
27382
11.36
17:22
CIBARTO
GTSRB
61.50
56.16
53.71
54.24
45.10
52.58
53.85
59,50
56.48
54,03
55.87
55.00
A
+99.65
+43.75
+42.42
-29.61
+23. 08
+24.95
+40.61
+47.11
+45.36
+20.21
+44.31
+37.78
baseline
27.90
$4.41
20.82
18.34
4287
41.50
25.97
35.10
21.34
37.69
27.15
33.15
ImageNer20
48.78
53.97
55.
52.18
50.16
49.40
50.98
$2.97
$2.05
52.71
46. 46.34
51.34
A
-21.18
+19.56
+25.3
-13.94
+7.29
+7.80
+22.01
+17.87
+30.81
+15.02
+19.21
+18.16
Baseime
9.76
11.68
14.99
14.55
21.31
10.35
11:10
9.21
8:95
15318
10:55
13:11
SVIN
DEES
55.84
61.37
52,97
64.75
50.86
59.39
62.74
68.26
64.37
58.24
55.34
59.47
A
+46.68
+49.69
+57.95
-50.17
+20 35
+43.01
+51.64
+58.09
+55.42
+42.16
+44.79
+46.35
baseline
11.78
11.78
12.80
25.31
16.49
12.66
14,00
11:33
15.30
12.48
20.42
14,89
CIFARIO
53.62
62.57
64.43
60.50
63.91
67.67
60.76
51.94
64.32
65.74
63,80
62.66
+52.63
+55.01
+43.36
147.77
biseline
33.02
21.42
46.23
42.57
25.12
41.95
33.28
42.77
40.84
35.97
STL.00
$7.90
65.88
69.58
66.16
66.77
65.32
+52.86
445.27
+27.21
443.04
+34.50
+25.36
+24.48
+31.06
baseline
32.68
37.47
53:04
46.21
GTSRB
ANIMALSIO
DLES
65.49
70.52
72.78
72.67
70.85
71.77
71.65
73.85
71.47
72.37
71.96
71.40
+27.32
+35.31
-19.43
15
+25.55
+34.22
+29.60
+30.77
+31.14
+23.60
+28.45
bsselinc
30.59
1581
2138
ImageNer20
39.44
45.67
48.10
48.81
47.53
48.83
46.13
46.30
48.30
47.61
43,83
46.41
+19.59
+20.30
+35.45
-1
+1.009
+20.49
+30.32
+17.83
+36.35
+25.7
+19.90
+22.89
6.25
6.74
36.38
1297
V11
6.35
19.38
SVIEN
DIES
55.72
68.17
71.15
63.38
65.98
67.89
62.79
69.54
74.75
63.19
57.09
63.42
A
+19.28
+61.91
+64.43
-37.00
+5..01
+56.01
+53.75
+62.99
+55.07
+47.75
+37.67
+52.63
can effectively enhance model robustness while improving generalization, achieving Zero-Sacrifice Adversarial Defense.
To investigate ZePAD's cross-task defense capabilities, we design experiments where the adversarial training dataset differs from the downstream task dataset, with CIFARIO uniformly selected as the pre-training dataset. The experimental results are provided in Tables B and 4 The results demon- strate that ZePAD can still enhance BA compared to baseline, even when the adversarial training dataset and downstream dataset are mismatched. For example, when the fine-tuning dataset is GT- SRB, BA improves by an average of 21.70% and 23.82% on CIFARIO and ANIMAL downstream tasks, respectively. Against adversarial examples, ZePAD continues to perform well, significantly boosting RA. When fine-tuning on CIFARIO, RA averages 73.19% on STL10 and 73.94% on AN- IMALS10, showing robust defense. This confirms ZePAD's effectiveness across tasks, achieving Persistent-Robustier adversarial defense.
To intuitively understand ZePAD's impact on model generalization and robustness, we conducted SNE visualization when the pre-trained dataset was CIFAR10 and the downstream task was SVHN with results shown in Figure 3 It is clear that ZePAD sharpens classification boundaries be- nign samples. When facing adversarial exsamples, the baseline method almost loses classification capability, but ZcPAD can still correctly classify samples.
5.3 ABLATION STUDY
In this section, we investigate the impact of different components of ZePAD. We use a pre-trained DINO encoder on CIFARIO and conduct experiments on ANIMALS1O.
Table 5: The ablation study of the proposed ZePAD.
Adversarial Fine-Tuning Tuning
MPAD
BMP
BA
RA
ASR
Branch
Branch
85.01
35.46
63.35
90.84
59.40
38.89
We analyze the effects of adversarial train- ing, the MPAD-Branch, and the BMPP-ranch on the overall method, with results in Ta-
x
91.57
70.35
26.71
/
x
84.51
72.47
19.36
4
V
91.66
74.71
21.99
ble [ Removing adversarial training signifi- cantly reduces RA and increases ASR, indicating its effectiveness in enhancing robustness. Nonethe- less, BA and RA still improve compared to standard testing. showing that pre-trained encoders from different self-supervised methods have complementary feature extraction patterns. When MPAD- Branch is removed, RA decreases with little change in BA. highlighting its role in boosting robust- ness without harming generalization. If levels, indicating that adversarial training can negatively affect generalization. Thus, BMP-Brand is crucial for maintaining and enhancing the model's generalization capability. The other ablation studies can be found in Appendix G
is removed, BA drops below standard testing

---

Published as a conference paper at ICLR 2026
Table 6: DAE sample detection accuracy($)), where D, and Da denote the adversarial training and
downstream datasets.
27 and D2
W-MSI
BYOL
NNCLR
SimCLR
MaCo2+
MoCo.
SupCon
RESSL
DINO
SwAV
VahCreg
AVG
CIFARIO
86.20
83.71
83.46
86.18
81.53
82.81
84.27
83.06
70.97
81.50
84.01
82.52
STL10
72.96
76.41
75.05
79.65
72.38
75.96
78.93
73.59
59.75
72.98
79.43
74.28
ANIMALSIO
80.91
81.55
80.31
81.98
74.57
76.66
83.32
81.46
71.02
73.11
84.55
79.04
73.93
72.09
Ne12
72.25
77.99
76.65
81.23
73.49
76.48
82.67
77.71
64.08
72.21
83.09
76.17
SVHN
59.26
64.93
63.82
70.75
62.17
66.69
67.75
62.69
52.38
64.07
68.45
63.91
5.4 DETECTING DAES BY CONFIDENCE SCORES
We find that the significant confidence score differences between the MPAE and BMP branches for benign and adversarial exsamples, as illustrated in Figure [ Then, we are curious that if we can utilize the inherent sensitivity to detect DAEs directly without additional DAE identification training.
Specifically, we define sample a DAE if the confidence of the MPAE-Branch exceeds that of the BMP-Branch by threshold of 0.1. The adversarial exsamples in this experiment were generated from AdvEncoder (Zhou et al.) 2023). The results in Table [ demonstrate high effectiveness. We achieve an average accuracy of 82.52% when the fine-tuning and downstream datasets are consis- tent (CIFARIO) and 79.04% when they are not (ANIMALS10) The higher accuracy with consistent datasets attributed to smoother confidence score distributions, which allows for clearer distinc- tion between benign and adversarial inputs. Additionally, to further discuss the extracted pattems from different branches for various inputs, the visualization can be found Appendix H
5.5 COMPARISON STUDY
In this section, we compare the proposed ZePAD with other advanced adversar- ial defense methods: Gen- AF, TRADES, MART, and PGD-AT. We use Ima- geNet for pre-training and STL10 as the downstream dataset. As shown in Ta- ble 7 ZePAD outperforms existing methods in both robustness and generaliza- tion. ZePAD's s BA remains above 80%, indicating its effectiveness in boosting model generalization. 70% RA against six attacks, outperforming all other
Table 7: Comparison study(%)
Model
Method
BA
LAP
R.A
LAPGD
RA
SSP
RA
PAP
RA
Advinacnder
RA
201
MART (CLR)
42.79
42.50
12.65
42.60
42.24
42,36
SinCER
TRADESI 2019 PMLR)
Gen-AF(2021 S&P)
68.69
50.79
61.43
50.40
68.65
50.51
65.92
50.95
65.48
50.59
62,28
50.66
ZaPADiours)
77.56
A
+13.38
+13.13
+7.86
+7.01
+6.62
+10.98
PF ICLR)
13.72
12.13
13:48
12.30
13.02
12.70
2ART
MoCo v2+
TRADES(2019 2019 PMIR))
Gen-AR (emn-AF(2024 2024
SRPP)
69.09
50.50
64.98
52.40
50.60
68.15
59.41
52.37
$2.60
60.47
66.00
52.28
ZoPADiours)
81.60
76.05
74.83
72.15
67.96
74.30
+12.51
+7.49
TA represents improvement ratio compard the sccond-bent metric
In terms of robustness, ZePAD achieves methods and further demonstrating its zero-sacrifice characteristic, Besides, the computational cost comparison can be found in Appendix ! Finally, we extend our defense method to other tasks, and the corresponding discussions and experiments are provided in Appendix [
6 CONCLUSION
In this paper, we introduced ZePAD, the first pero-sacrifice persistent-robusteess. adversarial defense method designed for pre-trained encoders. The core idea behind ZePAD is to enhance adversar- ial robustness and benign performance by simultaneously leveraging two complementary branches. Our experimental results across various SSL methods and multiple datasets demonstrate that ZePAD provides robust performance in different settings, effectively enhancing adversarial robustness with- out sacrificing benign performance. Additionally, ZePAD's persistent-robustness capability enables it to adapt to new tasks with a single tuning, making it scalable and adaptable to real-world appli- cations. Further investigation is required in several areas. One important direction for future work the ZePAD framework and DAEs. Gaining insights into them
improving the interpretability could help refine the model and offer more transparency.
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. In this study, no human subjects or animal exper- imentation was involved. All datasets used, including CIFARIO, STLI0, ANIMALS1O, SVHN, ImageNet and GTSRB were sourced in compliance with relevant usage guidelines, ensuring no vio- lation of privacy. We have taken care to avoid any biases or discriminatory outcomes our research process. No personally identifiable information was used, and no experiments were conducted that could raise privacy or security concerns. We are committed to maintaining transparency and integrity throughout the rescarch process.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure that the results presented in this paper are reproducible. Specifi- cally, our method is detailed in Section and the main steps are provided in Appendix(Djo facilitate reproducibility. Moreover, all code will be made publicly available after the review cycle to ensure the reproducibility. Details of the experimental settings, including datasets, models, SSL methods, training configurations, and hyperparameter choices, are presented in Section [ Appendix All datasets used in our experiments publicly available, and the download links for all open-source pre-trained models involved the experiments are released together with our code.
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant 62271332 1335, in part by the Sichuan Science and Technology Program under Grant 20252NSFSCO470 and in part by Tianfu Jiangxi Laboratory.
REFERENCES
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing reduuiin reprccettttce collapse. arXiv preprint arXiv:2008.03155 2020.
Corrado Alessio. Animals-10. 10. http://www..agggccoomm.msoooooom August 2020. Kaggle dataset.
Yuanhao Ban and Yinpeng Dong. Pre-trained adversarial perturbations. Advances in Neural Infor- mation Processing Systems, 35:1 196-1209, 2022.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14166 2020.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX security symposium (USENIX security 19), pp. 267-284, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912 12-9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEFE/CVF conference computer vision and pattern recognition, pp. 15750-15758 2021.
11

---

Published as a conference paper at ICLR 2026
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self- self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9640-9649, 2021.
Kenneth Co, Luis MunozeGonzález, Sixte de Maupeou, and Emil C Lupu. Procedural noise adversarial examples for black-box attacks deep convolutional networks. In Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, pp. 275-289, 2019.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statistics, pp. 215 223. JMLR Workshop and Conference Proceedings, 2011.
Yingpeng Deng and Lina Karam. Universal adversarial attack via enhanced projected gradient descent. In 2020 IEEE International Conference Image Processing (ICIP), pp. 1241-1245. IEEE, 2020.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighhor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9588-9597, 2021.
Alcksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In International conference on machine learning, pp. 3015- 3024. PMLR, 2021.
Enrico Fini, Victor G Turrisi Da Costa, Xavier Alameda-Pineda,, Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-supervised models continual learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9621-9630, 2022.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 1rXiv:1412.6572, 2014.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21 2020.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten. Countering adversarial images using input transformations. arXiv preprint 1rIivv 17, 2017.
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International conference on machine learning, pp. 2712-2721. PMLR, 2019.
Shengshan Hu, Xiaogeng Liu, Yechao Zhang, Minghui Li, Leo Yu Zhang, Hai Jin, and Libing Wu. Protecting facial privacy: Generating adversarial identity masks via style-robust makeup transfer. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp-
15014-15023, 2022.
Tianyu Huang, Bowen Dong, Yunhan Yang. Xiaoshui Huang, Rynson WH Lau, Wanli Ouyang, and Wangmeng Zuo. Clip2point: Transfer clip to point cloud classification with image-depth pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 22157-22167, 2023.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Samaa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. Supervised contrastive learning. Advances in neural information processing systems, 33:1 13:11616-1877, 2020.
Diederík Kingma. Adam: A method for stochastic optimization. arXiv preprint 11X.::1412.6980, 2014.
12

---

Published as a conference paper at ICLR 2026
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint 2rXi::2202.10054. 2022.
Alex Lamb, Vikas Verma, Juho Kannala, and Yoshua Bengio. Interpolated adversarial training: Achieving robust neural networks without sacrificing too much accuracy. In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security, pp. 95-103, 2019.
Daesoo Lee and Erlend Aune. Vibcreg: iiiiane-inaariaee--eette----arae self-supervised learning on time series. arXiv preprint arXiv:2709.000783, 2(5), 2021.
regularization for
Fengpeng Li, Kemou Li, Haiwei Wu, Jinyu Tian, and Jiantao Zhou. Dat: Improving adversarial
robustness via generative amplitude mix-up in frequency domain. Advances in Neural Information Processing Systems, 37:127099-122 10000-172220 128. 2024.
Zijin Lin, Yue Zhao, Kai Chen, and Jinwen He. I don't know you, but i can catch you: Real-time defense against diverse udversarial patches for object detectors. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp. 3823-3837, 2024.
Feng Liu, Bo Han, Tongliang Liu, Chen Gong, Gang Niu, Mingyuan Zhou, Masashi Sugiyama, et al. Probabilistic margins for instance reweighting in adversarial training. Advances in Neural Information Processing Systems, 01222288-22663 2021.
Hongbin Liu, Jinyuan Jia, and Neil Zhenqiang Gong. (PoisonedEncoder):: Poisoning the unlabeled pre-training data in contrastive learning. In 3/ USENIX Security Symposium (USENIX Security 22). pp. 3629-3645, 2022.
Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou. Clip-driven universal model for organ segmenta- tion and tumor detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21152-21164, 2023.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations, 2018.
John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International conference machine learning, pp. 7721-7735. PMLR, 2021.
Seyed-Mohsen Moosavi-Dezfooli. Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1765-1773, 2017.
Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. A self- supervised approach for adversarial robustness. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 262-271, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature
feature learning, volume 2011, pp. 4. Granada, 2011.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 0748-8763. PMLR, 18 24 Jul 2021. URI httph//ps///oceeedggs..r..ereeee 30rt
13

---

Published as a conference paper at ICLR 2026
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015.
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Bench- marking machine learning algorithms for traffic sign recognition. Neural networks, 32:323-332, 2012.
Chenxin Tao, Honghui Wang, Xizhou Zhu, Jiahua Dong, Shiji Song, Gao Huang, and Jifeng Dai. Exploring the equivalence of siamese self-superrvsed learning via unified gradient framework. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp-
14431-14440, 2022.
Yisen Wang. Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International conference on learning represenlations, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 1611:3361..03500 2016.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference machine learning, pp. 7472-7482. PMLR, 2019.
Jiaming Zhang, Qi Yi, and Jitao Sang. Towards adversarial attack vision-language pre-training models. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 5005- 5013, 2022.
Tong Zhang. Statistical behavior and consistency classification methods based on convex risk minimization. The Annals of Statistics, 32(1):56-88 85, 2004.
Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Ressl: Relational supe-supe learning with weak augmentation. Advances in Neural Information Processing Systems, 34:2543-2555 2021.
Ziqi Zhou, Shengshan Hu, Ruizhi Zhao, Qian Wang, Leo Yu Zhang, Junhui Hou, and Hai Jin. Downstream-agnosse adversarial examples. In Proceedings of the IEEE/CVF International Con- 2023.
ference on Computer Vision, pp.
Ziqi Zhou, Minghui Li, Wei Liu, Shengshan Hu, Yechao Zhang, Wei Wan, Lulu Xue, Leo Zhang, Dezhong Yao, and Hai Jin. Securely fine-tuning pre-trained encoders against adversarial examples. In 2024 IEEE Symposium on Security and Privacy (SP), pp. 3015-3033, IEEE, 2024.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression, arXiv preprint arXiv:1710.002. 2017.
14

---

Published as a conference paper at ICLR 2026
Appendix of Paper:
Zero-Sacrifice Persistent- Adversarial Defense for Pre-Trained Encoders
This appendix provides key information help enhance the understanding of our ZePAD. To help readers better understand the context of this field, we first provide the related works in Appendix A Then, Appendix [ covers the preliminaries, including the threat model, the defender's s objectives, and the associated challenges. We give through theoretical proof to support our proposed method Appendix [C] Then, in Appendix D we outline the main steps of the proposed ZePAD. To help readers reproduce our proposed method, we give the detailed experimental settings in Appendix E Additional experiments white-box scenarios are presented in Appendix(F] Further ablation studies The extracted patterns from different SSL methods are visualized in Finally, discussions regarding resource occupancy and consumption can be found in Appendix
are illustrated in Appendix G
Appendix H
A RELATED WORKS
A.1 SELF-SUPERVISED LEARNING
SSL is emerging machine learning paradigm that leverages vast amounts of unlabeled data to pre-train a generic encoder, which can be applied to various downstream tasks, such as image classi- fication and object detection. This approach overcomes the limitations of labeled data and achieves robust feature extraction, enabling users with limited resources to utilize pre-trained encoders with- out training from scratch. Fine-tuning, which generally lightweight, is the primary method used to adapt these encoders to specific tasks.
Fine-tuning pre-trained encoder enhances its adaptability downstream tasks. Common fine- tuning methods include linear evaluation (Aghajanyan et al. [0200] Kumar et al. (2022). which in- volves adjusting only the last layer, and full fine-tuning (Aghajanyan al. 2020; Kumar al., [022] Hendrycks et al. 2019. Miller et al. 2021), which involves tuning all layers of the encoder.
Existing ceff-supee learning approaches can be broadly categorized into several types (Fini et al. 2022; Tao et al., (2022). Contrastive learning methods, such as MoCo (Chen et al., 2020b 2021 and SimCLR (Chen et al., 2020a)), train models by ensuring that feature vectors of similar samples are close, while those of dissimilar samples are distant. Negative-freet methods, such as BYOL Grill et al.] 2020) and SimSiam (Chen & He 202I) improve representation by maintain- ing consistency between positive samples without the of negative samples. Clustering based methods, including SwAV (Caron et al. 2020), employ traditional clustering techniques to group similar samples into the same category. Redundancy reduction-bars
methods, such as W-MSE (Er- molov et af., (021) and VlbCReg (Lee & Aune (021). enhance the quality of representations by strengthening connections within the same dimension and attempting decouple across different dimensions.
A.2 ADVERSARIAL ATTACKS AND DEFENSES.
Recent studies GGodffllow et al. 2014 Hu et al. 2022) have revealed the vulnerability of deep
neural networks (DNNs) to adversarial examples. Universal adversarial perturbations (UAPs) (Co
et al. 2019) possess attack universality, as a single perturbation can affect multiple samples. Ad- versarial attacks can be categorized into white-box (Naseer et al., 2020) and black-box attacks (Lin)
et al. 2024) scenarios. In white-box scenarios, the attacker can access the model's 's internal structure and parameters, while in black-box scenarios, only the model's inputs and outputs are observable. With the increasing of pre-trained encoders, recent research (Ban & Dong 2022 Zhou et al.) 2023 Zhang et al.) 2022) has begun to explore attacks targeting these encoders. These attacks are downstream-agneet affecting any downstream model that uses the pre-trained encoder, thus high- lighting the security risks inherent the pre-trained paradigm.
Existing adversarial defense methods mainly fall into two categories: input filtering (Guo et al. 2017) and enhancing the model's inherent robustness (Zhu & Guptal 2017] Zhou et al. (024)) the latter of which often employs adversarial training. While adversarial training strengthen

---

Published as a conference paper at ICLR 2026
model's robustness, it may also affect its generalization ability, leading decreased performance on clean samples. Recent research (Lamb et al. 2019; Zhou et al. 2024 Liu et al, 2021 Li et al. 2024) has focused balancing the enhancement of model robustness with the preservation of gen- eralization capabilities. However, existing methods have yet to achieve the enhancement of model robustness without compromising, even while improving, its generalization ability.
B PRELIMINARIES
B.1
THREAT MODEL
This paper primarily evaluates the security and vulnerability of pre-trained encoders in the context of downstream classification tasks. To begin our analysis, we first define the attacker's goals and the defender's s capabilities.
Building on previous works Ban Dong (022)) Zhou et al. (2023 2024). we assume that attackers are likely to exploit publicly available pre-tra tre-trained encoders, which can be easily accessed through purchase or directly downloaded from open platforms. The primary goal of these attackers is to create adversarial examples targeting downstream models that utilize these pre-trained encoders, with the intention of launching non-targeted attacks. Due to their limited knowledge of downstream tasks, these attacks exhibit the following key characteristics:
Attacker's Goals:
Universality. Adversarial examples can successfully attack any downstream model using the pre-trained encoder, regardless the specific downstream task.
Effectiveness. These adversarial examples can compromise the performance of down- stream models, even after fine-tuning and the application of defense techniques.
Stealthiness. The adversarial perturbations added to normal examples must remain imper- ceptible humans, ensuring they go undetected.
Defender's Capabilities: Defenders do not control the pre-training procedure and lack direct knowledge of it, including the self-supervised learning method and the dataset used. However, they have full access publicly available pre-trained encoders and complete control over the fine-tuning process. This includes the ability to modify the structure of the downstream model as needed.
B.2 DEFENDER'S S GOAL.
Our method aims to leverage the powerful feature representations of pre- trained encoders to enhance downstream task performance. Specifically, aim to accomplish the following objectives:
Zero-Sacrifice Capability. This goal ensures that models built on pre-trained encoders maintain high classification accuracy on benign samples without compromising perfor- mance. Ideally, also seeks to improve performance benign data.
Adversarial Robustness. This goal enables downstream models to effectively mitigate adversarial examples, particularly those generated by pre-trained encoders. enhances the model's robustness, ensuring correct predictions even when the input is adversarially perturbed.
This goal ensures that the defense method remains resource-efficient requiring no additional external data or computational resources beyond those available for the downstream task, making practical and scalable for real-world applications.
Resource Efficiency:
Persistent-Robuttere Capability. This goal ensures that our method provides robust performance across wide range of downstream tasks with single tuning. It enables persistent-rohus robustness adaptability, allowing the model to perform effectively as new tasks emerge, without requiring frequent retraining task- tuning.
B.3 CHALLENGES
In this paper, we propose challenging but crucial goals. Unlike previous works, we aim to en- hance adversarial robustness without sacrificing benign performance, and even expect improve it.
2

---

Published as a conference paper at ICLR 2026
Furthermore, aim to establish persistent-robustness paradigm that enables one-time tuning process while ensuring robustness across all downstream tasks.
We aim leverage the inherent characteristics of neural networks to develop a novel defense paradigm. However, re-tuning the pre-trained encoder is not an option, as any inappropriate al- terations may compromise its powerful feature extraction capabilities. These constraints make goals particularly challenging, and the main challenges are outlined below:
Challenge I: The Challenge of Zero-Sacrifice. Our approach significantly differs from previous methods, particularly in how it addresses the trade-off between adversarial robustness and benign performance. Traditional methods aim minimize performance loss benign samples to avoid drastic decline in accuracy. However, these methods are inherently limited by the adversarial fine- tuning paradigm, where fine-tuning is performed based on adversarial examples after pre-training the encoder. This creates a confict between optimizing adversarial robustness and maintaining high performance on clean data. Beyond this, this paper proposes more challenging goal: achieving the zero-sacrifice characteristic, where both adversarial robustness and benign performance are opti- mized simultaneously. This more challenging because defenders have no access to the pre-training process. Furthermore, the optimization goals of benign pre-training and adversarial fine-tuning are inherently conflict, making difficult to optimize both adversarial robustness and benign perfor- mance simultancously.
Challenge II: The Challenge of -atiisttet-Roouseesss Capability. In addition to setting a higher goal for benign sample performance, introduce the novel capability of Persistent-Robustner Capability". which has not been addressed in previous approaches. As mentioned earlier, previous methods require task-specific tuning for each downstream task. This introduces significant overhead, as the network must be retrained whenever the downstream task changes, increasing training costs and computational demands. To address this, we introduce the concept of persistent-robustness capability, which ensures that the model can adapt to new tasks without frequent retraining, This a challenging goal because it requires the defense method to be both upstream- and downstream- agnostic. In other words, the method must adapt to various pre-training procedures and downstream tasks without relying on task specific or mooee--peccffc assumptions, adding significant complexity to its design. Although this goal is crucial, it has often been overlooked in previous works. Achieving will not only allow the defense method to support a broader range applications but also provide valuable insights for developing more explainable and transparent defense mechanisms.
C THEORETICAL PROOF
The key rationale of ZePAD is that neural networks naturally assign higher confidence to inputs that follow their training data distribution, and lower confidence to inputs that lie outside that distribution. This behavior is not manually designed, but emerges naturally from empirical risk minimization using cross-entropy. To support this statement, we present theoretical proof as follows:
C.I TRAINING DISTRIBUTIONS OF TWo BRANCHES
Training Distributions of Two Branches Let Q( y) denote the benign data distribution, where x the input image and y E {1,..., K} is the label. Then, Qs(x, v) represents the adversarial data distribution. Each sample has the form r + 8 with |öl|p S E, where is the adversarial perturbation vector crafted by an attacker
Each branch is optimized on its own empirical training distribution, denoted as Ps and Pa. respec- tively. In particular, the MPAE branch is trained on a mixture of benign and adversarial examples. Hence, Po and Pa can be formulated as:
PL(x, v) Qux,
(S1)
Pa v) = (1 a:) Q(x (x,,w) v) + a Qs( a a (0,1),
(S2)
where denotes the mixing coefficient that controls the proportion of adversarial examples used during training.
Formally, P. serves the empirical training distribution for the BMP branch, while P. represents the mixed training distribution for the MPAE branch, containing both benign and adversarial exam- ples.
3

---

Published as a conference paper at ICLR 2026
C.2
TRAINING AND POSTERIOR APPROXIMATION
Consider a classifier (() trained using the cross-entropy loss:
L(f = B(,,)>pp- log (x))
(S3)
where fy(x: denotes the predicted probability for class y.
Under sufficient model capacity and training convergence, the classifier trained on distribution P(x, 3) approximates the true posterior Zhang (2004), which can be formulated as f*)) s
P(y 4t), where f refers to the theoretically optimal classifier. For our two branches, we have f((x) R (u(y(x) and fax) R Pa (v)rr,, where fo(x)) and La (ar denote the output probability vec- tors the BMP branch and the MPAE branch, respectively. Then, we define the model confidence using the maximum softmax probability:
m=(x)) max (2) R max Po = k x),
(S4)
ma (2r) max Ja, (x) max P (y x)),
(S5)
where (x) (or (a))(z)) represents the predicted probability that input x belongs to class k. my and m. represent the confidence scores of the BMP and MPAE branches, respectively. Intuitively, mz(x)) (z) and m (z) measure how well an input aligns with the data distributions P. and Pa, respec- tively.
C.3 MODEL SENSITIVITY
Then, for benign samples Qzz, y), BMP's posterior is Po(ylx) Q(ylz), while MPAE's posterior is diluted by the adversarial mixture Pa (b(x) = (1 - ())((=r)) + aQa(vlz). Obviously, for x Q(x,y), Quylx) > Qs(ylæ). Hence, we have:
(S6) As proved above, we can obtain that for benign samples, the confidence of the BMP branch is higher than that under the confidence of the MPAE branch. Conversely, for adversarial inputs, the MPAE branch yields a higher confidence than the BMP branch. This confirms a clear confidence separation between the two branches, demonstrating the inherent sensitivity of neural networks different data distributions. Specifically, network tends to assign higher confidence to samples that align with its training distribution, while yielding lower confidence on unseen or -uittf-distiibution samples.
Pa(l = (1 - Q0)la))(==>-----00) + (Qs(ll)) < Q(12z)= = Po(y(z).
D THE MAIN STEPS OF ZEPAD
To help readers follow ZePAD, we illustrate the main steps in Algorithm I In brief, ZePAD first pre- pares three encoders two adversarially fine-tuned (MPAE-Branch) and one trained only on clean data (BMP-Branch)-aloog with their classification heads. During inference, it computes confi- dence scores for each branch and fuses their predictions via robust federal decision mechanism, enabling persistent-robusttre defense against DAEs without sacrificing clean accuracy
E EXPERIMENTAL SETTING
During adversarial training, set the value of À to 20 by default. The model is trained for 20 epochs with a learning rate of 0.0001 and a batch size of 256, using the Adam optimizer. For downstream model training, we employ a simple three-layer classification head, with learning rate of 0.005, batch size of 256, and 20 training epochs. For the AdvEncoder (Zhou et al., (023) attack method, adopt the default parameter settings. The perturbation budget is set 10/ 255, and the batch size is fixed at 256. During training, the hyperparameter a is set to its default value of 5, and we use a learning rate of 0.0002 with the Adam optimizer and 20 training epochs. To help readers follow manuscript, our codes will be made publicly available after the review process.
Details be found in codes, which would be relcased after the review process.
4

---

Published as a conference paper at ICLR 2026
Algorithm The main steps of the proposed ZePAD
Input: training dataset(x; y) E Db. pre-trained encoders ε,, and E, loss weight A, a testing dataset (Te, M/.) E D, adversarial noise generator G model 1: STEP 1: Encoder Preparation and Downstream Training
Output: a robust and persistent-robustner
2: &g f pre-train with D3 with SSL method DD, do
3: for each
4: x² = = (())
5: Dodu = Dado U {x*}
6: end for
7: 8,1 + fine-tune 1 with D Ddde and loss weight À
8: E, - fine-tune Ea with D Dadu and loss weight
9: Downstream Training
Ha,1 t- train with frozen E,1 and Db
Ha,2 f- train with frozen E22 and D
Ho 4- train with frozen E and D,
13: STEP 2: Robust Federal Decision
for each I4 E DD.
15: Va = =Ha,1
16:
Na = =Ha2(E2,221a))
17:
yb. Ho(Es((tt)
18:
Wa.1 Wa,2, W/ x- Based Eq. (7)
19: Upredict = Wa,18a,1 + Wa,28a,2 + Way
20: end for
F EXPERIMENTS UNDER THE WHITE-BOX SCENARIO
In the white-box scenario, compared to the semi-black-bor scenario, the attacker has access not only the victim encoder but also to an additional open-source encoder chosen by the defender. this scenario, the attacker can access all models used in the HIPAE-rrande well as the pre- pre-trained datasets. Therefore, compared with the previous setting, the white-box scenario poses greater challenge to the defense robustness of our proposed method.
Specifically, we conduct 60 experimental settings, covering multiple self-supervised method combi- nations and different datasets. We use CIFAR10 the pre-trained dataset and selected AdvEncoder as the attack method to evaluate ZePAD's defensive performance in white-box scenarios. The ex- perimental results are provided in Tables ST and S2
From the experimental results, it can be seen that ZePAD maintains high RA even in white-box sce- marios. For instance, when the downstream datasets are SVHN and ANIMALS10, the RA values 89.10% and 86.80% respectively, showing improvements of 74.66% and 45.31 % over the baseline. This indicates that with knowledge of the upstream pre-trained dataset and the pre-trained mod- els used in the MPAD-Branch, the attacker cannot achieve effective results, demonstrating ZePAD's high robustness.
To further evaluate ZePAD's defensive performance, we tested PAP (Ban & Dong 2022
UAP MMoosavi-Dezoooli et al., 2017). UAPGD (Deng Karam 2020 and SSP (Naseer et al. 2020) attacks in white-bon box scenarios, with results shown in Figure ST PAP was the most stable and effective. reducing baseline RA nearly 20%. UAP performed poorly, with baseline RA near- ing 80% in some cases, such as SwAV combined with MoCo v2+. However, 7ePAD maintained consistently high RA across all attacks white-box scenarios, indicating strong defense against existing DAEs.
Notably, we would like to emphasize that, due to the inherent uncertainty in user behavior and the vast array of available pre-trained encoders, it extremely challenging for the attacker to accurately determine which specific two encoders are being used in practice. is therefore extremely difficult for the attacker to perform precise attack, which aligns with our assumed white-box scenario. As a result, the white-box setting represents the most challenging case for downstream-agnostt adversarial attacks.
5

---

Published as a conference paper at ICLR 2026
dipico Dega
ding Seete
moroy Jagrom
sydt mooo
matid pepson
doo
tapt
PAP
924
amooe Apa
medt Mirdg
moror moco
Spon
ende stong
monora mend
CAp
(d) UAPGD
Figure S1: BA (%) and RA (%) of our ZePAD and the baseline method under four attack methods the white-box scenario.
Table S1: BA(%) comparison between the baseline method and ZePAD in the white-box scenario, where Dy and Da denote the adversarial training and downstream datasets, respectively.
D and D2
Method
W:MSE
BYOL.
NNCLK
VihCreg
MoCo3
SupCon RESSL.
SwAV
VibCreg
W-MSE
NNCLR
McCo2t
SwAV
SupCon
RESSL
DINO
baseline
94.24
94.59
95.66
96.23
93.51
93.30
95.01
94.99
96.82
93.72
94.81
CIFARIO
ours
-0.88
-1.30
-0.80
.6.25
.0.52
.0.97
-1.35
-4.33
-1.41
-1.94
baseline
$1.26
83.20
86.63
87.76
B3.05
83.34
B3.85
35,13
87.98
B3.56
35.70
ours
+1.15
+0.43
-0.7%
-2.68
-1.24
+1.97
-0.42
-0.86
-2.10
-1.90
-0.64
baseline
89.61
95.47
90.78
91.39
90.39
93.68
93.02
91.48
92.04
89.70
91.82
ANIMALSI0
ours
97.34
46.35
+1 80
+6.68
+5.41
+5.54
+3.22
+3.33
+5.86
+4.36
+7.54
+5.01
baseline
79.47
85.56
83.94
84,45
$6.35
82.96
82.58
86.05
36.57
K2.51
84,19
ours
93.78
96.16
93.94
A +15.4
+8. 66
+11.17
+9.52
+3.94
+12.28
+11.20
+10.11
-7.77
+11.33
+10.5
bascline
09.05
38
ImagoNet20
ours
69.25
70.34
71.13
63.10
64.88
69.84
68.75
68.06
65.08
67.91
A
+1.59
+1.39
+1.49
-3.77
-5.06
+3.97
-1.79
-0.40
+1.49
-3.67
-0.48
haseline
75.11
74.87
SVHN
ours A +28.66
94.23
94.14
95.67
94.50
94.05
92.59
95.09
95.48
94,28
94.71
94,44
+17.96
+25.03
+18.92
+16.78
+17.92
+25.20
+20.37
+19.40
+19.88
+21.01
G OTHER ABLATION STUDIES
G.1 IMPACT OF NUMBER OF ADVAU-MODELS
ZePAD is a multi-encoder architecture that increases di- versification the encoder level. Previous experiments have validated the effectiveness of our method. A natural idea is to introduce more multiple models to further boost performance.
RAUNO
To explore the impact of the number of AdvAu mod- els on the overall model, we conducted experiments us- ing encoders pre-trained on CIFAR10, 10, with STL10 as the downstream dataset. We chose W-MSE as the victim en- coder and considered SimCLR, BYOL, DINO, MoCo v3,
Numbers of Addutl-MMddle Models
Figure S2: BA (%) and RA across different numbers of Addell.dels.

---

Published as a conference paper at ICLR 2026
Table S2: RA (%) comparison between the baseline method and ZePAD in the white-box scenario,
where Dy and Da denote the adversarial training and downstream datasets, respectively.
Dj and D2
Mechod
WMSE BYOL.
NNCLR
VibCreg
McCol+
MoCo3
SupCon RESSI.
SwAV
W-MSE VibCreg
NNCLR
MoCo21
SwAV
SupCon
RESSL
DINO
AVG
baseline
14.67
10.77
13.32
12.93
12.74
18.17
10.08
11.60
26.97
10.55
14,18
CIFARIO
ours
79.73
76.72
79,90
77.25
82.63
A +66.60
+68.96
+63.40
+68. 10
+67.31
+63.35
+70.27
+68. 30
+50.28
+72.05
+65.86
haseline
49.95
58.25
23.00
45.08
$4.51
19.29
24.45
59.07
STL
ours
75.04
76.33
79.12
76.33
69.21
76.61
75.85
71,90
78.35
77.65
75.64
A +25.09
+38.07
+23.26
+39.74
+46.21
+51.53
+41 34
+52.61
+18.67
+49. 20
+36.57
haseline
56.06
50.99
43.53
54.27
11.37
52.70
39.97
13.68
61.46
30.88
41,49
ANIMALS10
ourS
90.88
87,81
90.18
91,28
87.97
88.42
92.14
66.27
86.05
87.00
86,80
A
+34,82
+36.82
+46.65
+37.01
+76.60
+35,72
+52.17
+52.59
+24.59
+56.12
+45.31
10.84
16.76
GTSRB
ours
73.45
75,84
77.27
70.60
94.08
76.99
84,69
93.36
78.05
82.99
80.73
A $55.89
665 00
+60.00
+53.84
180.23
160.74
+75.20
+83.85
58.78
+77.26
+67.08
baseline
28.08
21.23
6.15
20.73
ImageNet20
ours
61.21
$1,79
61,90
55.46
39.68
19.70
62.20
30.75
53.87
56.94
$2.35
A 221.9
+23.71
+37.79
+27.38
+31.45
117.85
140.97
+4460
+13.39
+36.21
+27.73
baseline
9.22
19.52
SVHN
OuPS
85.68
91.45
90.78
88.17
91.78
82.64
90.43
90.16
89.66
90.27
89.10
A
+66.
+83.23
+71.16
+77.92
+77.48
+72.43
473.26
+71.25
+73.99
+80.82
+74.66
Table S3: Performance under different ensemble methods.
Confidence
Metric
W-MSE
BYOL
NNCLR
SimCLR
MoCo2+
AVG
BA (%)
93.43
93.61
93.91
93.95
93.57
93.69
Average
RA (%)
85.46
79.85
77.58
71.93
64.38
75.84
BA (%)
90.97
90.32
92.07
91.60
80.36
89.06
Entropy
RA (%)
23.13
24.06
24.39
26.46
30.98
25.80
BA (%)
93.37
93.49
93.69
93.70
93.57
93.56
RFDM
RA (%)
87.14
81.70
79.66
74.30
65.91
77.74
MoCo v2+, and NNCLR as potential adversarial auxiliary encoders. The results are shown in Fig- where observe that increasing the number does not necessarily enhance performance. Additionally, considering the extra resource overhead introduced by incorporating multiple models, empirically recommend setting the number of Ady Au models the default choice.
S2
G.2 IMPACT OF DOWNSTREAM EPOCHS.
We study the effect of the number of epochs on the downstream model, with results shown in Figure(S3 Both BA and RA remain largely unaffected by the number of epochs, which sug- gests that the downstream model con- verges quickly. This is due to the model's ' small size and the strong fea- ture extraction ability of the upstream encoder. It also demonstrates that the generalization and robustness of the entire model stem primarily from the upstream model.
Figure S3: Performance of downstream models with differ- ent epochs. (a)-(b) denote BA (%) and RA (%) with differ- ent epochs, respectively.
G.3 IMPACT OF THE DIFFERENT ENSEMBLES
We compare RFDM with other ensemble strategies, including average ensemble and entropy en- semble methods. our experimental setup, the pre-training dataset for the encoder, the adversarial training dataset, and the downstream task dataset are all selected from CIFAR10. The experimental results are reported in Table SS3 As shown in the results, entropy weighting performs the worst, par- ticularly on RA. The performance of RFDM on BA is nearly identical to that of average ensemble. This is because, as illustrated Figure 2 the confidence scores the three branches for benign samples are almost the same, in which case RFDM is approximately equivalent to average ensem- ble method. In contrast, RFDM achieves slightly higher RA which is attributed to the pronounced confidence differences between the MAPE- MAPE-Branch and EME-Brande on adversarial examples. Such discrepancies allow RFDM to more effectively leverage the robustness of the MAPE- MAPE-Branch when encountering adversarial examples.

---

Published as a conference paper at ICLR 2026
Table S4:
RA (%) under different perturbation budgets.
6
BYOL
ReSSI.
SupCon
SimCLR
NNCLR
AVG
10/255
80.18
84.58
77.99
74.70
79.84
79.46
12/255
74.29
72.41
71.83
61.80
70.72
70.21
14/255
57.67
66.41
68.22
59.32
67.46
63.82
16/255
53,49
66.53
61.17
54,45
53.11
57.75
Table S5: Ablation study on the effect of A.
W-MSE
SupCon
SimCLR
NNCLR
BYOL
BA(%)
RA(%)
BA(%)
RA(%)
BA(%)
RA(%)
BA(%)
RA(%)
BA(%)
RA(%)
93.02
45.19
93.04
49.63
92.74
41.43
92.70
55.00
92.98
41.29
10
93.37
87.15
92.46
76.84
93.7
74.30
93.53
79.53
93.18
81.70
15
93.59
87.59
92.59
76.77
94.53
74.21
93.78
79.47
93.47
81.12
20
93.42
86.84
92.39
77.99
95.05
74.70
93.96
79.84
93,49
80.18
93.21
87.62
92.46
77.37
94.66
74.63
93.69
79.03
93.59
81.36
G.4
IMPACT OF THE PERTURBATION BUDGETS
In the previous experiments, the perturbation budget used by the attacker was set to 10/255 by default. To further investigate the defensive capability of ZePAD, evaluate its performance under larger perturbation budgets. In this experimental setup, the attacker's source dataset, the encoder's pre-training dataset, the adversarial training dataset, and the downstream task dataset all selected from CIFARIO. The experimental results are presented in Table S4 As shown the results, RA gradually decreases the perturbation budget increases. However, even when the perturbation budget is enlarged to 60% of the original value, RA remains above 50%, demonstrating the strong defensive performance of ZePAD.
G.5 IMPACT OF THE PARAMETER À
In this section, we provide an analysis of the hyperparameter X. the previous experiments, was set 20 by default. We investigate the performance of 7ePAD as À varies from 25. The pre- training dataset, adversarial training dataset, and downstream dataset are all selected from CIFAR10. The experimental results are shown Table From these results, we observe that when À is small, the RA remains low, indicating that the encoder is insufficiently adversarially tuned. As À increases, the model progressively balances BA and RA, achieving a better trade-off. We therefore adopt 20 as the default setting, as it provides stable performance across different settings.
H EXTRACTED PATTERNS OF DIFFERENT SSL METHODS
As discussed previously, we find that pre-trained models based on different SSL methods may have distinct and complementary feature extrac- tion patterns, with different SSL paradigms fo- cusing on and memorizing different patterns. The ablation study shows that simply combining three encoders without extra adversarial training can effectively enhance both BA and RA, which supports basic assumption that existing DAE generation methods struggle to identify common vulnerable patterns learned across different SSL methods.
BYOL
(b) MoCo X
To better understand these patterns, used Grad-CAM to visualize encoders from differ- self-supervised methods, including BYOL,
SupCon
W-MSE
Figure S4: The Grad-CAM visualizations of dif- ferent SSL methods.
8

---

Published as a conference paper at ICLR 2026
MoCo v2+, SupCon, and W-MSE, all pre-trate rre-traieed CIFARIO. The visualizations, shown in Fig- ure S4 reveal that different encoders focus on different features, indicating their diverse extrac- tion patterns and ability to handle varied samples. This diversity allows RFDM to leverage their strengths, thereby improving overall model performance.
I RESOURCE OCCUPANCY AND CONSUMPTION
In our proposed method, additional encoders are incorporated. It is expected that this would lead to increased resource consumption and occupancy, including inference time and GPU memory. Given that our method only utilizes multiple encoders during the inference phase, conducted evaluations specifically in this stage. The experiments were conducted on CIFAR1O using an NVIDIA RTX 4090 GPU. We performed inference the test sets and measured the average time overhead and GPU memory usage per batch, with a batch size of 256.
The results presented in Table S6 Our method introduces additional com- putational overhead; however, the extra inference time is only 0.00007 seconds per image, which represents approxi- mately a 30% increase compared to the baselines. We believe this overhead acceptable given the substantial performance improvements. Considering the substantial improvements in both BA and RA (over 30% and 80% respectively in some cases), we believe that the additional computational cost justified and acceptable.
Table S6: The Computational Resource Comparison.
Method Baseline
Time (s)
Memory (MB)
0.0542
67.04
ZePAD
0.0714
156.50
While our method introduces multiple branches and therefore incurs higher training cost per run, only requires single adversarial training process for different downstream tasks. In contrast, ex- isting methods typically require separate adversarial training for each downstream dataset. Hence, in practical scenarios involving multiple downstream tasks, our approach can be more efficient over- all. In particular, when the number of downstream tasks exceeds the number of branches, the total training cost of our method becomes lower than that of task-specific adversarial training.
In our future work, will focus reducing the computational overhead while further preserving both robustness and benign accuracy.
J EXTENSION TO OTHER DOWNSTREAM TASKS
To further evaluate the performance of ZePAD on other downstream tasks, we design a retrieval experiment. The encoder pre-trained on CIFAR1O, while both the adversarial training dataset and the retrieval dataset are chosen from SVHN. In this experiment, the training set is used as the gallery and the test set used as the query set. Table reports the Top- 10 accuracy of this task. From the results, observe that ZePAD achieves substantial improvements in both BA and RA compared to baseline, indicating that ZePAD retains its zero-sacrifice property on alternative downstream tasks.
Table S7: ZePAD's Performance on the Retrival Task.
Motre
method
W-MSE
BYOL
NNCLR
SinCLR
MoCo2+
MoCo3
RcSSL
DINO
SWAV
VibCCrg
SupCon
AVG
hascline
62.25
60.10
70.22
02.89
56.42
69.73
75.47
39.74
64.26
(%)
ZePAD
90.49
86.74
84.53
88.79
89.00
90.20
78.97
80.37
81.00
95.26
88.85
86.75
RA (%)
bascline
3064
58.41
51.25
39.37
67.42
64.80
57.38
67.17
61.12
64.70
53.13
57.76
While extending it to tasks such as detection or segmentation is theoretically possible, these tasks follow fundamentally different architectural pipelines (e.g., U-Net variants) and heavily rely skip connections or task-specific modules. Directly replacing their encoders with standalone pre-trained encoder would likely lead to suboptimal performance, and thus this direction is beyond the current scope of this paper. We appreciate this insightful suggestion and intend to explore the extension of method segmentation models, such as SAM, part of future research.

---

Published as a conference paper at ICLR 2026
Table S8: BA (%) comparison of Baseline and ZePAD with CLIP pre-trained encoder.
Dataset
Method
W-MSE
SimCLR
NNCLR
MoCo2+
MoCo3
AVG
Baseline
90.17
93.56
93.77
95.03
94.71
93.45
CIFARIO
ZePAD
93.95
92.63
94.31
95.22
94.23
94.07
Baseline
78.49
81.94
83.09
84.46
84.00
82.40
STL10
ZePAD
95.07
96.64
97.33
97.36
97.68
96.82
Baseline
80.39
90.82
91.22
88.03
88.05
87.70
ANIMALS10
ZePAD)
99.33
98.24
99.17
98.89
99.45
99.02
Table S9: RA (%) comparison of Baseline and ZePAD with CLIP pre-trained encoder.
Dataset
Method
W-MSE
SimCLR
NNCLR
MoCo2+
MoCo3
AVG
Baseline
10.09
47.13
10.71
45.01
36.86
29.96
CIFARIO
ZePAD
79.99
81.24
82.35
79.34
78.66
80.32
Baseline
41.16
67.52
44.05
65.55
64.28
56.51
STLI0
ZcPAD
92.65
91.73
92.78
91.04
92.33
92.11
Baseline
48.31
63.48
49.23
63.03
60.32
56.87
ANIMALSI0
ZePAD
97.26
96.63
97.28
95.34
97.33
96.77
K
EXPERIMENTS ON MULTIMODAL PRE-TRAINED ENCODERS
To investigate the performance of ZePAD on multimodal and larger models, we conduct experi- based CLIP. We use CLIP as the MAPE-Branch protect different pre-trained encoders, including W-MSE, SimCLR, NNCLR, MoCo v2+, and MoCo v3. Both CLIP variants, ResNet- 101- based and ViT-B/32- are adversarial fine-tuned on CIFARIO with a batch size of 64, and the hyperparameter À and learning rate are set to 20 and 0.00001, respectively. The experimental results are presented Tables and S9
respectively.
Even with CLIP, large-scale multimodal pre-trained encoder, our method preserves benign accu- racy while achieving substantial improvement in adversarial robustness. These results demonstrate the generalizability of our approach to multimodal pre-trained encoders. The main reason is that the fundamental threat model of downstream-agnoose single-mi modallty and multimodality encoders, and the inherent sensitivity leveraged by defense strategy does not depend on the modality used in pre-training but rather on the distribution alignment between samples and the encoder's training data.
adversarial attacks remains consistent across both
10


Reviews:
[{"review_id": "CfWkDLEdSJ", "reviewer": "Reviewer_vmRw", "summary": "This paper introduces **ZeLAD (Zero-Sacrifice Lifelong Adversarial Defense)**, a framework aimed at defending pre-trained self-supervised encoders against **Downstream-Agnostic Adversarial Examples (DAEs)**. Unlike prior task-specific fine-tuning approaches, ZeLAD claims to provide a single, lifelong adversarial defense applicable across diverse downstream tasks while maintaining or improving benign accuracy.\n\nZeLAD employs a dual-branch architecture:\n- MPAE-Branch (Multi-Pattern Adversarial Enhancement Branch): Combines multiple adversarially fine-tuned pre-trained encoders trained with diverse SSL methods to enhance robustness through representational diversity.\n- BMP-Branch (Benign Memory Preservation Branch): Trained solely on clean data to preserve benign performance.\n\nDuring inference, a Robust Federal Decision Mechanism (RFDM) fuses branch outputs by comparing confidence scores. The model also demonstrates the ability to detect adversarial samples based purely on branch confidence disparities, without explicit adversarial detection training.", "questions": "**1. Clarification on the “Lifelong” Claim**\nYou define ZeLAD as a *“lifelong adversarial defense”* requiring only a single tuning. However, lifelong learning typically implies *continuous adaptation to new tasks* without full retraining.  \n**Could you clarify how ZeLAD satisfies the lifelong learning property beyond single multi-task applicability?**  \nHave you tested whether ZeLAD can handle incremental task addition or domain shifts without catastrophic forgetting?\n\n **2. Justification for the “Zero-Sacrifice” Property**\nThe paper claims that ZeLAD achieves robustness improvements *without sacrificing benign accuracy*.  \n**What theoretical or empirical evidence supports this “zero-sacrifice” claim?**  \nDoes this property hold under stronger perturbation budgets or adaptive attacks that target both branches simultaneously?\n\n **3. Robustness of the Confidence-Based Fusion (RFDM)**\nThe Robust Federal Decision Mechanism (RFDM) fuses branch outputs using confidence-based weighting.  \n**How reliable is this mechanism under confidence miscalibration, label noise, or adaptive attacks explicitly designed to manipulate confidence distributions?**  \nHave you compared this heuristic to alternative fusion schemes (e.g., entropy weighting, temperature scaling)?\n\n **4. Computational and Practical Efficiency**\nZeLAD employs multiple encoders (two adversarially fine-tuned and one benign), which could increase inference cost.  \n**What is the computational and memory overhead of ZeLAD compared to single-encoder fine-tuning methods?**  \nCan you comment on its scalability to larger SSL encoders (e.g., ViT-L/16, CLIP) in real-world applications?\n\n~PS: Any chance you want to name the paper ZeLDA: Zero sacrifice Lifelong Defence against Adversaries?~", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Novelty and Conceptual Contribution:**\n- The paper introduces the idea of “zero-sacrifice lifelong adversarial defense”, reframing adversarial robustness as a feature combination problem rather than a tradeoff problem.\n- The dual-branch design (MPAE + BMP) and the federated confidence fusion mechanism are novel and well-motivated.\n2. **Comprehensive Empirical Evaluation:**\n- Extensive experiments across multiple SSL encoders (e.g., SimCLR, BYOL, MoCo, DINO) and datasets (CIFAR10, ImageNet, STL10, etc.) demonstrate the generality of the method.\n- Results consistently show significant improvements over baselines and other defenses (e.g., TRADES, MART, Gen-AF).\n3. **Strong Practical Relevance:**\n- Addresses a genuine and underexplored problem: the vulnerability of pre-trained encoders to DAEs in a downstream-agnostic setting.\n- The claim of “single tuning for all downstream tasks” has significant implications for scalable and resource-efficient deployment.\n4. **Adversarial Detection Without Supervision:**\n- The method’s ability to detect adversarial samples using confidence asymmetry without explicit training is an elegant byproduct.\n5. **Clarity and Organization:**\n- The paper is generally well-written and logically structured, with helpful figures (e.g., the overall ZeLAD architecture diagram and confidence distributions).", "weaknesses": "1. **Methodological Clarity and Rigor:**\n- While conceptually interesting, some mathematical formulations (e.g., hybrid loss and cosine distance adjustment) lack detailed derivations and theoretical justification.\n- The Robust Federal Decision Mechanism (RFDM) is empirically defined, but its weighting function (Eq. 8) seems heuristic and not theoretically grounded.\n- There is no formal analysis of why confidence alignment is a robust signal or how it generalizes across tasks.\n\n2. **Evaluation Limitations:**\n- Most experiments focus on classification tasks; it’s unclear whether ZeLAD extends to non-classification downstream tasks (e.g., segmentation, detection).\n- The adversarial attack diversity could be further improved—results rely heavily on AdvEncoder; other recent black-box attacks are not deeply explored.\n\n3. **Scope of “Lifelong” Claim:**\n- The “lifelong” aspect mainly refers to single fine-tuning across multiple tasks rather than continuous adaptation. There is no evidence of incremental task adaptation or continual learning capability, making the “lifelong” terminology somewhat overstated.\n\n4. **Computational Cost and Practicality:**\n- Maintaining multiple encoders (two adversarially fine-tuned and one benign) increases inference-time complexity and memory usage, which could hinder scalability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QRjR1jdcgf", "reviewer": "Reviewer_2cmk", "summary": "This paper introduces ZeLAD, a lifelong defense framework that protects self-supervised encoders from downstream-agnostic adversarial examples (DAEs) with a single tuning. ZeLAD employs dual branches to enhance adversarial robustness while preserving benign performance, and can even detect DAEs via branch confidence without explicit training.", "questions": "Given that the proposed method is claimed to build on a general property of deep neural networks — “neural networks inherently exhibit higher confidence in inputs that resemble the training data, a behavior attributed to the memorization of the data’s characteristics” — two questions arise.\n\n1. Whether the method can be applied to multimodal pre-trained models such as CLIP or BLIP remains unclear. Experiments are needed to verify its scalability.\n\n2. It is also unclear whether the method can generalize beyond image classification, for example to image retrieval, semantic segmentation, or object detection.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1、The paper proposes a zero-sacrifice, lifelong adversarial defense method that not only maintainsbut also improves benign performance, while enhancing adversarial robustness.\n\n2、Extensive experimental results demonstrate the effectiveness of the proposed method.\n\n3、The paper is easy to follow.", "weaknesses": "1、The paper claims to build on the inherent sensitivity of neural networks to data characteristics, yet this idea is only briefly mentioned in the introduction (L54–57) without deeper investigation. No experimental validation, theoretical analysis, or concrete insight is provided to support this claim, which substantially undermines the rationale and validity of the proposed method. A more thorough analysis through exploratory experiments or theoretical justification is necessary.\n\n2、The first and second claimed contributions both emphasize lifelong adversarial defense, which appear conceptually identical. I suggest merging them for conciseness and clarity.\n\n3、The paper inconsistently uses adversarial sample and adversarial example. The terminology should be unified using the widely accepted adversarial example to maintain professional consistency.\n\n4、The Related Work section remains too high-level and lacks discussion of key concepts (e.g., “Pre-trained encoders”， “pre-trained paradigm”) as well as core algorithmic ideas of representative methods—such as self-supervised learning approaches (SimCLR, MoCo) and DAEs on pre-trained encoders (PAP, AdvEncoder).\n\n5、The Threat Model is essential for any security-related study and should appear in the main text rather than the appendix. Similarly, the discussion of challenges addressed by the proposed method would be better placed before the methodology section for improved logical flow.\n\n6、The reported RA results (Tables 2 and 4) show large discrepancies compared to baseline values, raising concerns about correct reproduction of baseline methods. The causes of these gaps should be clarified.\n\n7、The proposed method demonstrates higher time and storage costs than the baselines (Table S3, L899–916).\n\n8、The overall writing quality requires improvement, particularly in table captions, many of which contain grammatical errors. For instance, “Table 1: BA(%) Baseline vs. ZeLAD in the semi-black-box scenario” is ungrammatical and should be revised for correctness and clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hsby1qP35S", "reviewer": "Reviewer_gcvV", "summary": "This paper addresses the vulnerability of self-supervised learning (SSL) models to downstream-agnostic adversarial examples (DAEs). The authors propose Zero-Sacrifice Lifelong Adversarial Defense (ZeLAD), a dual-branch framework designed to achieve adversarial robustness without sacrificing benign performance. ZeLAD integrates a Multi-Pattern Adversarial Enhancement branch and a Benign Memory Preservation branch to balance robustness and benign performance. It detects DAEs by evaluating branch confidence, eliminating the need for adversarial sample training. Extensive experiments on 11 SSL methods and 6 datasets show substantial improvements in both benign accuracy and adversarial resistance, validating its “zero-sacrifice” property.", "questions": "1. The paper introduces a weighting parameter $\\lambda$ in the loss function (Eq. 3), but its value or tuning procedure is not specified. \n2. Some hyperparameters (e.g., lambda, learning rate) are not explicitly stated.\n   Will the authors release code or configuration files to ensure experimental reproducibility?\n3. The paper uses a dual-branch architecture with independent encoders. Have the authors considered some techniques to reduce redundancy while preserving robustness?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces ZeLAD, the first lifelong adversarial defense for pre-trained encoders that achieves robustness across multiple downstream tasks with a single tuning. Unlike prior task-specific adversarial training methods, ZeLAD generalizes effectively across SSL models and datasets, marking a substantial conceptual advancement in adversarial robustness research.\n2. A major strength is ZeLAD’s dual-branch architecture: the Multi-Pattern Adversarial Enhancement (MPAE) branch for robustness and the Benign Memory Preservation (BMP) branch for maintaining clean-sample accuracy. This design enables the model to enhance adversarial defense without degrading benign performance, a key limitation of previous methods.", "weaknesses": "1. Although the paper compares ZeLAD to several classic defenses (e.g., TRADES, MART, Gen-AF), it does not include enough comparisons with the most recent or SSL-specific adversarial defense methods (only Table 7). This omission makes it harder to gauge ZeLAD’s relative progress within the latest research landscape.\n2. The proposed approach requires multiple encoders and dual-branch inference, which could increase computational and memory overhead compared to single-encoder defenses. The paper provides limited discussion or quantitative evaluation of training/inference efficiency. I think it is very important for real world deployment. \n3. Although the paper claims “lifelong” robustness, the experiments only cover a limited number of sequential tasks. There is no long-term continual learning evaluation (e.g., over dozens of tasks or domain shifts), so the claim of lifelong adaptation remains somewhat speculative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u8KakOVEnA", "reviewer": "Reviewer_pqbZ", "summary": "This paper proposes ZeLAD, a “Zero-Sacrifice Lifelong Adversarial Defense” framework that integrates multiple adversarial and benign branches to achieve both clean accuracy and robustness without further tuning. The method designs a Robust Federal Decision Mechanism (RFDM) to adaptively weight branch predictions based on confidence, aiming to provide a single-tuning defense applicable across tasks. Experiments are conducted on multiple datasets and architectures.", "questions": "See Weaknesses. If the authors address my concerns, I am willing to raise the score.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The experiments are relatively comprehensive, covering several datasets, model scales, and attack types.\n2. The paper is logically organized, and most sections follow a clear structure.\n3. The proposed method is systematic and addresses the problem of balancing robustness and generalization in a targeted manner.", "weaknesses": "1. Core claim overstatement: The claim of “lifelong + zero-sacrifice with single tuning” is not well supported. Although the paper emphasizes a one-time tuning process, in practice each new downstream task still requires training local classifiers and possibly branch adaptation. The experiments only test cross-dataset transfer, not true lifelong learning without any retraining, making the central claim over-stated.\n2. Over-packaged novelty: The technical novelty is limited. ZeLAD essentially combines multiple encoder branches with a handcrafted confidence-weighted ensemble (RFDM). The exponential weighting is heuristic, and there is no comparison with simpler ensemble or calibration baselines. The contribution is more about system integration than a fundamentally new defense principle.\n3. Experimental and analysis issues:\n    - It is suggested to include a comparison with a three-encoder average ensemble or PGD-AT baseline to validate RFDM’s effectiveness.\n    - Hyperparameter choices (e.g., weighting coefficients, confidence scaling) and sensitivity analysis are not explained.\n4. Writing and presentation problems:\n    - Figure 2 should be moved to Section 3.2.2.\n    - Eqs. (5) and (6) are unclear in mathematical form and explanation.\n    - Unify terminology: use adversarial example consistently.\n    - The introduction mentions RFDM, but later sections use inconsistent terms.\n    - The PGD-AT citation is incorrect; it should be *“Towards Deep Learning Models Resistant to Adversarial Attacks”*, ICLR 2018, not the CW attack paper.\n5. Mathematical writing issues:\n    - Use calligraphic letters (e.g., $\\mathcal{D}$) for datasets, not $D$; and use $D$ for distance.\n    - Keep notations of encoder and classifier consistent ($E,F$ in Section 3.1 while $\\mathcal{E},\\mathcal{F}$ in Section 3.2).\n    - Avoid blank lines after \\begin{equation}.\n    - Rewrite Eqs. (5) and (6) with explicit meaning of each symbol.\n    - Annotate dimensionality for key notations if possible.\n    - For clarity, it is suggested to include corresponding notation for each component in Figure 1.\n6. Typos and minor language errors:\n    - “generalibility” → “generalizability” (p.1 l.16)\n    - “differ with” → “different from” (p.1 l.18)\n    - “... uses two” → “that uses two” (p.1 l.23)\n    - “task-specfic” → “task-specific” (p.2 l.75)\n    - “taht” → “that”; “branche” → “branches” (p.9 l.446)\n    - Algorithm 1 needs a thorough proofreading.\n7. Related work limitations: The related work section is short and somewhat outdated. Please include more recent adversarial defense papers, such as:\n    - *Probabilistic Margins for Instance Reweighting in Adversarial Training*, NeurIPS 2021.\n    - *DAT: Improving Adversarial Robustness via Generative Amplitude Mix-up in Frequency Domain*, NeurIPS 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)