Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularization
Authors: Xilie Xu
Jingfeng Zhang
Feng Liu
Masashi Sugiyama
Mohan Kankanhalli
Venue: NeurIPS 2023 poster (2023)
Keywords: robust pre-training
adversarial contrastive learning

Abstract: Adversarial contrastive learning (ACL) is a technique that enhances standard contrastive learning (SCL) by incorporating adversarial data to learn a robust representation that can withstand adversarial attacks and common corruptions without requiring costly annotations. To improve transferability, the existing work introduced the standard invariant regularization (SIR) to impose style-independence property to SCL, which can exempt the impact of nuisance style factors in the standard representation. However, it is unclear how the style-independence property benefits ACL-learned robust representations. In this paper, we leverage the technique of causal reasoning to interpret the ACL and propose adversarial invariant regularization (AIR) to enforce independence from style factors. We regulate the ACL using both SIR and AIR to output the robust representation. Theoretically, we show that AIR implicitly encourages the representational distance between different views of natural data and their adversarial variants to be independent of style factors. Empirically, our experimental results show that invariant regularization significantly improves the performance of state-of-the-art ACL methods in terms of both standard generalization and robustness on downstream tasks. To the best of our knowledge, we are the first to apply causal reasoning to interpret ACL and develop AIR for enhancing ACL-learned robust representations. Our source code is at https://github.com/GodXuxilie/Enhancing_ACL_via_AIR.

Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularization
Xilie Xu'r, Jingfeng Zhang Feng Liu', Masashi Sugiyama2 Mohan Kankanhalli
School of Computing. National University of Singapore
2 RIKEN Center for Advanced Intelligence Project (AIP)
School of Computer Science, The University of Auckland
School of Computing and Information Systems, The University of Melbourne
Graduate School of Frontier Sciences, The University of Tokyo
xuxilie@come nus. edu. sg f fengliu. m1 @glmilmlll... sugi @k u-ko.u-oooo.aa .cppo..c.... ac. mohan@comp. nus ..s.edu....p.pe.e
jingfeng. channggananoo
ac nz
Abstract
Adversarial contrastive learning (ACL) is a technique that enhances standard contrastive learning (SCL) by incorporating adversarial data to learn robust repre- sentation that can withstand adversarial attacks and common corruptions without requiring costly annotations. duced the standard invariant regularization (SIR) to impose style- syle-inddependece property to SCL., which can exempt the impact of nuisance style factors the stan- dard representation. However, it is unclear how the inyleiindependence property benefits ACL-learned robust representations. In this paper, we leverage the tech- nique of causal reasoning to interpret the ACL and propose adversarial invariant regularization (AIR) to enforce independence from style factors. We regulate the ACL using both SIR and AIR to output the robust representation. Theoretically, we show that AIR implicitly encourages the representational distance between differ- ent views of natural data and their adversarial variants to be independent of style factors. Empirically, our experimental results show that invariant regularization significantly improves the performance of state-of-the- of-othe-tarrrr art ACL methods in terms both standard generalization and robustness on downstream tasks. To the best of our knowledge, are the first apply causal reasoning to interpret ACL and develop AIR for enhancing ACL-learned robust representations. Our source code is at
improve transferability. the existing work intro-
ACL mia_ARR
1 Introduction
The attention towards pre-trained models that can be easily finetuned for various downstream applications has significantly increased recently [17
Notably, foundation models 4 via
self-supervision large-scale unlabeled data, such as GPT and CIAAP [0], can be adapted to wide range of downstream tasks. Due the high cost of annotating large-scale data, unsupervised learning techniques [21 334 [49 [6] are commonly used to obtain generalizable representations, in which standard contrastive learning (SCL. has been shown as the most effective way [3 10).
Adversarial contrastive learning (ACL) 30 29 that incorporates adversarial data 37 with SCL 9 can yield robust representations that are both generalizable and robust against adversarial attacks 23
"The first two authors have made equal contributions. CCoresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

y1
:
Data generation procedure
yr
yr
y
i
Learning procedure
0
0
Standard contrastive learning
Adversarial contrastive learning
Figure I: Causal graph of data generation procedure in SCL [38] (left panel) and ACL (right panel). 8 is style variable, e is content variable, æ is unlabeled data, ã is the generated adversarial data, and the parameter of representation. The solid arrows are causal, but the dash arrows are not causal. The proxy label n E y is a refinement of the target labels y/ y [w: that represents an unknown downstream task. In the causal view, each augmented view of data point has its own proxy label, where both SCL and ACL aim to minimize the differences between the representation outputs of different views of the same data.
and common corruptions 26 ACL is attracting increasing popularity since the adversarial robustness of the pre-trained models is essential to safety-critical applications (6 33 Many studies have tried to improve ACL from various perspectives including increasing contrastive views and leveraging pseudo labels [22]. leveraging hard negative sampling [2 550, and dynamically scheduling the strength of data augmentations (36).
The style-independence property learned representations, which eliminates the effects of nuisance style factors in SCL., has been shown to improve the transferability of representations (38). To achieve style-independence property in learned representations, Mitrovic et al. [38] proposed standard invariant regularization (SIR) that natural data to be invariant to style factors. SIR has been shown effective in enhancing representation transferability. However, it remains unclear in the literature how to achieve style independence in robust representations learned via ACL,, especially when adversarial data is used during pre-training.
causal reasoning [39] 40] to enforce the representations of
Therefore, we leverage the technique of causal reasoning to enforce robust representations learned ACL to be style-independent. As shown in the right panel of Figure [ we construct the causal graph of ACL (details refer to Section 3.T). Different from the causal graph of SCL [38] ACL has an extra path x > since ACL will generate the adversarial data given the unlabeled data x (i.e., (+ .). Then, ACL learns representations by maximizing both the probability of the proxy label y given the natural data and that given the adversarial data (i.c., x + y and T - y Theorem shows that maximizing the aforementioned probability in the causal view is equivalent to the learning objective of ACL 29 which justifies the rationality of our constructed causal graph of ACL.
To enforce robust representations to be style- hyle-independent, we propose an adversarial invariant regu- larization (AIR). Specifically, AIR (see Eq. (7) aims to penalize the Kullback Leibler divergence between the robust representations of the unlabeled data augmented via two different data augmenta- tion functions during the learning procedure of x - + y Note that SIR (38 a special case of AIR in the standard context where x J.. Then, we propose to learn robust representations by minimizing the adversarial contrastive loss (29 22 50 36) together with a weighted sum of SIR and AIR and show the learning algorithm in Algorithm
Furthermore, we give a theoretical understanding of the proposed AIR and show that AIR is beneficial to the robustness against corruptions on downstream tasks. Based on the decomposition of AIR shown in Lemma 2 we propose Proposition 3 which explains that AIR implicitly enforces the representational distance between original as well augmented views of natural data and their adversarial variants to be independent of the style factors. In addition, we theoretically show that the style-independence property of robust representations learned via ACL will still hold on the downstream classification tasks in Proposition 4 which could be helpful to improve robustness against input perturbations on downstream tasks
3 45 28 38).

---

Empirically, we conducted comprehensive experiments on various datasets including CIFAR- 10 BI]
CIFAR-100 BT, STL-10 [2], CIFAR-IO-C [26]. and CIFAR- 100- [26 to show the effectiveness of
our proposed method in improving ACI. methods 22
22
50
36
We demonstrate that our proposed
method can achieve the new state-of-the-ar (SOTA) performance on various downstream tasks by
significantly enhancing standard generalization and robustness against adversarial attacks [3] 14,12 2
and common corruptions [26]. Notably, compared with the prior SOTA method DynACL [36. AIR improves the standard and robust test accuracy by 2.71% (from 69. 0..0%% to 72.30 10%) and 1.17% (from 46, 46.40.0 to 47.6 47.66==0.069) (.06%) on the STL-I0 10 task and increases the test accuracy under common corruptions by 1.15% (from 65.60 18% % 66. 66.75=0.105 on CIFAR-1O-C.
2 Related Works and Preliminaries
Here, we introduce the related works in ACL and causal reasoning and provide the preliminaries.
2.1
Related Works
Adversarial contrastive learning (ACL). Contrastive learning (CL) is frequently used to leverage
large unlabeled datasets for learning useful representations. Chen al. presented SimCLR that leverages contrastive loss for learning useful representations and achieved significantly improved standard test accuracy on downstream tasks. Recently, adversarial contrastive learning (ACL) 30
29 27 22
52 36 has become the most effective unsupervised approaches to learn robust representations. Jiang et al. [29, which the seminal work of ACI. that incorporates adversarial data 37 with contrastive loss (9) showed that ACL can exhibit better robustness against adversarial
50
attacks 233 13 and common corruptions [26] on downstream tasks compared with SCL. (
One research line focuses on improving the performance of ACL. AdvCL [22] leverages third contrastive view for high-frequency components and pseudo labels generated by clustering to improve ACL. Yu et al. [50] proposed the asymmetric InfoNCE objective (A-InfoNCE) that utilizes the hard negative sampling method 42 to further enhance AdvCL [22]. Recently, DynACL (36] dynamically schedules the strength of data augmentations and correspondingly schedules the weight for standard and adversarial contrastive losses, achieving the SOTA performance among existing ACL methods.
Another research line focuses on improving the efficiency of ACI. since ACI. requires tremendous running time to generate the adversarial training data during pre-training Xu et al. 48 proposed robustness- aware coreset selection (RCS) method to speed up ACI. by decreasing the number of training data. Our paper, belonging to the former research line, leveraged the technique of causal reasoning to further enhance ACL and achieved new SOTA results. We treat the incorporation of our proposed method with RCS
48 as future work.
4 has been widely applied to machine learning to identify causal relations and ignore nuisance factors by intervention. In particular, Zhang al. (53] (56) investigated the causality in supervised adversarial training [37 55 where label information is required and proposed the regularizer to eliminate the difference between the natural and adversarial distributions. Mitrovic al. (38) built the causal graph of SCL. in the standard context and introduced the standard invariant regularization (SIR) which aims to enforce the representation of unlabeled natural data to be invariant of the data augmentations. SIR can significantly improve robustness transferability to downstream tasks in terms of common corruptions 226]. which is empirically validated by Mitrovic al. 38
Causal reasoning. Causal reasoning B9 53 40
56
2.2 Preliminaries
Standard contrastive learning (SCL) 9 Let hg X' + 2 be representation extractor param- eterized by 0, g 2 + V a projection head that maps representations to the space where the contrastive loss is applied, and Tu, X' + X' be two transformation operations randomly sampled from a pre-defined transformation set T. Given a minibatch B ~ X consisting of 3 original samples, we denote the augmented minibatch Da = {nu (xx) | Vas E B} via augmentation function (-) We take fa(-) o (-) and =x (rxx ) for any Ia X' and u E {i,j). Given positive pair (ot, 2f)), the standard contrastive loss proposed by SimCLR O is formulated as follows:
sil (fo((()))..
EcL zf:0)= -- Σ log
(1)
e sim (im(tota
Σ
ZEB BBBBBBAAA (x+))
3

---

where spr(ps 9) = pT// q/llllllall is the cosine similarity function and t >> 0 is temperature parameter.
ACL 29 and DynACL 36
Let (X, doe) be the imput space X' with the infinity distance metric dx (2, x²) x - x' Il and B, [x] {a' E a dx, x') S c) be the closed ball of radius e >> 0
centered at x E X. Given a data point my X, the adversarial contrastive loss is as follows:
(ACU(Fx;)) = (1+ -ec zy;0) 0)+ + +11 (1-w) fCL ( x k 0),
(2)
where 5x arg max &Cr. lcr (F,, #3
in which E [0, 1] is hyperparameter and ø and 2 are adversarial data generated via projected
gradient descent (PGD) 37 54 55] within the e-balls centered at zt. and az. Note that ACL [29]
fixes = 0 while DynACL 36 dynamically schedules w according to its dynamic augmentation
scheduler that gradually anneals from strong augmentation weak one. We leave the details of the data augmentation scheduler (36] in Appendix C to the page limitation.
3 Methodology
In this section, we first create the causal graph for ACL which the fundamental premise for causal reasoning. Then, we introduce adversarial invariant regularization (AIR) according to the causal understanding in the context of ACL. and the learning algorithm of our proposed method. Lastly, we demonstrate the theoretical understanding of AIR and theoretically show that the style-independence property is generalizable to downstream tasks.
3.1 Adversarial Contrastive Learning (ACL) in the View of Causality
causal graph is arguably the fundamental premise for causal reasoning Mitrovic et al. 38 constructed the causal graph of SCL as shown in the left panel of Figure However, how to build causal graphs of ACL is still unclear in the literature. Therefore, we take the primitive step to construct the causal graph of ACL.
39
40
Previously,
To begin with, formalize the data generation procedure in the causal graph. Let X denote an unlabeled data point and y = {yz YT= denote a set of labels an unknown downstream task. Then, similar to Mitrovic et al. (38 and Zhang al. [56)
make the following assumptions: 1) the data point x is generated from the content factor c and the style factor s, i.e., c - x f- s, 2) the content independent of the style, i.e., IL 8, and 3) only the content, which is the original image data, is related to the downstream task, i.e., - yt-
Then, we build the learning procedure of CL. According to the above assumptions, the content serves an essential role in downstream classification tasks. Therefore, CL can be regarded as estimating the probability of the label given the content, i.e., p())lc). To learn the representations from the unlabeled data x, CL needs to maximize the conditional probability p(y/(z).
However, in practice, the label is unknown. Thanks to the causal concept of refinement g 38 we can construct proxy label 8d E y which is a refinement of y. This proxy label y should be consistent across different views of the unlabeled data which enables learning the representations. Intuitively, the proxy label yd can be viewed as a more fine-grained variant the original label ye- For example, refinement for recognizing cars would be the task of classifying various types of cars. Therefore, SCL. targets maximizing the probability of the proxy label given the unlabeled natural data, i.c., 1x) [38].
Besides + n ACL has an extra path x + nd since ACL needs first generate the adversarial data (Eq. and then learns representations from the unlabeled natural and adversarial data (Eq. According Eq. ( we observe that ACL only uses the observed unlabeled data for generating adversarial data, i.e., x + I + 8. Then, according to the paths - y and ă + y² from the causal view. ACL learns representations by maximizing both the probability of the proxy label R given the natural data (i.e., ((yx((r))) and that given adversarial data (i.e., p(y (()).
Lastly, we provide Theorem I to justify the rationality of constructed causal graph of ACL.
Theorem 1. From the causal view, in ACL, maximizing the conditional probability both p (2)) and ( is equivalent minimizing the learning objective of ACI. (29] that is the sum of standard and adversarial contrastive losses.

---

The proof of Theorem D in Appendix B.1
3.2 Adversarial Invariant Regularization
According to the independence of causal mechanisms 40
performing interventions on the style
variable should not change the probability of the proxy label given the unlabeled data, i.e., ply (z)) According to the path æ I y shown in the causal graph of ACL, we can obtain that
(x) = p(y ((){(=))(((1)))
(4)
under the assumption that the process of + ž y" satisfies the Markov condition. Therefore, we should make the conditional probability p pp( (z]]z) that is learned via ACL become style- independent. It guides to propose the following criterion that should be satisfied by ACL:
p pollt) (y"1))p ( = do( (y"))) do(r)))) VTa, E T,,
(5)
where do(r) as the short form of do(s 7) denotes that we perform the intervention on via data augmentation function 7(-). We leverage the representational distance between various views of natural data and their adversarial variants normalized by the softmax function calculate the conditional probability pdol (v" and pdo(r)) (ā(z) where {i,j} as follows:
gsim(fo(F")foot"")) )/f
p pootr)) (y
(
(6)
BLEB
BEEB
in which E B X is the original view of natural data, x² is the augmented view of natural data via augmentation T r,,)), 5" is the adversarial variant generated via Eq. (3) and is the temperature parameter. Note that we use x as the approximated surrogate of its true content c that incurs y (i.e.,
e c++g" R)).
To achieve the above criterion in Eq. (5) we propose an adversarial invariant regularization (AIR) to regulate robust representations as follows:
(y CAIR CAIR(B;6,c) KL (pdotr) (1 do) (f(=)) do(()) (1" (8|x)) B).
in which c 2 the adversarial budget and (p(=)) B) EXEB p(x) log HallbackkLeiibles (KL) divergence. AIR can enforce the representation to satisfy the criterion in Eq. (5), thus eliminating the effect of the style factors on the representation. Therefore, when setting e as a positive constant, CAIR( (B; 0, e) can effectively regulate robust representations of adversarial data to be style-independent
p(w)
q(z) denotes the
Besides, to explicitly regulate standard representations of natural data to be independent of style factors, we can simply set c = 0 of AIR. We formulate AIR with c 0 as follows:
CAIR CAIR(B;B,0) = KL (pde(r) (1 (1)
(8)
coimlfs(@)).o(=")))
where p do(tu) (v [2)
Vu {i.j}.
sim siin(tlless.:(())))))) Σ
24
Note that CAIR 0, 0) is the same formulation as the standard invariant regularization (SIR) [38] which can make the standard representation style-independent.
By incorporating adversarial contrastive loss with AIR, the learning objective function of our proposed method is shown as follows:
arg min Σ (ACI((E; + A (Am(U;0,0) X2 (U; ZEU
(9)
where N refers to an unlabeled dataset consisting of N samples, e > 0 is the adversarial budget, and X > 0 and X2 > 0 are two hyperparameters. The learning algorithm of ACL with AIR demonstrated Algorithm T Note that our proposed AIR is compatible with various learning objectives such as ACL 229 AdvCL 22), A-InfoNCE [50], and DynACL [36).
5

---

Algorithm ACL with Adversarial Invariant Regularization (AIR)
1: Input: Unlabeled training set U, total training epochs E, learning rate 7) batch size adversarial
budget c > O, hyperparameters X1 and Àg
2: Output: Pre-trained representation extractor hg
3: Initialize parameters of model fo o hg
4: for to E do
5: for batch m = I..., ||///1 do
6: Sample minibatch Bm from U
7: Update 0 + 0 -7 V Ex, fACL(ER; + + (AIR(Bm;0,0)) C
8: end for
9:
end for
3.3 Theoretical Analysis
We provide theoretical understanding of AIR in Proposition 3 based the decomposition of AIR shown in Lemma 2
We set e >> 0 of AIR by default in this section.
Lemma 2 (Decomposition of AIR). AIR in Eq.
be decomposed into terms as follows:
CAIR B;
=B xapdtte (2)
[KT (p (
+ B dolts (y do( (21
where (x ( (x)
is the expectation of KL divergence over Q³ and Q" Q2, Q²
are probability distributions.
The proof of Lemma is in Appendix B.2 Based on Lemma
propose Proposition that
provides the theoretical understanding of AIR.
Proposition 3. AIR implicitly enforces the robust representation satisfy the following two proxy criteria:
p do((T) (y (8) )oll))) (y (y ()),
(2) p do(T:) (i (f)
do(T,) (8/x).
The proof of Proposition 3 is in Appendix B.3
Remarks. Proposition 3 explains that AIR implicitly enforces the representational distance to be style-independent between the original view of natural data and their adversarial variants (i.e., as well as between the augmented view of natural data and their adversarial counterparts (i.e., do((Ta) (r) pdo(Ts) (ā(z)) Therefore, Proposition 3 explicitly presents that AIR is different from SIR (38, where SIR only enforces the augmented views natural
pdo(s)) [
data to be style-independent
Next, we theoretically show that on the assumption that y refinement of V, the style- independence property of robust representations will hold on downstream classification tasks in Proposition 4
Proposition 4. Ler y {1r be label set of downstream classification task, yR be refinement of y, and H. be the adversarial data generated on the downstream task. Assuming that T E B, [x]] and H. B.(x... we have the following results:
dolto (y ((7) (g)) = =p po(ls) ((G) = " do(T))) (ye (Tx)) =
VT, ET,
po((t) (ø(z) do(rs) (5(x) - p dot (i+(z) = =p polrs) (T,lz)) VT.,T ET.
The proof of Proposition is shown in Appendix B.4
Remarks. Proposition indicates that the robust representation of data on downstream tasks is still style-independent. Mitrovic et al. (38] empirically showed that the style-independence property of standard representations can significantly improve the robustness against common corruptions (26 downstream tasks. In addition, Proposition indicctes that AIR helps to find the invariant correlations that are independent of the style factors among different training distributions. It is similar to the objective of invariant risk minimization [ that can yield substantial improvement in the robustness against various corruptions incurred by distribution shifts. Therefore, the styleiidepenndene property of robust representations learned via ACL perhaps helps to enhance the robustness against input perturbations downstream tasks.

---

Table 1: Self-task adversarial robustness transferability.
SLF
ALF
AFF
Dataset
Pre-training
AA (%)
SA (%)
AA (%)
SA (%)
AA (%)
SA (%)
ACL [29
37.392006
78.27
40.61
75,56:00.9
49,4221007
82.14
ACL-AIR
38.89
80.03
41.39
78.29
49,84
82.42+006
CIFAR-10
DynACL 33
45.05
15.3
45.65
72.901000
50.321005
81.8610.11
DynACL-AIh
45.17
78.08
46.01
77.42
50.60
82.14
ACL [29]
15.78
45.70
17.36
42.69
24.166:0.29
56.6810.14
ACL-AIR
16.14
49.75
18.68
47.07
25.27
57.79
CIFAR-100
DynACL [36
19.31 #006
45.67
20.30
43.58:0012
24.70
57.224028
DyIACL-AIR
20.45.000
46.84
21.23 10.09
45.63
25.34
57.44
ACL 129
35,804006
67.904000
38. 11
69.9640.14
43.21
12.55+0.18
ACL-AIR
36.94.006
68.91
39.05
71.30
43.75
12.84.0.44
STL-10
DynACL 36
46.49 46.4990.0 10.05
69.59
47.69
67.63 10.12
43.64.0.13
12.1410.15
DynACL
47.6620066
72.30
48.89:008
71.60
48.10.0.1
73.10:0.17
Table 2: Self task Nommon--crruppin robustness transferability. The corruption severity (CS) ranging
from {1.3, 5} (denoted
The standard deviation is in Table 15
SLF
ALF
AFF
Dataset
Pre-training
CS-1
CS-3
CS-5
CS-1
CS-3
CS-5
CS-1
CS-3
CS-5
ACL 29
76.57
71.78
62.78
74.04
69.49
61.38
79.15
72.54
65.27
ACL-AIR
78.55
73.33
64.28
76.65
71.38
63.17
79.49
72.95
65.37
CIFAR-10
DynACL 33]
73.92
69.01
62.31
71.74
66.95
60.87
79.77
72.95
63.60
DynACL-AAIR
76.62
70.16
63.29
75.70
69.55
62.67
80.98
74.31
66.75
ACL 29
43.14
38.90
32.27
39.52
35.13
29.30
53.80
45.93
37.61
ACL-AIR
47.85
41.54
34.12
44.80
40.70
35.46
55.12
47.25
39.02
CIFAR-100
DynACL (36]
44.19
38.08
31.05
39.83
35.61
30.51
54.34
46.71
38.62
DynACL-AIR
45.36
39.21
32.44
42.03
36.48
30.49
55.31
47.33
39.13
Experiments
In this section, we demonstrate the effectiveness of our proposed AIR in improving ACL 29 and its variants 36 22 50) on various datasets including CIFAR-10 10 [II, CIFAR- 100 [31] STL- 10 [2]
CIFAR- I0-C 26 and CIFAR-100-C 22). Extensive experimental details are shown in Appendix
Pre-training, In the main paper, we demonstrate the results of applying our proposed AIR with d 0.5 and \u 0.5 to ACL 129 and DynACL (6) (denoted as "ACL-AIR" and "DynACL-AIR", respectively). We utilized ResNet- 18 [25] as the representation extractor following previous self-
supervised adversarial training methods 22 22 36 50]. We adopted the same training configuration of ACI. (29 using SGD for 1000 epochs with an initial learning rate of 5.0 and a cosine annealing schedule 35 The batch size B is fixed as 512. The adversarial budget 6 is set as 8/255. In the context of DynACL. we took the same data augmentation scheduler and the same scheduler of the hy- perparameter w as the setting in the original paper of DynACL [36). Note that, to reproduce the results of baseline methods, we downloaded the pre-trained weights of ACL [29] on and DynACL [36] on JUTAR-IO/CIIAAR-OOO/STT
from their official GitHub as the pre-trained representation extractor. We provide the extensive ablation studies C.2
the hyper-parameete (A1 and A2) in Appendix C.2 the ACL variants (including AdvCL (22 and A InfoNCCE (50)) in Appendix and the backbone models (including ResNet-50 and Wide ResNet (1I) in Appendix C.4
Finetuning procedures. We adopted the following three types of finetuning procedures: standard linear finetuning (SLF), adversarial linear finetuning (ALF), and adversarial full finetuning (AFF), to evaluate the learned representations. The former two finetuning procedures freeze the learned representation extractor and only train the linear classifier using natural or adversarial samples, respectively. We took the pre-trained representation extractor as weight initialization and trained the whole model using the adversarial data during AFF. The training config onfiguration of finetuning (e.g., the finetuning epoch and optimizer) exactly follows DynACL (36] Specifically, we used the official code provided in DynACL [36]* 's GitHub for finetuning and illustrate the experimental settings of finetuning in Appendix C as well. For each pre- pre-trained representation extractor, we repeated the finetuning experiments times and report the median results and the standard deviation.
Evaluation protocols. We let "AA" denote the robust test accuracy under AutoAttack (AA) 13 and "SA" stand for the standard test accuracy. To evaluate the robustness against common cornuption, report the mean test accuracy under 15 types of common corruptions (26) with the corruption
7

---

Table 3: Cross-task adversarial robustness transferability. D1 - D2 denotes pre-training and finetuning are conducted on the dataset D and D2(F D1), respectively.
SLF
ALF
AFF
D1 + D2
Pre-training
AA (%)
SA (%)
AA (%)
SA (%)
AA (%)
SA (%)
ACL 129
9.98
=204
11.09
28,582006
22.67:00.16
16.05200.1
CIFAR-IO
ACL-AIR
11.04
39.45
13.30
36.10.005
23.45::044
16.00
CIFAR-100
DynACL 136
11.01
27.66.0003
11.92.1005
24. 141009
24.1710.10
55.61
DynACL-AIR
12.20.0.:4
31.33
12.70.0.0:
28.701000
24.82
57.00...13
ACL 229
25.41
10.5310.10
27.17
51.7110.7
32.6610007
61.41
CIFAR-10
ACL-AIR
18.0010.12
61.9110.13
30,06
62.03
34.2610009
62.58.10.10
STL-10
DynACL 16
28.52
52.45
29.13
49.53
35.25
63.29
DynACL-AIR
29.88
29.
54.59
31.24:10.06
$7.14
35.66
63.7444012
ACL [29
18.72
60.901000
26.92.1..11
47.35
44.07
75.1910.10
LIFAR-100
ACL-AIR
19.90
64.89
27.65
60.794004
44.84
75.67 10.13
+ CIFAR-10
DynACL 36
25.23+0.12
59.12+0.1
28.92=#0.10
56,0940.14
47. 27.40=023
77.9240.18
DynACL-AIR
25.63+0.07
59.83+000
29.32=+0.06
56,65+006
47.92+0.32
78.44400.0
ACL 229
21.77
46. 19:00
24.46:20.09
45,40z011
28.7620007
16:103
CIFAR-100
ACL-AIR
22.44.0.04
10.52
26.55
53.241009
30.40
58.45-10.1
-> STL-10
DynACL 36
23.17 10.09
47.34
26.24
45,7010.1
31.1710.14
58.35
DynACL-AIR
23.24
48.20+008
26.60=00.5
48.55+0012
31.42=0007
58.59+0.10
Table 4: Cross-task common-corruption robustness transferabillity. The corruption severity (CS)
ranging from (1,3,5, 3,5 (denoted as (CS-(113,55")) The standard deviation is in Table 16
SLF
ALF
AFF
D1 - D2
Pre-training
CS-1
CS-3
CS-5
CS-1
CS-3
CS-5
CS-1
CS-3
CS-5
ACL 29
31.39
27.76
23.27
27.80
25.09
21.09
52.07
44.22
36.31
CIFAR- 11
ACL-AIR
37.82
32.83
27.06
34.67
30.09
25.07
52.81
44,95
37.01
+ CIFAR-100
DynACL 36
26.74
23.97
20.87
23.70
21.43
18.84
52.87
44.87
36.76
DynACL-AIR
30.35
26.53
22.76
27.67
24.35
20.96
54.00
46.01
37.75
ACL 29
59.65
55.14
49.09
56.15
51.96
47.04
72.94
66.05
59.17
CIFAR- 100 - CIFAR-IO
ACL-AIR
63.34
58.14
51.03
59.14
54.23
48.97
73.48
66.83
60.21
DynACL 36
57.14
52.07
47.03
56.31
52.03
47.40
75,89
69.08
62.02
DynACL-AIR
58.41
53.09
47.94
54.89
49.84
45.08
76.35
69.49
62.44
severity (CS) ranging from (1, 5) (denoted as "CS-(1, 5,)",, Specifically, we used the official
code of AutoAttack 13 and RobustBench [15] for implementing evaluations. In Appendix C.5
we
provide robustness evaluation under more diverse attacks [13 14 (2) In Appendix C.6
we provide
the test accuracy under each type of common corruption (26).
4.1 Evaluation of Self-Task Robustness Transferability
Self-task adversarial robustness transferability. rreports the self-task adversarial robust- ness transferability evaluated on three datasets where pre-trate pre-training and finetuning are conducted on the same datasets including CIFAR-10, CIFAR-100, and STL-10. In Appendix C.I we report the self-task robustness transferability evaluated in the semi-supervised settings. Table demonstrates that our proposed AIR can obtain new state-of-the-aae (SOTA) both standard and robust test accuracy compared with existing ACI. methods (29, 36). Therefore, It validates the effectiveness of our proposed AIR in consistently improving the self-task adversarial robustness transferability against adversarial attacks as well as standard generalization in downstream datasets via various finetuning procedures. Notably, compared with the previous SOTA method DynACL [36], DynACL-AIR increases standard test accuracy by 4.52% (from 72.90% to 77.42%) on the CIFAR-IO dataset via ALF, 2.05% (from 43. 43.58% to 45.63 %) on the CIFAR-100 100 dataset via ALF, and 1.17% (from 46.49% 47.66%) on the STL-10 dataset via SLF.
Self-task common-corruption robustness transferability. Table reports the self-task common- corruption 266 robustness transferability. Specifically, we conducted pre-training on CIFAR- 10/CIFAR-100 and then evaluated the test accuracy on CA--R----/CIFAR-III 100-C with various corruption severities after various finetuning procedures. Note that the test accuracy under each type of common corruption is reported in Appendix Table demonstraas that AIR leads to consistent and significant improvement in the robustness against common corruption. In addition, we observe that ACL always achieves much higher test accuracy under common corruptions than DynACL after SLF. We conjecture the reason is that DynACL uses weak data augmentations at the later phase of training, which makes DynACL more likely to overfit the training distribution of CIFAR- 10 and thus leads to worse performance under common corruptions via SLF.
8

---

Table 5: Self-task adversarial robustness transfcrability with post-processing. "++" denotes that the pre-trained models are post-processed via LP-AFF [32 36]. LP-AFF [32 first generates pseudo
labels via clustering and then adversarially pre-trains the model using the pseudo labels.
SLF
ALF
AFF
Dataset
Pre-training
AA (%)
SA (%)
AA (%)
SA (%)
AA (%)
SA (%)
DynACL++ [36
46.46'
79.81
47.951
78.84
50.31
81.94
CIFAR-10
DynACL-AIR ++++++++++++
46.99
81.80
48.23
79.56
50.65
82.36
DynACL++ [36
20.071
52.26
22.24
49.92
25.21
57.30
CIFAR-100
DynACL-AIR++
20.61
53.93
22.96
52.09
25,48
57.57
DynACL++ [36
47.21
70.931
48.06
69.51
41.84
72.36
STL-10
DynACL-AIR++
47.90
71.44
48.59
71,45
44.09
72.42
Table 6: Self-task adversarial robustness transferability evaluated on the Imagenette dataset.
ResNet-18
ResNet-50
Pre-training
AA (%)
SA (%)
AA (%)
SA (%)
DynACL [36]
57.15
79.41
58.98
80.74
DynACL-AIR
58.34
80.61
60.10
81.66
4.2 Evaluation of Cross- Task Robustness Transferability
Cross-task adversarial robustness transferability. Table 3 shows the cross-task adversarial robustness transferability where pre-training and finetuning are conducted on the different datasets. We can observe that AIR substantially improves both ACL's and DynACL's robustness against adversarial attacks [33 and standard generalization to other downstream datasets. Particularly, AIR improves the standard and robust test accuracy of ACL via ALF by 7.52% (from 28.58% to 36.10%) and 2.21% (from 11.09% to 13.30%), respectively.
Cross-task common-corruption robustness transferability. Table reports the cross-task common-corruption (26) robustness transferability where pre-training is conducted on CIFAR- 10/CIFAR-100 and finetuning is conducted on CETAR-000-/CCFAR-IIII
(26). The empirical
results show that our proposed AIR can consistently improve the accuracy under common corrup- tions with various severity, which validates the effectiveness of AIR in enhancing the robustness transferability against common corruptions.
4.3 Evaluation of Pre-Traine Models after Post- Post-Processing
Here, we report the self-task adversarial robustness transferability of pre-trained models after post- processing in Table 5] The post-processing method is linear probing and then adversarial full finetuning (LP-AFF)
32 which first generates pseudo labels via clustering and then further adversar- ially trains the pre-traine pre-trained model using the pseudo labels. "++" denotes that the pre-trained model is pest-processed via LP-AFF. We provide the comparison between DynACL++ and DynACL-AIR++ in T Note that the superscript denotes that the results of DynACL++ copied from Luo et al. [36]- The results validate that AIR is compatible with the trick of post-processing via LP-AFF 32 which can yield improved performance various downstream datasets.
4.4 Evaluation on Righh-Resolution Imagenette Dataset
To the best of knowledge, there exists reported result of ACL and DynACL high-resolution datasets in the existing papers. Therefore, we pre-trained 188 and ResNet-50 on Imagenette" of resolution 256 256 which is subset of ImageNet-IK using DynACL and DynACL-AIR. respectively. We compare and report the performance evaluated on Imagenette via SLF in Table 6 Our results validate that our proposed AIR is effective in enhancing the performance on high- resolution datasets. Note that, due to the limitation of our GPU memory, we set the batch size to 256 and 128 for pre-training ResNet-18 and ResNet-50 on Imagenette, respectively. We believe using a larger batch size during pre-tratee pre-training can even further improve the performance (9]
4.5 Evaluation via Automated Robust Fine-Tuning
While evaluating the transfcrability of pre-trained model, we observed that the performance on the downstream dataset sensitive to the hyper-paramett (e.g.. the learning rate) of the finetuning
We downloaded the Imagenette dataset from http:://iihbb..comm/stttnnnggggu

---

Table 7: Cross-task adversarial robustness transferability evaluated via AutoLoRa [7] Di - D2 denotes pre-training and finetuning are conducted on dataset D, and D2 Di). respectively. "Diff" refers to the gap between the performance achieved by AutoLoRa and that achieved by vanilla finetuning (reported in Table 3
Finetuning mode
AutoLoRa
[47
Diff
Dj D
Pre-training
AA (%)
SA (%)
AA
(%)
SA (%)
DynACL
33
29.68
58.24
+0.51
+5.83
SLF
DynACL-AII
29.75
60.53
+0.11
+4.69
CIFAR-IU 10
DynACL
33
31.34
57.74
+1.75
+8.19
ALF
STL-10
DynACL-AII AIR
31.65
59.44
+0.41
+2.30
DynACL 36
35.55
64.16
+0.30
+0.63
AFF
DynACL-AIR
35.81
64.28
+0.15
+0.54
DynACL [36]
23.31
50.93
+0.14
+3.39
S1.F
DynACL-AIR
23.49
51.28
+0.25
+3.08
CIFAR-100 STI--10
DynACL [36
26.53
51.74
+0.29
+6.04
AL.F
DynACL-AIR
26.89
49.02
+0.29
+0.47
DynACI. 13
31.25
58.44
+0.08
+0.09
AFF
DynACL-AIR 31,57
58.65
+0.15
+0.21
procedure. It would require extensive computational resources search for appropriate hyper- parameters to achieve a satisfactory performance. To mitigate this issue, we leverage an automated robust finetuning framework called LoRa 47], which automatically schedule the learning rate and set appropriate papee-pprameters during finetuning. We report the performance achieved by AutoLoRa to further justify the SOTA performance by our proposed AIR Table 7
Table 7 shows that AutoLoRa can further enhance the performance of a pre-trained model on the downstream tasks without searching for appropriate hyper-pan syper-parameters since the value of "Diff" is consistently larger than 0.0. Besides, Table 7 justifies that our proposed AIR is effective in enhancing robustness transferability via various finetuning methods.
5
Conclusions
This paper leveraged the technique of causal reasoning to interpret ACL and proposed adversarial invariant regularization (AIR) to enhance ACL. AIR can enforce the learned robust representations to be invariant of the style factors. We improved ACL by incorporating the adversarial contrastive loss with weighted sum of AIR and SIR that an extension of AIR the standard context. Theoretically, we showed that AIR implicitly encourages the representational distance between different views of natural data and their adversarial variants to be independent of style factors. Empirically, comprehensive results validate that proposed method achieve new state-of-th art performance in terms of standard generalization and robustness against adversarial attacks and common corruptions downstream tasks.
Acknowledgements
This research supported by the National Research Foundation, Singapore under its Strategic Capability Research Centres Funding Initiative, Australian Research Council (ARC) under Award No. DP230101540 and NSF and CSIRO Responsible Al Program under Award No. 2303037. Any opinions, findings and conclusions or recommendations expressed in this material those of the author(s) and do not reflect the views National Research Foundation, Singapore.
References
[1] Jean-Ban Jean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? Advances in Neural Information Processing Systems, 32, 2019.
[2] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: query-efficient black-box adversarial attack via random search. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII, pages 484-501. Springer, 2020.
10

---

[3]
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint arXiv:1907.0220 2019.
[4] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint (rxXi::1088002288 2021.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-11011, 2020.
[6] Varun H Buch, Irfan Ahmed, and Mahiben Maruthappu. Artificial intelligence in medicine: current trends and future possibilities. British Journal of General Practice, 68(668):143-144 2018.
[7] Peter Bühlmann. Invariance, causality and robustness. 2020.
[8] Krzysztof Chalupka, Pietro Perona, and Frederick Eberhardt. Visual causal feature learning. arXiv preprint arXivv1412.23309 2014.
(9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[10] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong cuni-supervised learners. Advances in neural information processing systems, 33:22243 33:22243-22255, 2020.
[11] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
[12] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings. 2011.
[13] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206-2216. PMLR, 2020.
[14] Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with a fast adaptive boundary attack. In International Conference on Machine Learning, pages 2196-2205. PMLR, 2020.
[15] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateck Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. arXiv preprint arXiv:2010. 09670, 2020.
[16] Zhiyuan Dang, Cheng Deng, Xu Yang, Kun Wei, and Heng Huang. Nearest neighbor matching for deep clustering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13693-13702, 2021.
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 leee, 2009.
[18) Jiahua Dong, Yang Cong, Gan Sun, Bineng Zhong, and Xiaowei Xu. What can be transferred: Unsupervised domain adaptation for endoscopic lesions segmentation. In CVPR, 2020.
[19] Jiahua Dong, Yang Cong, Gan Sun, Zhen Fang, and Zhengming Ding. Where and how to transfer: Knowledge aggregatioo-induued transferability perception for unsupervised domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1(2):1-18, 2021. doi: 10.1 10.1109/TA..M..022 2021,.1281660.
[20] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap: Learning audio concepts from natural language supervision. arXiv preprint 27X.v:2206.04769, 2022.
[21] Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre-training help deep learning? In Proceedings of the thirleenth international conference on artificial intelligence and statistics, pages 201-208. JMLR Workshop and Conference Proceedings, 2010.

---

[22] Lijie Fan, Sijia Liu, Pin- Yu Chen, Gaoyuan Zhang, and Chuang Gan. When does contrastive learning preserve adversarial robustness from pretraining to finetuning? Advances in Neural Information Processing Systems, 34:21480-21444 21492, 2021.
[23] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In Yoshua Bengio and Yann LeCun, editors, The Third International Conference on Learning Representations, 2015.
[24] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo. Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning, Advances in neural information processing systems, 33:21271-212222 2222444, 2020.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[26] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
[27] Chih-Hui Ho and Nuno Nvasconcelos. Contrastive learning with adversarial examples, Advances in Neural Information Processing Systems, 33: 13:17181177093, 2020.
[28] Tianjian Huang, Shaunak Ashish Halbe, Chinnadhurai Sankar, Pooyan Amini, Satwik Kottur, Alborz Geramifard, Meisam Razaviyayn, and Ahmad Beirami. Robustness through data augmentation loss consistency. Transactions on Machine Learning Research.
[29] Ziyu Jiang. Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. Advances in Neural Information Processing Systems, 33:16199-162101 2020.
[30] Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial superviid contrastive learning. Advances in Neural Information Processing Systems, 33:2983-209 2994, 2020.
[31] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[32] Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-diitributton. The Tenth International Conference on Learning Representations, 2022. URL httttt::///pprreeeeeee htt.fforummidd-UVVeeFZSSSS
[33] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security, pages 99-112 12. Chapman and Hall/CRC, 2018.
[34] Phuc H LehKhc, Graham Healy, and Alan F Smeaton, Contrastive representation learning: framework and review. IEEE Access, 8:193907- 2020.
[35] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608. 03983, 2016.
[36] Rundong Luo, Yifei Wang, and Yisen Wang. Rethinking the effect of data augmentation in adversarial contrastive learning. In The Eleventh International Conference on Learning Representations, 2023. URL httttss//tppprreeeee
[37] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018
[38] Jovana Mitrovic, Brian McWilliams, Jacob C Walker, Lars Holger Buesing, and Charles Blundell. Representation learning via invariant causal mechanisms. In International Con- ference on Learning Representations, 2021. URL. / net/forum?id
9p2ekP904Rs
[39] Judea Pearl. Causality. Cambridge university press, 2009.
[40] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: founda- tions and learning algorithms. The MIT Press, 2017.
[41] Tal Ridnik, Emanuel Ben- Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint XxXv:2214411972, 2021.
12

---

[42] Joshua Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592. 2020.
[43] Axel Sauer and Andreas Geiger. Counterfactual generative networks. arXiv preprini arXiv:2101.06046, 2021.
[44] Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-taile classification by keeping the good and removing the bad momentum causal effect. Advances in Neural Information Processing Systems, 13:1513-1524, 2020.
[45] Ruoyu Wang, Mingyang Yi, Zhitang Chen, and Shengyu Zhu. Out-of-distribution generalization with causal invariant transformations. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pages 375-385, 2022.
[46] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoe Le. Adver- sarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 819-828, 2020.
[47] Xilie Xu. Jingfeng Zhang, and Mohan Kankanhalli. Autolora: A parameter-free automated robust fine-tuning framework. arXiv preprint arXiv:2310.018/8, 2023.
[48] Xilie Xu, Jingfeng Zhang, Feng Liu, Masashi Sugiyama, and Mohan Kankanhalli. Efficient adversarial contrastive learning via robustness-awan coreset selection. In NeurlPS, 2023.
[49] Xu Yang, Cheng Deng. Kun Wei, Junchi Yan, and Wei Liu. Adversarial learning for robust deep clustering. Advances in Neural Information Processing Systems, 33:9098-9108 2020.
[50] Qiying Yu, Jieming Lou, Xianyuan Zhan, Qizhang Li, Wangmeng Zuo, Yang Liu, and Jingjing Liu. Adversarial contrastive learning via asymmetric infonce. In European Conference on Computer Vision, pages 53-69. Springer, 2022.
[51] Sergey 7agoruyko and Nikos Komodakis.
Wide residual networks. arXiv preprint
WxXiv:165.007166 2016.
[52] Chaoning Zhang. Kang Zhang, Chenshuang Zhang, Axi Niu, Jiu Feng, Chang D Yoo, and In So Kweon. Decoupled adversarial contrastive learning for self-supervised adversarial robustness. In European Conference on Computer Vision, pages 725-742. Springer, 2022.
[53] Cheng Zhang, Kun Zhang, and Yingzhen Li. causal view on robustness of neural networks. Advances in Neural Information Processing Systems, 33:289-301, 2020.
[54] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
[55] Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan S. Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In Interna- tional Conference on Machine Learning, pages 1 11778-1 127. PMLR, 2020.
[56] Yonggang Zhang, Mingming Gong, Tongliang Liu, Gang Niu, Xinmei Tian, Bo Han, Bernhard Schölkopf, and Kun Zhang. Adversarial robustness through the lens of causality. In International Conference on Learning Representations, 2022. URL https::/opeerrviiee nww.fforet id=cZAilyHpixQ
13

---

A Mathematical Notations
Table 8: Notation Table
Notation
Description
input space
X
data point
T
transformation
7 E T
data augmentation fraction
augmented data point via data augmentation fraction vu(-)
adversarial data
B x D
minihatch of 3 original image samples
minibutch of 3 auginented image sumples via the data augmentation fraction mu(-) minibatch of adversarial counterparts of 0 augmented samples via the data augmentation fraction s.(()
B"
U AW
unlabeled data set of N image samples
label space on the downstream tasks target label on downstream tasks the refinement the target lahel space
Mr DRR
R E
the proxy label
2 E E
the proxy label of the data point zi augmented via Tu(-) projection space
feature space where the contrastive loss is applied the representation
by 2
parameterized 8 the projector
9: : -
(6) go
the composition of the representation extractor and the projector the cosine similarity function KL divergence function style variable
simi
KL
content variable
the weight of standard regularization
the weight of adversarial regularization
B Proof
B.1 Proof of Theorem 1
Theorem I (restated). From the causal view, in ACL, maximizing the conditional probability both (x) and ())) is equivalent to minimizing the learning objective ACL [29] that the sum of standard and adversarial contrastive losses.
Proof. To begin with, we formulate the proxy label driven y by the data augmentation the context of contrastive learning. We denote the index æt E B' and æy. E B as ki and kj. respectively. For the augmented data zt. E B and arf E Bí, we denote their corresponding proxy labels as y and pky. which is formulated as follows:
yki = 1kj1
Økj = 1
(10)
where ka E (0, 2 is one-hot label where E j. the value at index ku is 1, and the values other indexes are The one-hot label refers to the refined proxy label of the augmented view of the data point, which directs to its peer view. In the causal view of SCL and ACL, each augmented view of the data has its own proxy label. Therefore, (A)CL aims to minimize the differences in representation output of different views of the same data. Specifically, the conditional probability p(y lar,; 0) and p(v | 0) is formulated as follows:
sim clin(ooe().sottut)))
p(uilo:0) p (11g12630)
(11)
sinu siin(foltsssaaaalll
Σ
xEBiUBi
p(uBl=(:0) =pppp1al =p
(12)
REBLUBI ((a
where sim(p.g) =pTg/lpllllgl is the cosine similarity function and t 0 temperature parameter.
14

---

Then, we have the following results:
arg min
Σ
(13)
EUEU
arg min
log
log sim @5im(16005).6000)))))
Σ
TyE
# B.UBAMM
BEB
(14)
ssim csim(fotas)hooook))))
ciim(fa(e2..oottt)))
=
arg max
log
SERUBBA ((a)) cdin(fols))/oltt)))/
log
sim (iin(o(=2()dol)))))
0
Iy ENEU
LEB.UB-Mar
(15)
cdim((ole())oo(ss))))
Sim sim comm(f66r)).fooo55)))
arg max
(16)
ciin(toeet)daott)))t
Σ
#VEU
SEB ((r1))
REB'BBA(
arg max
p( 1 0) + Σ p(tnlla(;))
(17)
8
BIEU
#2EU
arg Σ
pluit p(u1f (():0).
(18)
8
UUEU
Therefore, we can conclude that SCL actually maximizes the conditional probability of the proxy label given the unlabeled data, i.e., (z))
In the adversarial context, we only need to replace the natural data of and oi in Eq. 13 and 18 with the adversarial data o and oie generated in Eq. (3). Therefore, we have the following results:
Σ lcr ( z4: tou (o #z:
arg min
9
SAEU
arg
p(v : 0)
plut) (z p(u + p(
(19)
0
where the adversarial data ž and 2 are generated in Eq. 3 Therefore, from the causal view, in ACL [29], maximizing both the conditional probability of the proxy label given the natural data (i.e., (x)) and that given adversarial data (i.e., p(y R1 (()))) is equivalent to minimizing the sum of standard and adversarial contrastive losses.
B.2 Proof of Lemma 2
Lemma (restated).
AIR in Eq. can be decomposed into two terms as follows:
CAIR B: c) - (2)z) (p (y (1" (y
+ E Ba (8))8) (o (2)2))
where Ex= (TT (x))) the expectation of KI. divergence over Q3 and 22,0Q.Q Q probability distributions.
15

---

Proof. We transform AIR in Eq. as follows:
CATR( = KL
p po(tt)) (y T) doltd (1)=)]1 (a)x))
(o(o)) (w (y "1 d (f/ar); B)
(20)
pdotto ( ( (8/zz)
Σ EE
p pdo(ri) (1) p) (i)x) log pdo(os))
(21)
) ( (r)
pdl(T)) (1)
 p (8) [1)
Σ
da((T))
(y [i) do
(
Bol
pdo(T))
(y ()) (.)
log
 pdo(T)) (á]z²)
(22)
#EB
p
(1"12)
d (y ( log
(do(To) (8)z))
pollo
p
p
+
do(r=) (i|x) log
(
pdo(t))( (g (g) (1)
(23)
ZEB
p
( (ج]
Σ
pdo(Te) (y ( log
pdo(T)) (2|z)
ploo pdo(Ts) ( (+))
#
xE/=}
pd((t)) (7)1)) log pdo(os) 
pdo(ts) (2) ( (G)
(24)
ZEB
(a)
Σ (L (p do(ts)
(i)
sEB
+
KL (L do(()) (i (T
pdo(t)) (1))))
(25)
ZEB
E Ex
(z); {x)))
E Paggor
(a)a);; (x))
(26)
[
(i) [+]=))
(27)
Eq. (2T) and 25) are obtained by using the KL divergence (L(p - EvED p(x) log p(z)
q(z)
We obtained Eq. (27) by omiting the term {œ} in Eq. (26 for simplicity and finished proving
Lemma 2
B.3 Proof of Proposition 3
Proposition 3 (restated).
AIR implicitly enforces the robust representation to satisfy the following
two proxy criteria:
(1) do( (y ()) = p dolls)
(2)
p do(to (H)z) = do(ts) (f)z))
Proof. To enforee the robust representation to satisfy the two proxy criteria shown in Eq. 28). we need to regulate the ACI with the following regularization
KT. (pdo(T))
do( (y
(());
B) + KT. ( (paottd (3 do(ss) (F(a); B
(28)
do(n)) log p [T)
do(T.) (ā xx)
do(ns) (5/z)) log
(29)
p
pdo(os) (5|z)
#EB
BEB
-Σ KL (L do
(x)) +
KL
do
(ilz))lp dot)) (a(x); {x]). (30)
BEB
#EB
Compared with Eq.
25 in the proof of Lemma 2 we can find that AIR also penalizes the same two KI. divergence terms while AIR has two extra calibration terms do(t.) (f(xx) and (y that adjust the confidence of each KL divergence term, respectively. We empirically studied the impact of
16

---

the calibration term in Appendix C.8 and found that the calibration term is beneficial to improving the performance of ACL. Therefore, indicates that AIR implicitly enforces the robust representation to satisfy the two proxy criteria shown in Eq. (28).
B.4 Proof of Proposition
Proposition 4 (restated). Y {y be label set of a downstream classification task, y be a refinement of y. and ie be the adversarial data generated on the downstream task. Assuming that it E B.(x]] and it Be[x]], we have the following results:
pdolod (1) =- p R (T) = pdolto) (T)) = =poo(ss) (velTe) VTa ET, p = p pdllts) (z/or) - p (Frlz)) = (lol VT.,y ET.
Proof. The proof is inspired by Mitrovic et al.
138
Fort E {I,.... T), we have
( p dalte (Hele) (p (2
(31)
f plualo" ( dol (5 ddy
(32)
/ ((ull"")ddotz (1 (x) Jdy"
(33)
da(ls) (wel
(34)
Eq. (31) holds due to the fact that y is a refinement of according to the definition of the refinement shown in Mitrovic et al. [38]. Eq. (32) holds since the relationship between yt and y is independent of the style, i.e., pdo(T.) (ye by
(Bely") Eq. (33) holds because of the assumption that the prediction of adversarial data is independent of the style. Due to that the condition pdo(tt) ()- = p pdo(T)) (2) holds for any ă E B, [z]. thereby, this condition also holds for o. E B, [ar]. Lastly, we obtain that the prediction of adversarial data will be still independent of the style factors on downstream tasks.
Next, due to that the condition (o(o)))(()) = ( holds for any I E [x),). this condition will hold for & E [x[x] well, i.e., p (ā)(z) = (1,l2r) Therefore, the consistency between adversarial and natural data will be still invariant of the style factors on the downstream tasks.
C Extensive Experimental Details and Results
Experimental environments. We conducted all experiments on Python 3.8.8 (PyTorch 1.13) with NVIDIA RTX A5000 GPUs (CUDA 11.6).
Pre-training details of ACL 29 Following Jiang et al. 29 we leveraged ResNet- 18 25 with
the dual batch normalization (BN) [46] as the representation extractor, where one BN is used for the standard branch of the feature extractor and the other BN is used for the adversarial branch, during conducting ACL [29] and its variant DynACL (36) We pre-trained ResNet- 18 models using SGD for 1000 epochs with initial learning rate 5.0 and a cosine annealing schedule (35] During pre-training, we set the adversarial budget e as 8/255, the hyperparameter w/ as 0.0, and the strength of data augmentation as 1.0. As for the reproduction of the baseline, we used the pre-trained weights published in the GitHub of ACL as the pre-trained representation extractor for finetuning.
Pre-training details of DynACL (36]. The training configurations of DynACL 336 followed ACL 29 except for the strength of data augmentation and the hyperparameter w. We denote the
Link of pre-trained weights via ACL. on CIFAR-10.
Link of pre-trained weights via ACL on CIFAR- 100.
17

---

Table 9: The robust/standard test accuracy (%) achieved by ACL.AIR with different A and X2-
A
0.00
0.25
0.50
1.00
A1
0.00
37.39/78.27
38.61/79.48
38.70/79.96
38.76/79.81
0.25
37.55/78.53
38.70/79.53
38.79/79.65
38.73/79.76
0.50
37.51/78.97
38.63/79.72
38.89/80.03
38,82/80.08
1.00
37.04/79.17
38.24/79.64
38.39/80.09
38.77/79.83
Table 10: Performance in semi- sumi--upervised settings evaluated on the CIFAR- 10 task.
Label ratio
ACL [29
ACL-AIR
DynACL [3
DynACL-AIR
AA (%)
SA (%)
AA (%)
SA (%)
AA (%)
SA (%)
AA (%)
SA (%)
1 %
45.63
74.84
45.97
76.63
45.78
76.89
46.25
78.57
10%
45.68
76.30
46.17
77.52
47.08
78.22
47.41
79.94
strength of data augmentation and the hyperparameter at epoch as Me and we respectively, where
K
He = 1 - KKK E' CE
1-1}
(35)
We = V- 11-111114
E E-1)
(36)
in which the decay period K = 50, the reweighting rate v = 2/3, the total training epoch F 1000.
In our implementation of DynACI., we only take the dynamic strategy and do not take the trick of the stop gradient operation and the momentum encode (24 [1]. As for the reproduction of the baseline,
we downloaded the pre-trained weights published in the GitHub of DynACL as the pre-trained representation extractor for finetuning.
Details of finetuning procedures. As for SLF and ALE, we fixed the parameters of the representa- tion extractor and only finetuned the linear classifier using the natural training data and adversarial training data respectively. As for AFF, we finetuned all the parameters using the adversarial training data. For all finetuning procedures, we used SGD for 25 epochs for linear finetuning and full finetun- ing respectively and momentum is set as 2e 4. The initial learning of linear finetuning is set as 0.01 on CIAFR- 10, 0.05 on CIFAR- 100, and 0.1 on STL-10. The adversarial budget is fixed as 8/255 for ALF and AFF. In practice, we used the finetuning code published in the GitHub of DynACL for implementing finetuning procedures.
C.1 Performance in Semi-Supervised Settings
Table I0 reports the performance evaluated in semi-supervised settings [| during the finetuning procedure. In semi-supervised settings, following ACL [29] and DynACL 36 we first standardly finetuned the pre-trained model using the labelled data and then generated the pseudo labels using the standardly finetuned model. Then, we finetuned the model using the data with pseudo labels as
well as the labelled data via AFF. The results validate that AIR (Aı 0.5, A2 0.5) consistently
enhance both robust and standard test accuracy of ACL and its variant DynACL in semi-supervised settings.
C.2 Impact of the Hyper-Parameterr X and X2
We show the performance achieved by ACL-AIR of different A E (0.00, 0.25, 0.50, 1.00) and X2 (0.00, 0.25, 0.50, 1.00) on the CIFAR-10 task in Table 9 We can notice that when only leveraging adversarial regularization (A, = 0, X2 > 0), both standard and robust test accuracy gain significant improvement, which indicates that adversarial regularization serves an important role in improving the standard generalization and robustness of ACL. By incorporating adversarial and standard regularization together (when both A₁ > 0 and A2 0), the standard generalization of ACL gets further improved. Table 9 guides us to set λ = 0.5 and A2 0.5 since it can yield the best performance.
C,3 Incorporating AIR with Two Extra Variants of ACL
Here, we show extra results to validate that our proposed AIR can enhance two extra variants of ACL, including AdvCL 22] and AdvCL with A-InfoNCE (50]
Link of pre-trained weights via DvnACL CIFAR- 10.
Link of pre-trained weights via DynACL. on CIFAR-TUM 100.
Link of pre-trained weights via DynACL on 10.
18

---

Table 11: Self-task adversarial robustness transferability evaluated on CIFAR-I0 using various
backbone networks including ResNet-34, ResNet-50, and WRN-28-10.
Network
Pre-training
AA (%)
SA (%)
DynACL [6
47.01
78.84
ResNet-34
DynACL-AIK
47.56
80.67
DynACL [6
47.19
79.65
ResNet-50
DynACL-AIR
47.82
81.27
DynACL [36
41.13
71.23
WRN-28-10
DynACL-AIR
43.86
73.41
Table 12: Self-task adversarial robustness transferability evaluated on CIFAR- 10 achieved by Ad-
vCL [22], AdvCL-AIR, A-InfoNCE [50], and A-InfoNCE-AIR,
Pre-training
AdvCL [22]
AdvCL-AIR
A-InfoNCE (50)
A-InfoNCE-AIR
AA (%)
42.58
43.24
42.68
42.84
SA (%)
80.78
81.53
83.18
83.99
Table 13: Self-task adversarial robustness transferability evaluated on the CIFAR-10 dataset via SLF
under various attacks including APGD-CE [3] APGD-DIR 13
FAB
14] and Square Attack (2]
Pre-training
APGD-CE (%)
APGD-DLR (%)
FAB (%)
Square Allack (%)
ACI. [29
39.66
41.18
39.27
48.98
ACL-AIR
40.39
40.98
41.01
49.80
DynACL [36]
46.28
46.41
45.56
50.06
DynACL-AIR
47.05
47.29
46.02
50.58
AdvCL [2)] leverages extra contrastive view of high- figh-frreuency components and the pseudo labels
generated by the clustering method. We leveraged our proposed method IR (λ1 = 0.5, A2 = 0.5) to further improve the performance of AdvCL. Yu et al. (50] proposed asymmetric InfoNCE objective (A-InfoNCE) that treats adversaries as inferior positives that induce weaker learning signals, hard negatives exhibiting higher contrast to other negative samples.
We used the code provide by Yu al. [50] their GitHut to implement the pre-training of AdvCL. as well as A-InfoNCE, and the finetuning procedure. In Table [2] we show that our proposed AIR (A1 = 0.5, A 0.5) can consistently improve the robust and standard test accuracy of two extra
variants 22 50
C.4 Applicability with Different Backbone Networks
In this subsection, demonstrate that DynACL-AIR can consistently improve the performance
ResNet-34, ResNet-50 (25), and WRN-28- 10 [5I) We followed the same training configurations of
pre-training and finetuning (SLF) in Section 4 except for the backbone network. Table [1 validates
that IR can further enhance the robust and standard test accuracy DynACL on various backbone networks.
We observe that WRN-28- 10 yields a worse performance compared ResNet-50. We conjecture that it is because set the batch size to 128 during pre-training due to the limitation of our GPU memory. We believe using a larger batch size during pre-training can further improve the performance of WRN-28-10 10 according to Chen et al. 19)
C.5 Robustness Evaluation under Various Attacks
In this subsection, we conducted the robustness self- eeffttranferrability against three strong white-bon box attacks (APGD- CE [3]. APGD-DLR [3 and FAB (14) and one strong black-box attack (i.e.. Square Attack (2). We evaluate the robustness on the 11 dataset via SLF and report the results in Table 13 The results validate that our proposed method can consistently improve robust
test accuracy over various adversaries.
C.6 Test Accuracy under Each Type of Common Corruption
We report the test accuracy under each type of common corruption 26 with corruption severity being fixed as 5 in Table 14] We used pre-trained models on CIFAR-10 via DynACL and DYR,CL-AIR,
GitHub provided et
[50]
19

---

Table 14: Test accuracy evaluated on CIFAR-I0-C under each type of common corruptions with corruption severity being fixed as 5 of CIFAR- 10 pre-trained models after SLF and AFF.
Prodmining
Finctuning
Naise
Inpuise
Digial
DysACT:
01
2064
DynACI-AAI
71.18
66.72
63.94
DynACI-AJR
DyaAct Bi
APP
74.54
1400
75.33
1482
06.87
67.06
7263
1144
74.78
13.31
09:22
70.39
75.66
75.05
07.33
69.07
67.47
66.20
26.74
27.30
09.21
71.24
1982
18.2
7585
10.00
79,09
1730
30.64
19325
Table 15: Standard deviation of robustness self celf-transferabbiiit against common corruptions reported T
SLF
ALF
AFF
Datasct
Pre-training
CS-1
CS-3
CS-5
CS-1
CS-3
CS-5
CS-1
CS-3
CS-5
ACL 29
0.06
0.25
0.13
0.13
0.17
0.10
0.11
0.17
0.21
ACL-AIR
0.03
0.02
0.02
0.07
0.09
0.08
0.16
0.20
0.14
CIFAR-10
DynACL 36
0.02
0.01
0.02
0.07
0.16
0.09
0.13
0.20
0.18
DynACL-AIR
0.01
0.03
0.03
0.07
0.11
0.14
0.16
0.11
0.10
ACL. [29
0.01
0.09
0.02
0.12
0.14
0.11
0.09
0.04
0.17
ACL-AIR
0.04
0.02
0.02
0.13
0.09
0.04
0.17
0.04
0.16
CIFAR-100
DynACL 36
0.03
0.02
0.01
0.14
0.07
0.12
0.08
0.00
0.16
DynACL-AIR
0.08
0.05
0.02
0.08
0.12
0.05
0.16
0.16
0.10
Table 16: Standard deviation of robustness self- transffrrrrrb against common corruptions reported
Table
SLF
ALF
AFF
D, D2
Pre-training
CS-1
CS-3
CS-5
CS-1
CS-3
CS-5
CS-1
CS-3
CS-5
ACL 29
0.05
0.02
0.03
0.07
0.09
0.05
0.15
0.10
0.31
CIFAR-10
ACL-AIR
0.01
0.04
0.04
0.09
0.12
0.10
0.11
0.16
0.15
CIFAR-100
DynACL 336
0.04
0.04
0.05
0.11
0.07
0.07
0.19
0.18
0.10
DynACL-AI
0.03
0.02
0.03
0.12
0.13
0.09
0.18
0.04
0.07
ACL 29
0.03
0.03
0.03
0.08
0.09
0.14
0.08
0.07
0.15
CIFAR-100
ACL-AIR
0.03
0.02
0.03
0.09
0.15
0.13
0.13
0.14
0.20
CIFAR-10
DynACL [36]
0.03
0.01
0.02
0.11
0.12
0.14
0.05
0.14
0.19
DynACL-AIR
0.02
0.08
0.02
0.06
0.08
0.09
0.14
0.21
0.17
Table 17: Impact of calibration terms evaluated on the CIFAR 10 task.
SLF
ALF
AFF
Pre-training
Calibration
AA (%)
SA (%)
AA (%)
SA (%)
AA (%)
SA (%)
ACL-AIR
38.55
79.80
40.80
77.57
49.51
81.95
ACL-AIR
V
38.70
79.96
41.09
77.99
49.59
82.30
DynACL-AIR
45.09
77.79
46.01
76.12
50.54
82.35
DynACL-AIR
45.23
78.01
46.12
77.01
50.66
82.62
Experimental settings are the same as Section [ The results validate that can consistently improve the test accuracy under each type of common corruption.
C.7 Standard Deviation of Test Accuracy under Common Corruptions
Here, we provide the standard deviation the test accuracy under common corruptions in Tables 16
C.8 Impact of Calibration Terms
In the proof of Proposition 3 we show that there exist calibration terms do((t) (t)zz and p do( in AIR. Here, we empirically investigate the impact of calibration on the performance in Table The results empirically validate that the calibration term is an essential component of AIR to adjust confidence of the KI. divergence and further enhance performance of ACL.
D Limitations
22 50 36 our proposed ATR is not One of the limitations that similar to previous works 29 helpful in improving the efficiency and scalability of ACL methods. Therefore, robust pre-training via
20

---

ACL-AIR is still unfriendly to the environment due to emitting much carbon dioxide and consuming much electricity. Besides, applying ACL-AIR to large-scale datasets such ImageNet-21K 44 still computationally prohibitive with limited GPU sources.
E
Possible Negative Societal Impacts
Our proposed method aims to improve the performance of robust ceff-supervise pre-training meth- ods 29 22 50 36). The robust self-supervised pre- pre-training procedure is extremely time-consuming since needs to spend much time generating adversarial training data, which thus leads to consuming lot of electricity and emitting lots of carbon dioxide. Unfortunately, helpful in improving the efficiency of ACL. Consequently, our study could exacerbate the greenhouse effect and be not conducive to environmental protection.
proposed method is not
21


Reviews:
[{"review_id": "U42jOXJBQo", "reviewer": "Reviewer_fVht", "summary": "This paper proposed a method to advance adversarial contrastive learning by utilizing a technique called causal reasoning. The adversarial invariant regularization (AIR) proposed in this paper demonstrated a style factor. Additionally, the effectiveness of the proposed method was empirically shown using CIFAR10 and CIFAR100.", "questions": "- It would be good to mention y^R in the main text.\n- In equation 3, adversarial for x_k^i and x_k^j is expressed in one formula. I'm curious if x_k^j also gets its perturbation updated when creating the adversarial for x_k^i.\n- It would be great to mention how much performance difference there is, how much slower the speed is, and how much the transferability improves when compared to Supervised Adversarial Training (Madry, TRADES).", "limitations": "- There is a limitation in the hyperparameter tuning of lambda1 and lambda2.\n- There is a limitation that the gain in performance is marginal.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Theoretically, it explained how the KL divergence can provide invariant regularization.\n- The paper is well organized and most explanations are clearly written and easy to understand the approach and the theoretical explanations.", "weaknesses": "- The theoretical analysis of this paper (section 3.2 and section 3.3) appears to apply adversarial examples to the theorem shown in paper [1], which seems closer to an application rather than a novel combination, resulting in a perceived lack of originality.\n- Although most theorems and reasoning methods are well explained, expressing which input types the KL divergence loss of SIR and AIR applies to would be more effective for reproducing this paper through implementation.\n- Furthermore, the performance gain shown in robustness experiments from the proposed regularization is considered to have low significance as it is less than 1% in most datasets and tasks.\n- Also, there is insufficient explanation as to why the proposed method is more helpful for self-supervised adversarial robustness than existing methods, making it difficult to verify the experimental significance of the proposed method. Especially in the experimental results, AIR helps both clean performance and robustness compared to SIR, but there is no clear justification for why these two components should be used simultaneously.\n- It is unclear why a model trained with SIR and AIR regularization for adversarial x and natural x performs well against common corruption.\n- As it's a self-supervised pretraining method, the fact that experiments were only conducted on small datasets (CIFAR10, CIFAR100) leaves concerns about its applicability.\n\n[1] Mitrovic et al., REPRESENTATION LEARNING VIA INVARIANT CAUSAL MECHANISMS, ICLR 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your constructive comments! Please find our replies below.\n\n> 1. [Reply to W1] We argue that our theoretical analysis is **non-trivial**.\n\nDirectly applying adversarial data to paper [1] cannot obtain AIR.  It is because SIR [1] in Eq. (8) aims to enforce $p(y^R|x)$ to be style-independent; however, AIR in Eq. (7) aims to enforce $p(y^R|\\tilde{x})p(\\tilde{x}|x)$, instead of simply applying adversarial data into SIR (i.e., $p(y^R|\\tilde{x})$), to be style-independent. \n\nBesides, our theoretical analysis in Section 2.3 is unique to AIR. We explain AIR in Proposition 3 based on AIR's decomposition shown in Lemma 2, as well as analyze the generalization ability of the style-independence property of robust representations via AIR in Proposition 4. \n\n> 2. [Reply to W2] We explain inputs below and provide source code in Anonymized GitHub (link is in Abstract).\n\nAIR uses $p(y^R|\\tilde{x})p(\\tilde{x}|x)$ while SIR uses $p(y^R|x)$ under two different augmentations as the inputs. $p(y^R|\\tilde{x})$ or $p(\\tilde{x}|x)$ in Eq. (6) is calculated as a normalized representational distance (RD) between an original or augmented view of natural data and their adversarial variants by softmax function (SM). $p(y^R|x)$ in Eq. (8) is normalized RD between the original and augmented view of natural data by SM. \n\n> 3. [Reply to W3] In ***Table A*** of Rebuttal Highlights, we report the p-value obtained by conducting a Student's t-test to show that our method gains **significant** improvement.\n\n***Table A*** shows that the p-value is consistently much smaller than 0.05, which validates that the performance gain of our proposed method is significant.\n\n> 4. [Reply to W4] AIR is a simple plug-in that can help all existing ACL methods. We empirically find that the incorporation of SIR and AIR can obtain SOTA performance.\n\nThe embedding should be exempted from nuisance style factors for better transferability [1].  \nAIR is a useful plug-in to enhance ACL methods by regulating robust representations, and SIR regulates the standard representations. \nIncorporating AIR and SIR can regulate both standard and robust representations to be style-independent, thus improving both natural generalization and adversarial robustness in downstream tasks.\nExperiments in the main paper and Appendix C.5 validate AIR improved ACL, DynCAL, AdvCL [2], and A-InfoNCE [3] significantly. \n\n> 5. [Reply to W5] AIR and SIR help to find the style-invariant correlations among standard and robust representations across different distributions, which could enhance the robustness against common corruptions.\n\nProposition 4 in our paper and Theorem 1 in Paper [1] indicate that the style-independent property brought by AIR and SIR is generalizable to downstream tasks. Therefore, it means that AIR and SIR help to find the style-invariant correlations among standard and robust representations across different distributions. It is similar to the objective of invariant risk minimization [4] which can yield substantial improvement in robustness against perturbations incurred by distribution shifts. Therefore, IR could help enhance robustness against common corruption.\n\n> 6. [Reply to W6] We also conducted experiments on **STL10 datasets** in the main paper. Here, we provide the new results by conducting on **Imagenette datasets** which is a subset of ImageNet-1K.\n\nTo our knowledge, there is no reported result of ACL on large-scale datasets. We report new results of the performance evaluated on Imagenette in ***Table B.1*** and ***Table B.2*** of Rebuttal Highlights. The results validate that our method is still effective in sub-imagenet.\n\n> 7. [Reply to Q1] We intuitively explain $y^R$ in the main text (refer to Line 144) and a formal formulation of $y^R$ is shown in Eq. (10) in Appendix B.1. In the context of ACL, each augmented data point's proxy label $y^R$ refers to the index of its augmented variant in a minibatch. We will mention the formulation of $y^R$ in the main paper in revisions.\n\n> 8. [Reply to Q2] We formulate each step of the PGD attack following ACL.\n\nGiven an initial pair $(x^{i,(0)}, x^{j,(0)})$, PGD step $\\kappa \\in \\mathbb{N}$, step size $\\rho > 0$, adversarial budget $\\epsilon \\geq 0$, PGD iteratively update the pair from $t=0$ to $\\kappa-1$ as follows: $x^{u,(t+1)} =  \\Pi_{B_{\\epsilon}[x^{u,(0)}]} ( x^{u,(t)} +\\rho \\cdot \\mathrm{sign} (\\nabla_{x^{u,(t)}} \\ell_\\mathrm{CL}(x^{i,(t)}, x^{j,(t)})   )$, where $u \\in \\\\{ i,j\\\\}$ and $\\Pi_{B_{\\epsilon}[x^{u,(0)}]}$ projects the adversarial data into the epsilon ball around the initial point. At each step $t$, both $x^{i,(t)}$ and $x^{j,(t)}$ are updated. \n\n> 9. [Reply to Q3] We provide a comparison between DynACL with IR and supervised pre-training (SPT) including Madry and TRADES in ***Table C*** of Rebuttal Highlights.\n\n(1) *Speed*. DynACL with IR consumes more pre-training time compared with SPT.\n\n(2) *Performance*. DynACL with IR achieves worse performance on CIFAR-10 than SPT via SLF. \n\n(3) *Transferability*. DynACL with IR achieves better transferability to CIFAR-100 than SPT via SLF. \n\nDue to limited space, we provide a detailed analysis of these phenomena in ***Point 3*** of Rebuttal Highlights.\n\n> 10. [Reply to L1] Our proposed AIR is less sensitive to the hyper-parameter (HP) than SIR. We treat how to automatically schedule the HP as future work.\n\nWe provide an ablation study of HP in Table 7. In ***Point 4*** of Rebuttal Highlights, we provide a detailed analysis to show that AIR is less sensitive to HP than SIR. We would like to treat finding an automatic scheduler of HP as future work.\n\n*References* \n\n[1] Representation learning via invariant causal mechanisms. Mitrovic et al., ICLR 2021.\\\n[2] When does contrastive learning preserve adversarial robustness from pretraining to finetuning? Fan et al., NeurIPS 2021.\\\n[3] Adversarial contrastive learning via asymmetric infonce. Yu et al., ECCV 2022.\\\n[4] Invariant risk minimization. Arjovsky et al., 2020."}, {"review_id": "UmOVoudwXL", "reviewer": "Reviewer_ZaSM", "summary": "This paper proposes to tackle adversarial contrastive learning via causal reasoning. Specifically, the authors discuss the scenario where adversarial examples are involved in standard invariant regularization. The authors also provide analysis of proposed invariant regularization to justify the rationality. The experiments on different datasets show improvement compared with baselines.", "questions": "1. Please include the comparison with baseline in Table 4.\n2. Please include more evaluation on WideResNet.\n", "limitations": "The authors have discussed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. Taking adversarial example in the SIR via Markov condition seems natural and the analysis could provide some insights.\n3. The experiments on various datasets show improvement.\n", "weaknesses": "I have several concerns:\n\n1. The comparison with other baselines is missing in Table 4, such as ACL with SIR and DynACL with SIR.\n2. The authors conduct experiments on ResNet family. However, WideResNet backbone networks are always evaluated in the field of adversarial robustness. I wonder whether the superiority of IR can be generalized to different networks.\n\nMinors:\n\nIt seems a typo in Eq. 8, the KL divergence should be computed between $\\tau_i$ and $\\tau_j$.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your positive comments! Please find our replies below.\n\n> 1. [Reply to Q1] We demonstrate the results of the baseline as follows. We will update them in revision.\n\nThe results show that our proposed IR consistently performs better than the baseline.\n\n|Label ratio = 1\\%  | ACL | ACL with SIR| ACL with IR |\n|---|---|---|---|\n|AA (\\%) | 45.63 | 45.71 | 45.97 |\n|SA (\\%) | 74.84 | 74.99 | 76.63 |\n\n|Label ratio = 10\\%  | ACL | ACL with SIR| ACL with IR |\n|---|---|---|---|\n|AA (\\%) | 45.68 | 45.79 | 46.17 |\n|SA (\\%) | 76.30 | 76.67 | 77.52 |\n\n|Label ratio = 1\\%  | DynACL | DynACL with SIR| DynACL with IR |\n|---|---|---|---|\n|AA (\\%) | 45.78 | 45.93 | 46.25 |\n|SA (\\%) | 76.89 | 77.34 | 78.57 |\n\n|Label ratio = 10\\%  | DynACL | DynACL with SIR| DynACL with IR |\n|---|---|---|---|\n|AA (\\%) | 47.08 | 47.12 | 47.41 |\n|SA (\\%) | 78.22 | 78.84 | 79.94 |\n\n\n> 2. [Reply to Q2] We provide the results evaluated on the Wide ResNet with a depth 28 and width 10 (WRN-28-10) as follows.\n\nWe pre-trained WRN-28-10 via DynACL and DynACL with IR on CIFAR-10 and then evaluated the performance on CIFAR-10 via SLF. The results are shown in the following table. It validates that our method is effective in various models.\n\n| WRN-28-10 | SLF on CIFAR-10 | SLF on CIFAR-10 |\n|---|---|---|\n| Methods | AA (\\%) | SA (\\%) \n| DynACL | 41.13 | 71.23 |\n|DynACL with IR| 43.86 | 73.41 |\n\nNote that we set the batch size to 128 during pre-training due to the limitation of our GPU memory. We believe using a larger batch size during pre-training can further improve the performance of WRN-28-10 since [1].\n\n> 3. [Comment on typo] Thanks for pointing out this typo. We have corrected this typo in the revision.\n\n\n*References*\n\n[1] Chen, Ting, et al. \"A simple framework for contrastive learning of visual representations.\" International conference on machine learning. PMLR, 2020."}, {"review_id": "nKB90wCcJD", "reviewer": "Reviewer_paeS", "summary": "This paper proposes a novel adversarial contrastive learning method, which introduces causal reasoning method to obtain robust feature representation and enforce independence from style factors. The idea is simple and effective. In addition, the experiments are sufficient to prove the effectiveness of the proposed method. The source codes are released.", "questions": "Shown as Weaknesses.", "limitations": "Shown as Weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is interesting and easy to understand.\n2. The theoretical analysis is detailed and proves the effectiveness of the proposed method.\n", "weaknesses": "1. The improvement of the proposed method is limited.\n2. Without causal reasoning, this paper only proposes two distribution alignment regularizations in the adversarial samples to obtain robust feature representation. Thus, this paper is an incremental work based on ACL and DynACL.\n3. I want to see the performance of this paper in the large-scale datasets, such as sub-imagenet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your constructive comments! Please find our replies below.\n\n> 1. [Reply to W1] In ***Table A*** of Rebuttal Highlights, we report the p-value obtained by conducting a Student's t-test to show that our improvement is **significant**.\n\n***Table A*** shows that the p-value is consistently much smaller than 0.05, which validates that the performance gain of our proposed method is significant.\n\n> 2. [Reply to W2] We argue it is **non-trivial** to derive our proposed regularization term AIR. We use the causal graph of ACL to provide a unified framework.  Kindly note that our proposed regularization is a handy plug-in that can improve all existing ACL methods.\n\nWithout causal reasoning, we cannot build the causal graph of ACL. Further, without ACL's causal graph, we cannot obtain Eq. (4) according to the unique path $x \\rightarrow \\tilde{x} \\rightarrow y^R$ shown in ACL's causal graph. Besides, without causal reasoning of ACL's causal graph, we cannot propose the criterion shown in Eq. (5) and further propose AIR. Therefore, it is non-trivial to derive our proposed regularization term AIR.\n\nOur proposed method can complement all ACL-style pre-training methods such as ACL, DynACL, AdvCL [1], and A-InfoNCE [2], which is validated in the main paper and Appendix C.5. Therefore, our proposed method is a useful plug-in for all ACL-style pre-training methods.\n\n> 3. [Reply to W3] We provide the results of conducting DynACL with IR on **Imagenette datasets** which is a subset of ImageNet-1K.\n\nTo our knowledge, there is no reported result of ACL on large-scale datasets in existing papers. We report new results of the performance evaluated on Imagenette in ***Table B.1*** and ***Table B.2*** of Rebuttal Highlights. The results validate that our method is effective in sub-imagenet.\n\n*References* \n\n[1] When does contrastive learning preserve adversarial robustness from pretraining to finetuning? Fan et al., NeurIPS 2021.\\\n[2] Adversarial contrastive learning via asymmetric infonce. Yu et al., ECCV 2022."}, {"review_id": "gWkTOIzfqn", "reviewer": "Reviewer_Ai1a", "summary": "The authors propose AIR to regulate the the process of contrastive learning. They first analyze the causal graph of contrastive learning under the adversarial setting, and try to enforce  $p\\left(y^R | \\tilde x\\right)\\cdot p\\left(\\tilde x | x\\right)$ (Eq.4) to be invariant under different interventions. Intuitively, this regularizer would force the crafted adversarial examples to be style-independent and thus the learned representations shall enjoy better robustness. ", "questions": "See \"Weaknesses\" part.", "limitations": "N.A.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The core idea of this paper is technically novel and clearly presented.\n\nExperiments are reported by applying the proposed AIR / SIR terms to ACL / DynACL, and consistent performance gains are observed. Ablation studies for different values of $\\lambda_1$ and $\\lambda_2$ are also provided, to verify the effectiveness of the AIR term itself.\n\nThe authors also provide theoretical results to justify the rationality of the AIR term.", "weaknesses": "The proposed method would require more tunning since it incorporates two more hyper-parameters $\\lambda_1$ and $\\lambda_2$, and we can see from Table 7 that these two hyper-parameters interact in a non-trival way. For example, when compared with the case that only AIR is leveraged ($\\lambda_1=0,\\ \\lambda_2>0$), incorporating AIR with SIR (i.e., $\\lambda_1>0,\\ \\lambda_2>0$) could result in better **robust accuracy**.\n\nLittle type in Eq. 8: two augmentations in KL divergence should be different.\n\n--- Update ---\nAfter reading additional experimented provieded by the authors, I decide to raise my score to 8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your supportive comments! Please find our responses below.\n\n> 1. [Reply to W1] Our proposed AIR is less sensitive to the hyper-parameter (HP) than SIR. \n Automatic schedule of the HP could be a good extension to our work.\n\nWe provide an ablation study of HP in Table 7. In ***Point 4*** of Rebuttal Highlights, we provide a detailed analysis to show that AIR is less sensitive to HP than SIR. We would like to treat finding an automatic scheduler of HP as future work.\n\n> 2. [Comment on typo] Thanks for pointing out this typo. We have corrected this typo in the revision."}]

Author rebuttal:
[**Rebuttal Highlights**]

Many thanks for supportive comments from Reviewers **ZaSM** and **Ai1a**  as well as constructive comments from Reviewers **fVht** and **paeS**! 

Here, we would like to demonstrate extra empirical results to resolve your concerns.

> 1. [For Reviewers **fVht** and **paeS**] In ***Table A***, we used a Student's t-test test to show that our proposed method gains **significant** improvement across tasks and datasets.

***Table A**: We report the p-value obtained by conducting a t-test between the baseline and our proposed IR corresponding to the results reported in Table 1 of the main paper.*
| p-value  | AA on CIFAR-10 | SA on CIFAR-10 | AA on CIFAR-100 | SA on CIFAR-100 | AA on STL10 | SA on STL10 |
|---|---|---|---|---|---|---|
| ACL vs ACL with IR | 9.76e-05 | 9.75e-05 | 0.0196 | 4.46e-07 | 0.0049 | 2.22e-06 |
| DynACL vs DynACL with IR | 0.0019 | 4.48e-05 | 2.22e-05 | 8.16e-06 | 0.0009 | 4.69e-09 |

According to ***Table A***, we can find that the p-value is consistently much smaller than 0.05. Therefore, our proposed method achieves significant improvement over baseline methods.

> 2. [For Reviewers **fVht** and **paeS**] In ***Table B.1*** and ***Table B.2***, we run additional experiments and provide the results of conducting DynACL with IR on **Imagenette datasets** which is a subset of ImageNet-1K.

The results validate that our proposed method is still effective. 

***Table B.1**: The performance evaluated on Imagenette of ResNet-18.*
|  ResNet-18 | AA (\%) | SA (\%) |
|---|---|---|
| DynACL | 57.15 | 79.41 |
| DynACL with IR | 58.34 | 80.61 |

***Table B.2**: The performance evaluated on Imagenette of ResNet-50.*
|  ResNet-50 | AA (\%) | SA (\%) |
|---|---|---|
| DynACL | 58.98 | 80.74 |
| DynACL with IR | 60.10 | 81.66 |

To the best of our knowledge, there exits no reported result of ACL and DynACL on large-scale datasets in the existing papers. Therefore, we pre-trained ResNet-18 and ResNet-50 on Imagenette (https://github.com/fastai/imagenette) of $256 \times 256$ resolution which is a subset of ImageNet-1K using DynACL and DynACL with IR (ours), respectively. We compare and report the performance evaluated on Imagenette via SLF in ***Table B.1*** and ***Table B.2***. 

Note that, due to the limitation of our GPU memory, we set the batch size to 256 and 128 for pre-training ResNet-18 and ResNet-50, respectively. We believe using a larger batch size during pre-training can even further improve the performance [1].

> 3. [For Reviewer **fVht**] In ***Table C***, we provide a comparison between DynACL with IR and supervised pre-training (SPT) including Madry and TRADES in terms of performance, pre-training time, and transferability.

***Table C**: The comparison between Madry, TRADES, and DynACL with IR.*
|  |  | SLF on CIFAR-10 | SLF on CIFAR-10 | SLF on CIFAR-100 | SLF on CIFAR-100| 
|---|---|---|---|---|---|
| Pre-training on CIFAR-10 | Pre-training time (hours)| AA (\%) | SA (\%) | AA (\%) | SA (\%) | 
| Madry (need labels) | 3.6 | 46.20 | 83.79 | 4.45 | 23.47 |
| TRADES  (need labels) | 5.5 | 47.50 | 84.68 | 5.23 | 22.19 | 
| DynACL with IR  (no need labels)| 43.1 | 45.27 | 78.08 | 12.20 | 31.33 |

We pre-trained ResNet-18 on CIFAR-10 via Madry and TRADES. In ***Table C***, we report the standard accuracy (SA) and robust accuracy evaluated by AutoAttack (AA) on CIFAR-10 and CIFAR-100 after SLF.

(1) *Speed*. ACL-style pre-training methods do not acquire labels and thus need 1000 pre-training epochs (more time) to converge. Whereas, SPT using labels only needs 100 pre-training epochs to converge. Therefore, DynACL with IR consumes much more pre-training time compared with SPT.

(2) *Performance*. DynACL with IR achieves worse performance on CIFAR-10 than SPT via SLF. The reason could be that SPT can leverage the labels during training, thus making the feature extractor better overfit the training distribution of CIFAR-10 during pre-training. Therefore, SPT can perform better than DynACL with IR via SLF where the feature extractor is frozen.

(3) *Transferability*. DynACL with IR achieves better transferability to CIFAR-100 than SPT via SLF. Compared to DynACL with IR, SPT can use the labels and thus easily overfit the training distribution of the CIFAR-10 dataset, consequently performing worse in the generalization to other datasets such as the CIFAR-100 dataset.

> 4. [For Reviewers **fhVt** and **Ai1a**] Our proposed AIR is less sensitive to the hyper-parameter (HP) than SIR. 

We provide an ablation study of $\lambda_1$ and $\lambda_2$ in Table 7. We can observe that when $\lambda_1 = 0.5$ and $\lambda_2 > 0$ (only tunning the HP of AIR), the standard deviation (STD) of the natural and robust test accuracy is 0.08 and 0.13, respectively. When $\lambda_2 = 0.5$ and $\lambda_1 > 0$ (only tunning the HP of SIR), the STD of the natural and robust test accuracy is 0.24 and 0.27, respectively. Therefore, our proposed AIR is less sensitive to the HP than SIR since AIR yields a much smaller STD of results under different HPs than SIR.

*References*

[1] Chen, Ting, et al. "A simple framework for contrastive learning of visual representations." International conference on machine learning. PMLR, 2020.




Meta-review: The reviewers acknowledge the technical novelty of the paper, which introduces an advancement in adversarial contrastive learning using causal reasoning. The proposed adversarial invariant regularization (AIR) method is commended for demonstrating a style factor and its empirical effectiveness on CIFAR10 and CIFAR100 datasets. The reviewer appreciates the authors' thorough explanation of their decision to perform multiple runs during the finetuning phase, clarifying that it aims to eliminate the impact of randomness on downstream task performance and validate the robustness of their approach. The design of the regularizers is clarified by explaining the integration of style-independent regularization (SIR) and AIR to regulate both standard and robust representations, and comparisons with ACL and DynACL are outlined. Theoretical justifications, consistent performance gains through experiments, and ablation studies on the proposed AIR term's effectiveness contribute to the paper's strength.
Decision: Accept (poster)