Title: Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularization
Venue: NeurIPS 2023
Accepted: yes
Keywords: robust pre-training

adversarial contrastive learning
Abstract: Adversarial contrastive learning (ACL) is a technique that enhances standard contrastive learning (SCL) by incorporating adversarial data to learn a robust representation that can withstand adversarial attacks and common corruptions without requiring costly annotations. To improve transferability, the existing work introduced the standard invariant regularization (SIR) to impose style-independence property to SCL, which can exempt the impact of nuisance style factors in the standard representation. However, it is unclear how the style-independence property benefits ACL-learned robust representations. In this paper, we leverage the technique of causal reasoning to interpret the ACL and propose adversarial invariant regularization (AIR) to enforce independence from style factors. We regulate the ACL using both SIR and AIR to output the robust representation. Theoretically, we show that AIR implicitly encourages the representational distance between different views of natural data and their adversarial variants to be independent of style factors. Empirically, our experimental results show that invariant regularization significantly improves the performance of state-of-the-art ACL methods in terms of both standard generalization and robustness on downstream tasks. To the best of our knowledge, we are the first to apply causal reasoning to interpret ACL and develop AIR for enhancing ACL-learned robust representations. Our source code is at https://github.com/GodXuxilie/Enhancing_ACL_via_AIR.

Enhancing Adversarial Contrastive Learning via Adversarial Invariant Regularization

Abstract

Adversarial contrastive learning (ACL) is a technique that enhances standard contrastive learning (SCL) by incorporating adversarial data to learn a robust representation that can withstand adversarial attacks and common corruptions without requiring costly annotations. To improve transferability, the existing work introduced the standard invariant regularization (SIR) to impose style-independence property to SCL, which can exempt the impact of nuisance style factors in the standard representation. However, it is unclear how the style-independence property benefits ACL-learned robust representations. In this paper, we leverage the technique of causal reasoning to interpret the ACL and propose adversarial invariant regularization (AIR) to enforce independence from style factors. We regulate the ACL using both SIR and AIR to output the robust representation. Theoretically, we show that AIR implicitly encourages the representational distance between different views of natural data and their adversarial variants to be independent of style factors. Empirically, our experimental results show that invariant regularization significantly improves the performance of state-of-the-art ACL methods in terms of both standard generalization and robustness on downstream tasks. To the best of our knowledge, we are the first to apply causal reasoning to interpret ACL and develop AIR for enhancing ACL-learned robust representations. Our source code is at https://github.com/GodXuxilie/Enhancing_ACL_via_AIR.

1 Introduction

The attention towards pre-trained models that can be easily finetuned for various downstream applications has significantly increased recently [17, 18, 19, 41]. Notably, foundation models [4] via self-supervision on large-scale unlabeled data, such as GPT [5] and CLAP [20], can be adapted to a wide range of downstream tasks. Due to the high cost of annotating large-scale data, unsupervised learning techniques [21, 34, 49, 16] are commonly used to obtain generalizable representations, in which standard contrastive learning (SCL) has been shown as the most effective way [9, 38, 10].

Adversarial contrastive learning (ACL) [30, 29], that incorporates adversarial data [37] with SCL [9], can yield robust representations that are both generalizable and robust against adversarial attacks [23]

4

𝑥

𝑦

!

θ

𝑠

𝑐

𝑦

"

𝑦

#

Data generation

procedure

Learning procedure

𝑥

𝑦

!

θ

𝑠

𝑐

𝑦

"

𝑦

#

& 𝑥

⋮

⋮

Standard contrastive learning

Adversarial contrastive learning

and common corruptions [26]. ACL is attracting increasing popularity since the adversarial robustness of the pre-trained models is essential to safety-critical applications [6, 33]. Many studies have tried to improve ACL from various perspectives including increasing contrastive views and leveraging pseudo labels [22], leveraging hard negative sampling [42, 50], and dynamically scheduling the strength of data augmentations [36].

The style-independence property of learned representations, which eliminates the effects of nuisance style factors in SCL, has been shown to improve the transferability of representations [38]. To achieve style-independence property in learned representations, Mitrovic et al. [38] proposed a standard invariant regularization (SIR) that uses causal reasoning [39, 40] to enforce the representations of natural data to be invariant to style factors. SIR has been shown effective in enhancing representation' transferability. However, it remains unclear in the literature how to achieve style independence in robust representations learned via ACL, especially when adversarial data is used during pre-training.

Therefore, we leverage the technique of causal reasoning to enforce robust representations learned via ACL to be style-independent. As shown in the right panel of Figure 1, we construct the causal graph of ACL (details refer to Section 3.1). Different from the causal graph of SCL [38], ACL has an extra path x → ˜ x since ACL will generate the adversarial data ˜ x given the unlabeled data x (i.e., x → ˜ x ). Then, ACL learns representations by maximizing both the probability of the proxy label y R given the natural data and that given the adversarial data (i.e., x → y R and ˜ x → y R ). Theorem 1 shows that maximizing the aforementioned probability in the causal view is equivalent to the learning objective of ACL [29], which justifies the rationality of our constructed causal graph of ACL.

To enforce robust representations to be style-independent, we propose an adversarial invariant regularization (AIR). Specifically, AIR (see Eq. (7)) aims to penalize the Kullback-Leibler divergence between the robust representations of the unlabeled data augmented via two different data augmentation functions during the learning procedure of x → ˜ x → y R . Note that SIR [38] is a special case of AIR in the standard context where x = ˜ x . Then, we propose to learn robust representations by minimizing the adversarial contrastive loss [29, 22, 50, 36] together with a weighted sum of SIR and AIR and show the learning algorithm in Algorithm 1.

Furthermore, we give a theoretical understanding of the proposed AIR and show that AIR is beneficial to the robustness against corruptions on downstream tasks. Based on the decomposition of AIR shown in Lemma 2, we propose Proposition 3 which explains that AIR implicitly enforces the representational distance between original as well as augmented views of natural data and their adversarial variants to be independent of the style factors. In addition, we theoretically show that the style-independence property of robust representations learned via ACL will still hold on the downstream classification tasks in Proposition 4, which could be helpful to improve robustness against input perturbations on downstream tasks [3, 45, 28, 38].

Empirically, we conducted comprehensive experiments on various datasets including CIFAR-10 [31], CIFAR-100 [31], STL-10 [12], CIFAR-10-C [26], and CIFAR-100-C [26] to show the effectiveness of our proposed method in improving ACL methods [29, 22, 50, 36]. We demonstrate that our proposed method can achieve the new state-of-the-art (SOTA) performance on various downstream tasks by significantly enhancing standard generalization and robustness against adversarial attacks [13, 14, 2] and common corruptions [26]. Notably, compared with the prior SOTA method DynACL [36], AIR improves the standard and robust test accuracy by 2.71% (from 69.59 ± 0.08% to 72.30 ± 0.10%) and 1.17% (from 46.49 ± 0.05% to 47.66 ± 0.06%) on the STL-10 task and increases the test accuracy under common corruptions by 1.15% (from 65.60 ± 0.18% to 66.75 ± 0.10%) on CIFAR-10-C.

2 Related Works and Preliminaries

Here, we introduce the related works in ACL and causal reasoning and provide the preliminaries.

2.1 Related Works

Adversarial contrastive learning (ACL). Contrastive learning (CL) is frequently used to leverage large unlabeled datasets for learning useful representations. Chen et al. [9] presented SimCLR that leverages contrastive loss for learning useful representations and achieved significantly improved standard test accuracy on downstream tasks. Recently, adversarial contrastive learning (ACL) [30, 29, 27, 22, 50, 52, 36] has become the most effective unsupervised approaches to learn robust representations. Jiang et al. [29], which is the seminal work of ACL that incorporates adversarial data [37] with contrastive loss [9], showed that ACL can exhibit better robustness against adversarial attacks [23, 13] and common corruptions [26] on downstream tasks compared with SCL [9].

One research line focuses on improving the performance of ACL. AdvCL [22] leverages a third contrastive view for high-frequency components and pseudo labels generated by clustering to improve ACL. Yu et al. [50] proposed the asymmetric InfoNCE objective (A-InfoNCE) that utilizes the hard negative sampling method [42] to further enhance AdvCL [22]. Recently, DynACL [36] dynamically schedules the strength of data augmentations and correspondingly schedules the weight for standard and adversarial contrastive losses, achieving the SOTA performance among existing ACL methods. Another research line focuses on improving the efficiency of ACL since ACL requires tremendous running time to generate the adversarial training data during pre-training. Xu et al. [48] proposed a robustness-aware coreset selection (RCS) method to speed up ACL by decreasing the number of training data. Our paper, belonging to the former research line, leveraged the technique of causal reasoning to further enhance ACL and achieved new SOTA results. We treat the incorporation of our proposed method with RCS [48] as future work.

Causal reasoning. Causal reasoning [39, 53, 40, 8, 56, 38, 7, 44, 43] has been widely applied to machine learning to identify causal relations and ignore nuisance factors by intervention. In particular, Zhang et al. [53, 56] investigated the causality in supervised adversarial training [37, 55] where label information is required and proposed the regularizer to eliminate the difference between the natural and adversarial distributions. Mitrovic et al. [38] built the causal graph of SCL in the standard context and introduced the standard invariant regularization (SIR) which aims to enforce the representation of unlabeled natural data to be invariant of the data augmentations. SIR can significantly improve robustness transferability to downstream tasks in terms of common corruptions [26], which is empirically validated by Mitrovic et al. [38].

2.2 Preliminaries

Standard contrastive learning (SCL) [9]. Let h θ : X → Z be a representation extractor parameterized by θ , g : Z → V be a projection head that maps representations to the space where the contrastive loss is applied, and τ i , τ j : X → X be two transformation operations randomly sampled from a pre-defined transformation set T . Given a minibatch B ∼ X β consisting of β original samples, we denote the augmented minibatch B u = { τ u ( x k ) | ∀ x k ∈ B } via augmentation function τ u ( · ) . We take f θ ( · ) = g ◦ h θ ( · ) and x u k = τ u ( x k ) for any x k ∼ X and u ∈ { i, j } . Given a positive pair ( x i k , x j k ) , the standard contrastive loss proposed by SimCLR [9] is formulated as follows:

where sim( p, q ) = p ⊤ q/ ∥ p ∥∥ q ∥ is the cosine similarity function and t > 0 is a temperature parameter.

ACL [29] and DynACL [36]. Let ( X , d ∞ ) be the input space X with the infinity distance metric d ∞ ( x, x ′ ) = ∥ x -x ′ ∥ ∞ , and B ϵ [ x ] = { x ′ ∈ X | d ∞ ( x, x ′ ) ≤ ϵ } be the closed ball of radius ϵ > 0 centered at x ∈ X . Given a data point x k ∈ X , the adversarial contrastive loss is as follows:

in which ω ∈ [0 , 1] is a hyperparameter and ˜ x i k and ˜ x j k are adversarial data generated via projected gradient descent (PGD) [37, 54, 55] within the ϵ -balls centered at x i k and x j k . Note that ACL [29] fixes ω = 0 while DynACL [36] dynamically schedules ω according to its dynamic augmentation scheduler that gradually anneals from a strong augmentation to a weak one. We leave the details of the data augmentation scheduler [36] in Appendix C due to the page limitation.

3 Methodology

In this section, we first create the causal graph for ACL which is the fundamental premise for causal reasoning. Then, we introduce adversarial invariant regularization (AIR) according to the causal understanding in the context of ACL and the learning algorithm of our proposed method. Lastly, we demonstrate the theoretical understanding of AIR and theoretically show that the style-independence property is generalizable to downstream tasks.

3.1 Adversarial Contrastive Learning (ACL) in the View of Causality

A causal graph is arguably the fundamental premise for causal reasoning [39, 40]. Previously, Mitrovic et al. [38] constructed the causal graph of SCL as shown in the left panel of Figure 1. However, how to build causal graphs of ACL is still unclear in the literature. Therefore, we take the primitive step to construct the causal graph of ACL.

To begin with, we formalize the data generation procedure in the causal graph. Let x ∈ X denote an unlabeled data point and Y = { y t } T t =1 denote a set of labels in an unknown downstream task. Then, similar to Mitrovic et al. [38] and Zhang et al. [56] we make the following assumptions: 1) the data point x is generated from the content factor c and the style factor s , i.e., c → x ← s , 2) the content is independent of the style, i.e., c ⊥ ⊥ s , and 3) only the content, which is the original image data, is related to the downstream task, i.e., c → y t .

Then, we build the learning procedure of CL. According to the above assumptions, the content serves an essential role in downstream classification tasks. Therefore, CL can be regarded as estimating the probability of the label given the content, i.e., p ( y t | c ) . To learn the representations from the unlabeled data x , CL needs to maximize the conditional probability p ( y t | x ) .

However, in practice, the label y t is unknown. Thanks to the causal concept of refinement [8, 38], we can construct a proxy label y R ∈ Y R which is a refinement of Y . This proxy label y R should be consistent across different views of the unlabeled data x , which enables learning the representations. Intuitively, the proxy label y R can be viewed as a more fine-grained variant of the original label y t . For example, a refinement for recognizing cars would be the task of classifying various types of cars. Therefore, SCL targets maximizing the probability of the proxy label given the unlabeled natural data, i.e., p ( y R | x ) [38].

Besides x → y R , ACL has an extra path x → ˜ x → y R since ACL needs to first generate the adversarial data (Eq. (3)) and then learns representations from the unlabeled natural and adversarial data (Eq. (2)). According to Eq. (3), we observe that ACL only uses the observed unlabeled data x for generating adversarial data, i.e., x → ˜ x ← θ . Then, according to the paths x → y R and ˜ x → y R , from the causal view, ACL learns representations by maximizing both the probability of the proxy label y R given the natural data (i.e., p ( y R | x ) ) and that given adversarial data (i.e., p ( y R | ˜ x ) ).

Lastly, we provide Theorem 1 to justify the rationality of our constructed causal graph of ACL.

Theorem 1. From the causal view, in ACL, maximizing the conditional probability both p ( y R | x ) and p ( y R | ˜ x ) is equivalent to minimizing the learning objective of ACL [29] that is the sum of standard and adversarial contrastive losses.

The proof of Theorem 1 is in Appendix B.1.

3.2 Adversarial Invariant Regularization

According to the independence of causal mechanisms [40], performing interventions on the style variable s should not change the probability of the proxy label given the unlabeled data, i.e., p ( y R | x ) . According to the path x → ˜ x → y R shown in the causal graph of ACL, we can obtain that

under the assumption that the process of x → ˜ x → y R satisfies the Markov condition. Therefore, we should make the conditional probability p ( y R | ˜ x ) p (˜ x | x ) that is learned via ACL become styleindependent. It guides us to propose the following criterion that should be satisfied by ACL:

where do ( τ ) as the short form of do ( s = τ ) denotes that we perform the intervention on s via data augmentation function τ ( · ) . We leverage the representational distance between various views of natural data and their adversarial variants normalized by the softmax function to calculate the conditional probability p do ( τ u ) ( y R | ˜ x ) and p do ( τ u ) (˜ x | x ) where u ∈ { i, j } as follows:

in which x ∈ B ∼ X β is the original view of natural data, x u is the augmented view of natural data via augmentation τ u ( · ) , ˜ x u is the adversarial variant generated via Eq. (3), and t is the temperature parameter. Note that we use x as the approximated surrogate of its true content c that incurs y R (i.e., c → y R ).

To achieve the above criterion in Eq. (5), we propose an adversarial invariant regularization (AIR) to regulate robust representations as follows:

in which ϵ ≥ 0 is the adversarial budget and KL( p ( x ) ∥ q ( x ); B ) = ∑ x ∈ B p ( x ) log p ( x ) q ( x ) denotes the Kullback-Leibler (KL) divergence. AIR can enforce the representation to satisfy the criterion in Eq. (5), thus eliminating the effect of the style factors on the representation. Therefore, when setting ϵ as a positive constant, L AIR ( B ; θ, ϵ ) can effectively regulate robust representations of adversarial data to be style-independent.

Besides, to explicitly regulate standard representations of natural data to be independent of style factors, we can simply set ϵ = 0 of AIR. We formulate AIR with ϵ = 0 as follows:

Note that L AIR ( B ; θ, 0) is the same formulation as the standard invariant regularization (SIR) [38], which can make the standard representation style-independent.

By incorporating adversarial contrastive loss with AIR, the learning objective function of our proposed method is shown as follows:

where U ∼ X N refers to an unlabeled dataset consisting of N samples, ϵ > 0 is the adversarial budget, and λ 1 ≥ 0 and λ 2 ≥ 0 are two hyperparameters. The learning algorithm of ACL with AIR is demonstrated in Algorithm 1. Note that our proposed AIR is compatible with various learning objectives such as ACL [29], AdvCL [22], A-InfoNCE [50], and DynACL [36].

Algorithm 1 ACL with Adversarial Invariant Regularization (AIR)

1: Input: Unlabeled training set U , total training epochs E , learning rate η , batch size β , adversarial budget ϵ > 0 , hyperparameters λ 1 and λ 2 2: Output: Pre-trained representation extractor h θ 3: Initialize parameters of model f θ = g ◦ h θ 4: for e = 0 to E -1 do 5: for batch m = 1 , . . . , ⌈| U | /β ⌉ do 6: Sample a minibatch B m from U 7: Update θ ← θ -η · ∇ θ ∑ x k ∈ B m ℓ ACL ( x k ; θ ) + λ 1 · L AIR ( B m ; θ, 0) + λ 2 · L AIR ( B m ; θ, ϵ ) 8: end for 9: end for

3.3 Theoretical Analysis

We provide a theoretical understanding of AIR in Proposition 3 based on the decomposition of AIR shown in Lemma 2. We set ϵ > 0 of AIR by default in this section.

Lemma 2 (Decomposition of AIR) . AIR in Eq. (7) can be decomposed into two terms as follows:

where E x ∼ Q 3 ( x ) [KL( Q 1 ( x ) ∥ Q 2 ( x ))] is the expectation of KL divergence over Q 3 and Q 1 , Q 2 , Q 3 are probability distributions.

The proof of Lemma 2 is in Appendix B.2. Based on Lemma 2, we propose Proposition 3 that provides the theoretical understanding of AIR.

Proposition 3. AIR implicitly enforces the robust representation to satisfy the following two proxy criteria:

The proof of Proposition 3 is in Appendix B.3.

Remarks. Proposition 3 explains that AIR implicitly enforces the representational distance to be style-independent between the original view of natural data and their adversarial variants (i.e., p do ( τ i ) ( y R | ˜ x ) = p do ( τ j ) ( y R | ˜ x ) ), as well as between the augmented view of natural data and their adversarial counterparts (i.e., p do ( τ i ) (˜ x | x ) = p do ( τ j ) (˜ x | x ) ). Therefore, Proposition 3 explicitly presents that AIR is different from SIR [38], where SIR only enforces the augmented views of natural data to be style-independent.

Next, we theoretically show that on the assumption that Y R is a refinement of Y , the styleindependence property of robust representations will hold on downstream classification tasks in Proposition 4.

Proposition 4. Let Y = { y t } T t =1 be a label set of a downstream classification task, Y R be a refinement of Y , and ˜ x t be the adversarial data generated on the downstream task. Assuming that ˜ x ∈ B ϵ [ x ] and ˜ x t ∈ B ϵ [ x ] , we have the following results:

The proof of Proposition 4 is shown in Appendix B.4.

Remarks. Proposition 4 indicates that the robust representation of data on downstream tasks is still style-independent. Mitrovic et al. [38] empirically showed that the style-independence property of standard representations can significantly improve the robustness against common corruptions [26] on downstream tasks. In addition, Proposition 4 indicates that AIR helps to find the invariant correlations that are independent of the style factors among different training distributions. It is similar to the objective of invariant risk minimization [3] that can yield substantial improvement in the robustness against various corruptions incurred by distribution shifts. Therefore, the style-independence property of robust representations learned via ACL perhaps helps to enhance the robustness against input perturbations on downstream tasks.

Table 2: Self-task common-corruption robustness transferability. The corruption severity (CS) ranging from { 1 , 3 , 5 } (denoted as 'CS-{1,3,5}'). The standard deviation is in Table 15.

4 Experiments

In this section, we demonstrate the effectiveness of our proposed AIR in improving ACL [29] and its variants [36, 22, 50] on various datasets including CIFAR-10 [31], CIFAR-100 [31], STL-10 [12], CIFAR-10-C [26], and CIFAR-100-C [26]. Extensive experimental details are shown in Appendix C.

Pre-training. In the main paper, we demonstrate the results of applying our proposed AIR with λ 1 = 0 . 5 and λ 2 = 0 . 5 to ACL [29] and DynACL [36] (denoted as 'ACL-AIR' and 'DynACL-AIR', respectively). We utilized ResNet-18 [25] as the representation extractor following previous selfsupervised adversarial training methods [29, 22, 36, 50]. We adopted the same training configuration of ACL [29] using SGD for 1000 epochs with an initial learning rate of 5.0 and a cosine annealing schedule [35]. The batch size β is fixed as 512. The adversarial budget ϵ is set as 8 / 255 . In the context of DynACL, we took the same data augmentation scheduler and the same scheduler of the hyperparameter ω as the setting in the original paper of DynACL [36]. Note that, to reproduce the results of baseline methods, we downloaded the pre-trained weights of ACL [29] on CIFAR-10/CIFAR-100 and DynACL [36] on CIFAR-10/CIFAR-100/STL-10 from their official GitHub as the pre-trained representation extractor. We provide the extensive ablation studies on the hyper-parameters ( λ 1 and λ 2 ) in Appendix C.2, the ACL variants (including AdvCL [22] and A-InfoNCE [50]) in Appendix C.3, and the backbone models (including ResNet-50 and Wide ResNet [51]) in Appendix C.4.

Finetuning procedures. We adopted the following three types of finetuning procedures: standard linear finetuning (SLF), adversarial linear finetuning (ALF), and adversarial full finetuning (AFF), to evaluate the learned representations. The former two finetuning procedures freeze the learned representation extractor and only train the linear classifier using natural or adversarial samples, respectively. We took the pre-trained representation extractor as weight initialization and trained the whole model using the adversarial data during AFF. The training configuration of finetuning (e.g., the finetuning epoch and optimizer) exactly follows DynACL [36]. Specifically, we used the official code provided in DynACL [36]'s GitHub for finetuning and illustrate the experimental settings of finetuning in Appendix C as well. For each pre-trained representation extractor, we repeated the finetuning experiments 3 times and report the median results and the standard deviation.

Evaluation protocols. We let 'AA' denote the robust test accuracy under AutoAttack (AA) [13] and 'SA' stand for the standard test accuracy. To evaluate the robustness against common corruption, we report the mean test accuracy under 15 types of common corruptions [26] with the corruption

̸

Table 4: Cross-task common-corruption robustness transferability. The corruption severity (CS) ranging from { 1 , 3 , 5 } (denoted as 'CS-{1,3,5}'). The standard deviation is in Table 16.

severity (CS) ranging from { 1 , 3 , 5 } (denoted as 'CS-{ 1 , 3 , 5 } '). Specifically, we used the official code of AutoAttack [13] and RobustBench [15] for implementing evaluations. In Appendix C.5, we provide robustness evaluation under more diverse attacks [13, 14, 2]. In Appendix C.6, we provide the test accuracy under each type of common corruption [26].

4.1 Evaluation of Self-Task Robustness Transferability

Self-task adversarial robustness transferability. Table 1 reports the self-task adversarial robustness transferability evaluated on three datasets where pre-training and finetuning are conducted on the same datasets including CIFAR-10, CIFAR-100, and STL-10. In Appendix C.1, we report the self-task robustness transferability evaluated in the semi-supervised settings. Table 1 demonstrates that our proposed AIR can obtain new state-of-the-art (SOTA) both standard and robust test accuracy compared with existing ACL methods [29, 36]. Therefore, It validates the effectiveness of our proposed AIR in consistently improving the self-task adversarial robustness transferability against adversarial attacks as well as standard generalization in downstream datasets via various finetuning procedures. Notably, compared with the previous SOTA method DynACL [36], DynACL-AIR increases standard test accuracy by 4.52% (from 72.90% to 77.42%) on the CIFAR-10 dataset via ALF, 2.05% (from 43.58% to 45.63%) on the CIFAR-100 dataset via ALF, and 1.17% (from 46.49% to 47.66%) on the STL-10 dataset via SLF.

Self-task common-corruption robustness transferability. Table 2 reports the self-task commoncorruption [26] robustness transferability. Specifically, we conducted pre-training on CIFAR10/CIFAR-100 and then evaluated the test accuracy on CIFAR-10-C/CIFAR-100-C with various corruption severities after various finetuning procedures. Note that the test accuracy under each type of common corruption is reported in Appendix C.6. Table 2 demonstrates that AIR leads to consistent and significant improvement in the robustness against common corruption. In addition, we observe that ACL always achieves much higher test accuracy under common corruptions than DynACL after SLF. We conjecture the reason is that DynACL uses weak data augmentations at the later phase of training, which makes DynACL more likely to overfit the training distribution of CIFAR-10 and thus leads to worse performance under common corruptions via SLF.

4.2 Evaluation of Cross-Task Robustness Transferability

Cross-task adversarial robustness transferability. Table 3 shows the cross-task adversarial robustness transferability where pre-training and finetuning are conducted on the different datasets. We can observe that AIR substantially improves both ACL's and DynACL's robustness against adversarial attacks [13] and standard generalization to other downstream datasets. Particularly, AIR improves the standard and robust test accuracy of ACL via ALF by 7.52% (from 28.58% to 36.10%) and 2.21% (from 11.09% to 13.30%), respectively.

Cross-task common-corruption robustness transferability. Table 4 reports the cross-task common-corruption [26] robustness transferability where pre-training is conducted on CIFAR10/CIFAR-100 and finetuning is conducted on CIFAR-100-C/CIFAR-10-C [26]. The empirical results show that our proposed AIR can consistently improve the accuracy under common corruptions with various severity, which validates the effectiveness of AIR in enhancing the robustness transferability against common corruptions.

4.3 Evaluation of Pre-Trained Models after Post-Processing

Here, we report the self-task adversarial robustness transferability of pre-trained models after postprocessing in Table 5. The post-processing method is linear probing and then adversarial full finetuning (LP-AFF) [32] which first generates pseudo labels via clustering and then further adversarially trains the pre-trained model using the pseudo labels. '++' denotes that the pre-trained model is post-processed via LP-AFF. We provide the comparison between DynACL++ and DynACL-AIR++ in Table 5. Note that the superscript † denotes that the results of DynACL++ are copied from Luo et al. [36]. The results validate that AIR is compatible with the trick of post-processing via LP-AFF [32], which can yield improved performance on various downstream datasets.

4.4 Evaluation on High-Resolution Imagenette Dataset

To the best of our knowledge, there exists no reported result of ACL and DynACL on high-resolution datasets in the existing papers. Therefore, we pre-trained ResNet-18 and ResNet-50 on Imagenette 3 of resolution 256 × 256 which is a subset of ImageNet-1K using DynACL and DynACL-AIR, respectively. We compare and report the performance evaluated on Imagenette via SLF in Table 6. Our results validate that our proposed AIR is effective in enhancing the performance on highresolution datasets. Note that, due to the limitation of our GPU memory, we set the batch size to 256 and 128 for pre-training ResNet-18 and ResNet-50 on Imagenette, respectively. We believe using a larger batch size during pre-training can even further improve the performance [9].

4.5 Evaluation via Automated Robust Fine-Tuning

While evaluating the transferability of a pre-trained model, we observed that the performance on the downstream dataset is sensitive to the hyper-parameters (e.g., the learning rate) of the finetuning

̸

procedure. It would require extensive computational resources to search for appropriate hyperparameters to achieve a satisfactory performance. To mitigate this issue, we leverage an automated robust finetuning framework called AutoLoRa [47], which can automatically schedule the learning rate and set appropriate hyper-parameters during finetuning. We report the performance achieved by AutoLoRa to further justify the SOTA performance by our proposed AIR in Table 7.

Table 7 shows that AutoLoRa can further enhance the performance of a pre-trained model on the downstream tasks without searching for appropriate hyper-parameters since the value of 'Diff' is consistently larger than 0 . 0 . Besides, Table 7 justifies that our proposed AIR is effective in enhancing robustness transferability via various finetuning methods.

5 Conclusions

This paper leveraged the technique of causal reasoning to interpret ACL and proposed adversarial invariant regularization (AIR) to enhance ACL. AIR can enforce the learned robust representations to be invariant of the style factors. We improved ACL by incorporating the adversarial contrastive loss with a weighted sum of AIR and SIR that is an extension of AIR in the standard context. Theoretically, we showed that AIR implicitly encourages the representational distance between different views of natural data and their adversarial variants to be independent of style factors. Empirically, comprehensive results validate that our proposed method can achieve new state-of-theart performance in terms of standard generalization and robustness against adversarial attacks and common corruptions on downstream tasks.

References

Jean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? Advances in Neural Information Processing Systems , 32, 2019.

Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-efficient black-box adversarial attack via random search. In Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII , pages 484-501. Springer, 2020.

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Varun H Buch, Irfan Ahmed, and Mahiben Maruthappu. Artificial intelligence in medicine: current trends and future possibilities. British Journal of General Practice , 68(668):143-144, 2018.

Peter Bühlmann. Invariance, causality and robustness. 2020.

Krzysztof Chalupka, Pietro Perona, and Frederick Eberhardt. Visual causal feature learning. arXiv preprint arXiv:1412.2309 , 2014.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning , pages 1597-1607. PMLR, 2020.

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems , 33:22243-22255, 2020.

Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9640-9649, 2021.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 215-223. JMLR Workshop and Conference Proceedings, 2011.

Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning , pages 2206-2216. PMLR, 2020.

Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with a fast adaptive boundary attack. In International Conference on Machine Learning , pages 2196-2205. PMLR, 2020.

Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670 , 2020.

Zhiyuan Dang, Cheng Deng, Xu Yang, Kun Wei, and Heng Huang. Nearest neighbor matching for deep clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13693-13702, 2021.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Jiahua Dong, Yang Cong, Gan Sun, Bineng Zhong, and Xiaowei Xu. What can be transferred: Unsupervised domain adaptation for endoscopic lesions segmentation. In CVPR , 2020.

Jiahua Dong, Yang Cong, Gan Sun, Zhen Fang, and Zhengming Ding. Where and how to transfer: Knowledge aggregation-induced transferability perception for unsupervised domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 1(2):1-18, 2021. doi: 10.1109/TPAMI.2021.3128560.

Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap: Learning audio concepts from natural language supervision. arXiv preprint arXiv:2206.04769 , 2022.

Dumitru Erhan, Aaron Courville, Yoshua Bengio, and Pascal Vincent. Why does unsupervised pre-training help deep learning? In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 201-208. JMLR Workshop and Conference Proceedings, 2010.

Lijie Fan, Sijia Liu, Pin-Yu Chen, Gaoyuan Zhang, and Chuang Gan. When does contrastive learning preserve adversarial robustness from pretraining to finetuning? Advances in Neural Information Processing Systems , 34:21480-21492, 2021.

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Yoshua Bengio and Yann LeCun, editors, The Third International Conference on Learning Representations , 2015.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems , 33:21271-21284, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations , 2019.

Chih-Hui Ho and Nuno Nvasconcelos. Contrastive learning with adversarial examples. Advances in Neural Information Processing Systems , 33:17081-17093, 2020.

Tianjian Huang, Shaunak Ashish Halbe, Chinnadhurai Sankar, Pooyan Amini, Satwik Kottur, Alborz Geramifard, Meisam Razaviyayn, and Ahmad Beirami. Robustness through data augmentation loss consistency. Transactions on Machine Learning Research .

Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. Advances in Neural Information Processing Systems , 33:16199-16210, 2020.

Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning. Advances in Neural Information Processing Systems , 33:2983-2994, 2020.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.

Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. In The Tenth International Conference on Learning Representations , 2022. URL https://openreview. net/forum?id=UYneFzXSJWh .

Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security , pages 99-112. Chapman and Hall/CRC, 2018.

Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. IEEE Access , 8:193907-193934, 2020.

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 , 2016.

Rundong Luo, Yifei Wang, and Yisen Wang. Rethinking the effect of data augmentation in adversarial contrastive learning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=0qmwFNJyxCL .

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR , 2018.

Jovana Mitrovic, Brian McWilliams, Jacob C Walker, Lars Holger Buesing, and Charles Blundell. Representation learning via invariant causal mechanisms. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id= 9p2ekP904Rs .

Judea Pearl. Causality . Cambridge university press, 2009.

Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms . The MIT Press, 2017.

Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972 , 2021.

Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592 , 2020.

Axel Sauer and Andreas Geiger. Counterfactual generative networks. arXiv preprint arXiv:2101.06046 , 2021.

Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. Advances in Neural Information Processing Systems , 33:1513-1524, 2020.

Ruoyu Wang, Mingyang Yi, Zhitang Chen, and Shengyu Zhu. Out-of-distribution generalization with causal invariant transformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 375-385, 2022.

Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoc V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 819-828, 2020.

Xilie Xu, Jingfeng Zhang, and Mohan Kankanhalli. Autolora: A parameter-free automated robust fine-tuning framework. arXiv preprint arXiv:2310.01818 , 2023.

Xilie Xu, Jingfeng Zhang, Feng Liu, Masashi Sugiyama, and Mohan Kankanhalli. Efficient adversarial contrastive learning via robustness-aware coreset selection. In NeurIPS , 2023.

Xu Yang, Cheng Deng, Kun Wei, Junchi Yan, and Wei Liu. Adversarial learning for robust deep clustering. Advances in Neural Information Processing Systems , 33:9098-9108, 2020.

Qiying Yu, Jieming Lou, Xianyuan Zhan, Qizhang Li, Wangmeng Zuo, Yang Liu, and Jingjing Liu. Adversarial contrastive learning via asymmetric infonce. In European Conference on Computer Vision , pages 53-69. Springer, 2022.

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146 , 2016.

Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Axi Niu, Jiu Feng, Chang D Yoo, and In So Kweon. Decoupled adversarial contrastive learning for self-supervised adversarial robustness. In European Conference on Computer Vision , pages 725-742. Springer, 2022.

Cheng Zhang, Kun Zhang, and Yingzhen Li. A causal view on robustness of neural networks. Advances in Neural Information Processing Systems , 33:289-301, 2020.

Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML , 2019.

Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan S. Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In International Conference on Machine Learning , pages 11278-11287. PMLR, 2020.

Yonggang Zhang, Mingming Gong, Tongliang Liu, Gang Niu, Xinmei Tian, Bo Han, Bernhard Schölkopf, and Kun Zhang. Adversarial robustness through the lens of causality. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum? id=cZAi1yWpiXQ .

A Mathematical Notations

B Proof

B.1 Proof of Theorem 1

Theorem 1 (restated). From the causal view, in ACL, maximizing the conditional probability both p ( y R | x ) and p ( y R | ˜ x ) is equivalent to minimizing the learning objective of ACL [29] that is the sum of standard and adversarial contrastive losses.

Proof. To begin with, we formulate the proxy label driven y R by the data augmentation in the context of contrastive learning. We denote the index of x i k ∈ B i and x j k ∈ B j as ki and kj , respectively. For the augmented data x i k ∈ B i and x j k ∈ B j , we denote their corresponding proxy labels as y R ki and y R kj , which is formulated as follows:

where 1 ku ∈ { 0 , 1 } 2 β -1 is a one-hot label where u ∈ { i, j } , the value at index ku is 1 , and the values at other indexes are 0 . The one-hot label refers to the refined proxy label of the augmented view of the data point, which directs to its peer view. In the causal view of SCL and ACL, each augmented view of the data has its own proxy label. Therefore, (A)CL aims to minimize the differences in representation output of different views of the same data. Specifically, the conditional probability p ( y R ki | x i k ; θ ) and p ( y R kj | x j k ; θ ) is formulated as follows:

where sim( p, q ) = p ⊤ q/ ∥ p ∥∥ q ∥ is the cosine similarity function and t > 0 is a temperature parameter.

Then, we have the following results:

Therefore, we can conclude that SCL actually maximizes the conditional probability of the proxy label given the unlabeled data, i.e., p ( y R | x ) .

In the adversarial context, we only need to replace the natural data x i k and x j k in Eq. (13) and (18) with the adversarial data ˜ x i k and ˜ x j k generated in Eq. (3). Therefore, we have the following results:

where the adversarial data ˜ x i k and ˜ x j k are generated in Eq. (3). Therefore, from the causal view, in ACL [29], maximizing both the conditional probability of the proxy label given the natural data (i.e., p ( y R | x ) ) and that given adversarial data (i.e., p ( y R | ˜ x ) ) is equivalent to minimizing the sum of standard and adversarial contrastive losses.

B.2 Proof of Lemma 2

Lemma 2 (restated). AIR in Eq. (7) can be decomposed into two terms as follows:

where E x ∼ Q 3 ( x ) [KL( Q 1 ( x ) ∥ Q 2 ( x ))] is the expectation of KL divergence over Q 3 and Q 1 , Q 2 , Q 3 are probability distributions.

Proof. We transform AIR in Eq. (7) as follows:

Eq. (21) and (25) are obtained by using the KL divergence KL( p ( x ) ∥ q ( x ); B ) = ∑ x ∈ B p ( x ) log p ( x ) q ( x ) . We obtained Eq. (27) by omiting the term { x } in Eq. (26) for simplicity and finished proving Lemma 2.

B.3 Proof of Proposition 3

Proposition 3 (restated). AIR implicitly enforces the robust representation to satisfy the following two proxy criteria:

Proof. To enforce the robust representation to satisfy the two proxy criteria shown in Eq. (28), we need to regulate the ACL with the following regularization

Compared with Eq. (25) in the proof of Lemma 2, we can find that AIR also penalizes the same two KL divergence terms while AIR has two extra calibration terms p do ( τ i ) (˜ x | x ) and p do ( τ i ) ( y R | ˜ x ) that adjust the confidence of each KL divergence term, respectively. We empirically studied the impact of

the calibration term in Appendix C.8 and found that the calibration term is beneficial to improving the performance of ACL. Therefore, it indicates that AIR implicitly enforces the robust representation to satisfy the two proxy criteria shown in Eq. (28).

B.4 Proof of Proposition 4

Proposition 4 (restated). Let Y = { y t } T t =1 be a label set of a downstream classification task, Y R be a refinement of Y , and ˜ x t be the adversarial data generated on the downstream task. Assuming that ˜ x t ∈ B ϵ [ x ] and ˜ x ∈ B ϵ [ x ] , we have the following results:

Proof. The proof is inspired by Mitrovic et al. [38].

For t ∈ { 1 , . . . , T } , we have

Eq. (31) holds due to the fact that y R is a refinement of y t according to the definition of the refinement shown in Mitrovic et al. [38]. Eq. (32) holds since the relationship between y t and y R is independent of the style, i.e., p do ( τ i ) ( y t | y R ) = p do ( τ j ) ( y t | y R ) . Eq. (33) holds because of the assumption that the prediction of adversarial data is independent of the style. Due to that the condition p do ( τ i ) ( y R | ˜ x ) = p do ( τ j ) ( y R | ˜ x ) holds for any ˜ x ∈ B ϵ [ x ] , thereby, this condition also holds for ˜ x t ∈ B ϵ [ x ] . Lastly, we obtain that the prediction of adversarial data will be still independent of the style factors on downstream tasks.

Next, due to that the condition p do ( τ i ) (˜ x | x ) = p do ( τ j ) (˜ x | x ) holds for any ˜ x ∈ B ϵ [ x ] , this condition will hold for ˜ x t ∈ B ϵ [ x ] as well, i.e., p do ( τ i ) (˜ x t | x ) = p do ( τ j ) (˜ x t | x ) . Therefore, the consistency between adversarial and natural data will be still invariant of the style factors on the downstream tasks.

C Extensive Experimental Details and Results

Experimental environments. We conducted all experiments on Python 3.8.8 (PyTorch 1.13) with NVIDIA RTX A5000 GPUs (CUDA 11.6).

Pre-training details of ACL [29]. Following Jiang et al. [29], we leveraged ResNet-18 [25] with the dual batch normalization (BN) [46] as the representation extractor, where one BN is used for the standard branch of the feature extractor and the other BN is used for the adversarial branch, during conducting ACL [29] and its variant DynACL [36]. We pre-trained ResNet-18 models using SGD for 1000 epochs with an initial learning rate 5.0 and a cosine annealing schedule [35]. During pre-training, we set the adversarial budget ϵ as 8 / 255 , the hyperparameter ω as 0.0, and the strength of data augmentation as 1.0. As for the reproduction of the baseline, we used the pre-trained weights published in the GitHub of ACL 45 as the pre-trained representation extractor for finetuning.

Pre-training details of DynACL [36]. The training configurations of DynACL [36] followed ACL [29], except for the strength of data augmentation and the hyperparameter ω . We denote the

strength of data augmentation and the hyperparameter at epoch e as µ e and ω e respectively, where

in which the decay period K = 50 , the reweighting rate ν = 2 / 3 , the total training epoch E = 1000 . In our implementation of DynACL, we only take the dynamic strategy and do not take the trick of the stop gradient operation and the momentum encode [24, 11]. As for the reproduction of the baseline, we downloaded the pre-trained weights published in the GitHub of DynACL 678 as the pre-trained representation extractor for finetuning.

Details of finetuning procedures. As for SLF and ALF, we fixed the parameters of the representation extractor and only finetuned the linear classifier using the natural training data and adversarial training data respectively. As for AFF, we finetuned all the parameters using the adversarial training data. For all finetuning procedures, we used SGD for 25 epochs for linear finetuning and full finetuning respectively and momentum is set as 2 e -4 . The initial learning of linear finetuning is set as 0 . 01 on CIAFR-10, 0 . 05 on CIFAR-100, and 0 . 1 on STL-10. The adversarial budget is fixed as 8 / 255 for ALF and AFF. In practice, we used the finetuning code published in the GitHub of DynACL for implementing finetuning procedures.

C.1 Performance in Semi-Supervised Settings [1]

Table 10 reports the performance evaluated in semi-supervised settings [1] during the finetuning procedure. In semi-supervised settings, following ACL [29] and DynACL [36], we first standardly finetuned the pre-trained model using the labelled data and then generated the pseudo labels using the standardly finetuned model. Then, we finetuned the model using the data with pseudo labels as well as the labelled data via AFF. The results validate that AIR ( λ 1 = 0 . 5 , λ 2 = 0 . 5 ) can consistently enhance both robust and standard test accuracy of ACL and its variant DynACL in semi-supervised settings.

C.2 Impact of the Hyper-Parameters λ 1 and λ 2

We show the performance achieved by ACL-AIR of different λ 1 ∈ { 0 . 00 , 0 . 25 , 0 . 50 , 1 . 00 } and λ 2 ∈ { 0 . 00 , 0 . 25 , 0 . 50 , 1 . 00 } on the CIFAR-10 task in Table 9. We can notice that when only leveraging adversarial regularization ( λ 1 = 0 , λ 2 > 0 ), both standard and robust test accuracy gain significant improvement, which indicates that adversarial regularization serves an important role in improving the standard generalization and robustness of ACL. By incorporating adversarial and standard regularization together (when both λ 1 > 0 and λ 2 > 0 ), the standard generalization of ACL gets further improved. Table 9 guides us to set λ 1 = 0 . 5 and λ 2 = 0 . 5 since it can yield the best performance.

C.3 Incorporating AIR with Two Extra Variants of ACL

Here, we show extra results to validate that our proposed AIR can enhance two extra variants of ACL, including AdvCL [22] and AdvCL with A-InfoNCE [50].

AdvCL [22] leverages an extra contrastive view of high-frequency components and the pseudo labels generated by the clustering method. We leveraged our proposed method IR ( λ 1 = 0 . 5 , λ 2 = 0 . 5 ) to further improve the performance of AdvCL. Yu et al. [50] proposed an asymmetric InfoNCE objective (A-InfoNCE) that treats adversaries as inferior positives that induce weaker learning signals, or as hard negatives exhibiting higher contrast to other negative samples.

We used the code provide by Yu et al. [50] in their GitHub 9 to implement the pre-training of AdvCL as well as A-InfoNCE, and the finetuning procedure. In Table 12, we show that our proposed AIR ( λ 1 = 0 . 5 , λ 2 = 0 . 5 ) can consistently improve the robust and standard test accuracy of two extra variants [22, 50].

C.4 Applicability with Different Backbone Networks

In this subsection, we demonstrate that DynACL-AIR can consistently improve the performance on ResNet-34, ResNet-50 [25], and WRN-28-10 [51]. We followed the same training configurations of pre-training and finetuning (SLF) in Section 4 except for the backbone network. Table 11 validates that IR can further enhance the robust and standard test accuracy of DynACL on various backbone networks.

We observe that WRN-28-10 yields a worse performance compared to ResNet-50. We conjecture that it is because we set the batch size to 128 during pre-training due to the limitation of our GPU memory. We believe using a larger batch size during pre-training can further improve the performance of WRN-28-10 according to Chen et al. [9].

C.5 Robustness Evaluation under Various Attacks

In this subsection, we conducted the robustness self-transferability against three strong white-box attacks (APGD-CE [13], APGD-DLR [13] and FAB [14]) and one strong black-box attack (i.e., Square Attack [2]). We evaluate the robustness on the CIFAR-10 dataset via SLF and report the results in Table 13. The results validate that our proposed method can consistently improve robust test accuracy over various adversaries.

C.6 Test Accuracy under Each Type of Common Corruption

We report the test accuracy under each type of common corruption [26] with corruption severity being fixed as 5 in Table 14. We used pre-trained models on CIFAR-10 via DynACL and DynACL-AIR.

Experimental settings are the same as Section 4. The results validate that IR can consistently improve the test accuracy under each type of common corruption.

C.7 Standard Deviation of Test Accuracy under Common Corruptions

Here, we provide the standard deviation of the test accuracy under common corruptions in Tables 15 and 16.

C.8 Impact of Calibration Terms

In the proof of Proposition 3, we show that there exist calibration terms p do ( τ i ) (˜ x | x ) and p do ( τ i ) ( y R | ˜ x ) in AIR. Here, we empirically investigate the impact of calibration on the performance in Table 17. The results empirically validate that the calibration term is an essential component of AIR to adjust the confidence of the KL divergence and further enhance the performance of ACL.

D Limitations

One of the limitations is that similar to previous works [29, 22, 50, 36], our proposed AIR is not helpful in improving the efficiency and scalability of ACL methods. Therefore, robust pre-training via

ACL-AIR is still unfriendly to the environment due to emitting much carbon dioxide and consuming much electricity. Besides, applying ACL-AIR to large-scale datasets such as ImageNet-21K [41] is still computationally prohibitive with limited GPU sources.

E Possible Negative Societal Impacts

Our proposed method aims to improve the performance of robust self-supervised pre-training methods [29, 22, 50, 36]. The robust self-supervised pre-training procedure is extremely time-consuming since it needs to spend much time generating adversarial training data, which thus leads to consuming a lot of electricity and emitting lots of carbon dioxide. Unfortunately, our proposed method is not helpful in improving the efficiency of ACL. Consequently, our study could exacerbate the greenhouse effect and be not conducive to environmental protection.

Peer reviews:
comments: SUMMARY:
The authors propose AIR to regulate the the process of contrastive learning. They first analyze the causal graph of contrastive learning under the adversarial setting, and try to enforce  $p\left(y^R | \tilde x\right)\cdot p\left(\tilde x | x\right)$ (Eq.4) to be invariant under different interventions. Intuitively, this regularizer would force the crafted adversarial examples to be style-independent and thus the learned representations shall enjoy better robustness.

STRENGTHS:
The core idea of this paper is technically novel and clearly presented.

Experiments are reported by applying the proposed AIR / SIR terms to ACL / DynACL, and consistent performance gains are observed. Ablation studies for different values of $\lambda_1$ and $\lambda_2$ are also provided, to verify the effectiveness of the AIR term itself.

The authors also provide theoretical results to justify the rationality of the AIR term.

WEAKNESSES:
The proposed method would require more tunning since it incorporates two more hyper-parameters $\lambda_1$ and $\lambda_2$, and we can see from Table 7 that these two hyper-parameters interact in a non-trival way. For example, when compared with the case that only AIR is leveraged ($\lambda_1=0,\ \lambda_2>0$), incorporating AIR with SIR (i.e., $\lambda_1>0,\ \lambda_2>0$) could result in better **robust accuracy**.

Little type in Eq. 8: two augmentations in KL divergence should be different.

--- Update ---
After reading additional experimented provieded by the authors, I decide to raise my score to 8.

QUESTIONS:
See "Weaknesses" part.

LIMITATIONS:
N.A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-19

comments: SUMMARY:
This paper proposes a novel adversarial contrastive learning method, which introduces causal reasoning method to obtain robust feature representation and enforce independence from style factors. The idea is simple and effective. In addition, the experiments are sufficient to prove the effectiveness of the proposed method. The source codes are released.

STRENGTHS:
1. The idea is interesting and easy to understand.
2. The theoretical analysis is detailed and proves the effectiveness of the proposed method.

WEAKNESSES:
1. The improvement of the proposed method is limited.
2. Without causal reasoning, this paper only proposes two distribution alignment regularizations in the adversarial samples to obtain robust feature representation. Thus, this paper is an incremental work based on ACL and DynACL.
3. I want to see the performance of this paper in the large-scale datasets, such as sub-imagenet.

QUESTIONS:
Shown as Weaknesses.

LIMITATIONS:
Shown as Weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-22

comments: SUMMARY:
This paper proposes to tackle adversarial contrastive learning via causal reasoning. Specifically, the authors discuss the scenario where adversarial examples are involved in standard invariant regularization. The authors also provide analysis of proposed invariant regularization to justify the rationality. The experiments on different datasets show improvement compared with baselines.

STRENGTHS:
1. The paper is well-written and easy to follow.
2. Taking adversarial example in the SIR via Markov condition seems natural and the analysis could provide some insights.
3. The experiments on various datasets show improvement.

WEAKNESSES:
I have several concerns:

1. The comparison with other baselines is missing in Table 4, such as ACL with SIR and DynACL with SIR.
2. The authors conduct experiments on ResNet family. However, WideResNet backbone networks are always evaluated in the field of adversarial robustness. I wonder whether the superiority of IR can be generalized to different networks.

Minors:

It seems a typo in Eq. 8, the KL divergence should be computed between $\tau_i$ and $\tau_j$.

QUESTIONS:
1. Please include the comparison with baseline in Table 4.
2. Please include more evaluation on WideResNet.

LIMITATIONS:
The authors have discussed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a method to advance adversarial contrastive learning by utilizing a technique called causal reasoning. The adversarial invariant regularization (AIR) proposed in this paper demonstrated a style factor. Additionally, the effectiveness of the proposed method was empirically shown using CIFAR10 and CIFAR100.\n\nSTRENGTHS:\n- Theoretically, it explained how the KL divergence can provide invariant regularization.\n- The paper is well organized and most explanations are clearly written and easy to understand the approach and the theoretical explanations.\n\nWEAKNESSES:\n- The theoretical analysis of this paper (section 3.2 and section 3.3) appears to apply adversarial examples to the theorem shown in paper [1], which seems closer to an application rather than a novel combination, resulting in a perceived lack of originality.\n- Although most theorems and reasoning methods are well explained, expressing which input types the KL divergence loss of SIR and AIR applies to would be more effective for reproducing this paper through implementation.\n- Furthermore, the performance gain shown in robustness experiments from the proposed regularization is considered to have low significance as it is less than 1% in most datasets and tasks.\n- Also, there is insufficient explanation as to why the proposed method is more helpful for self-supervised adversarial robustness than existing methods, making it difficult to verify the experimental significance of the proposed method. Especially in the experimental results, AIR helps both clean performance and robustness compared to SIR, but there is no clear justification for why these two components should be used simultaneously.\n- It is unclear why a model trained with SIR and AIR regularization for adversarial x and natural x performs well against common corruption.\n- As it's a self-supervised pretraining method, the fact that experiments were only conducted on small datasets (CIFAR10, CIFAR100) leaves concerns about its applicability.\n\n[1] Mitrovic et al., REPRESENTATION LEARNING VIA INVARIANT CAUSAL MECHANISMS, ICLR 2021\n\nQUESTIONS:\n- It would be good to mention y^R in the main text.\n- In equation 3, adversarial for x_k^i and x_k^j is expressed in one formula. I'm curious if x_k^j also gets its perturbation updated when creating the adversarial for x_k^i.\n- It would be great to mention how much performance difference there is, how much slower the speed is, and how much the transferability improves when compared to Supervised Adversarial Training (Madry, TRADES).\n\nLIMITATIONS:\n- There is a limitation in the hyperparameter tuning of lambda1 and lambda2.\n- There is a limitation that the gain in performance is marginal.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}