SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 3. AIR implicitly enforces the robust representation to satisfy the following two proxy criteria:
(1)  $p^{do(\tau_i)}(y^R |\tilde{x}) = p^{do(\tau_j)}(y^R |\tilde{x}),$  (2)  $p^{do(\tau_i)}(\tilde{x} |x) = p^{do(\tau_j)}(\tilde{x} |x).$
The proof of Proposition 3 is in Appendix B.3.
Remarks. Proposition 3 explains that AIR implicitly enforces the representational distance to be style-independent between the original view of natural data and their adversarial variants (i.e.,  $p^{do(\tau_i)}(y^R|\tilde{x}) = p^{do(\tau_j)}(y^R|\tilde{x})$ ), as well as between the augmented view of natural data and their adversarial counterparts (i.e.,  $p^{do(\tau_i)}(\tilde{x}|x) = p^{do(\tau_j)}(\tilde{x}|x)$ ). Therefore, Proposition 3 explicitly presents that AIR is different from SIR [38], where SIR only enforces the augmented views of natural data to be style-independent.
Next, we theoretically show that on the assumption that  $\mathcal{Y}^R$  is a refinement of  $\mathcal{V}$ , the style-independence property of robust representations will hold on downstream classification tasks in Proposition 4.
Proposition 4. Let  $\mathcal{Y} = \{y_t\}_{t=1}^T$  be a label set of a downstream classification task,  $\mathcal{Y}^R$  be a refinement of  $\mathcal{Y}$ , and  $\tilde{x}_t$  be the adversarial data generated on the downstream task. Assuming that  $\tilde{x} \in \mathcal{B}_{\epsilon}[x]$  and  $\tilde{x}_t \in \mathcal{B}_{\epsilon}[x]$ , we have the following results:
$$
\begin{array}{l} p ^ {d o \left(\tau_ {i}\right)} \left(y ^ {R} | \tilde {x}\right) = p ^ {d o \left(\tau_ {j}\right)} \left(y ^ {R} | \tilde {x}\right) \Longrightarrow p ^ {d o \left(\tau_ {i}\right)} \left(y _ {t} | \tilde {x} _ {t}\right) = p ^ {d o \left(\tau_ {j}\right)} \left(y _ {t} | \tilde {x} _ {t}\right) \quad \forall \tau_ {i}, \tau_ {j} \in \mathcal {T}, \\ p ^ {d o \left(\tau_ {i}\right)} (\tilde {x} | x) = p ^ {d o \left(\tau_ {j}\right)} (\tilde {x} | x) \Longrightarrow p ^ {d o \left(\tau_ {i}\right)} (\tilde {x} _ {t} | x) = p ^ {d o \left(\tau_ {j}\right)} (\tilde {x} _ {t} | x) \quad \forall \tau_ {i}, \tau_ {j} \in \mathcal {T}. \\ \end{array}
$$
The proof of Proposition 4 is shown in Appendix B.4.
Remarks. Proposition 4 indicates that the robust representation of data on downstream tasks is still style-independent. Mitrovic et al. [38] empirically showed that the style-independence property of standard representations can significantly improve the robustness against common corruptions [26] on downstream tasks. In addition, Proposition 4 indicates that AIR helps to find the invariant correlations that are independent of the style factors among different training distributions. It is similar to the objective of invariant risk minimization [3] that can yield substantial improvement in the robustness against various corruptions incurred by distribution shifts. Therefore, the style-independence property of robust representations learned via ACL perhaps helps to enhance the robustness against input perturbations on downstream tasks.
Table 1: Self-task adversarial robustness transferability.
Dataset	Pre-training	SLF	ALF	AFF
AA (%)	SA (%)	AA (%)	SA (%)	AA (%)	SA (%)
CIFAR-10	ACL [29]	37.39±0.06	78.27±0.09	40.61±0.07	75.56±0.09	49.42±0.07	82.14±0.18
ACL-AIR	38.89±0.06	80.03±0.07	41.39±0.08	78.29±0.10	49.84±0.04	82.42±0.06
DynACL [36]	45.05±0.04	75.39±0.05	45.65±0.05	72.90±0.08	50.52±0.05	81.86±0.11
DynACL-AIR	45.17±0.04	78.08±0.06	46.01±0.07	77.42±0.10	50.60±0.08	82.14±0.11
CIFAR-100	ACL [29]	15.78±0.05	45.70±0.10	17.36±0.16	42.69±0.13	24.16±0.29	56.68±0.14
ACL-AIR	16.14±0.07	49.75±0.10	18.68±0.11	47.07±0.15	25.27±0.10	57.79±0.18
DynACL [36]	19.31±0.06	45.67±0.09	20.30±0.08	43.58±0.12	24.70±0.23	57.22±0.28
DynACL-AIR	20.45±0.07	46.84±0.12	21.23±0.09	45.63±0.10	25.34±0.12	57.44±0.14
STL-10	ACL [29]	35.80±0.06	67.90±0.09	38.10±0.11	69.96±0.14	43.21±0.16	72.55±0.18
ACL-AIR	36.94±0.06	68.91±0.07	39.05±0.10	71.30±0.12	43.75±0.13	72.84±0.14
DynACL [36]	46.49±0.05	69.59±0.08	47.69±0.10	67.65±0.12	45.64±0.13	72.14±0.15
DynACL-AIR	47.66±0.06	72.30±0.10	48.89±0.08	71.60±0.09	48.10±0.11	73.10±0.17
Table 2: Self-task common-corruption robustness transferability. The corruption severity (CS) ranging from  $\{1,3,5\}$  (denoted as "CS-\{1,3,5\}") The standard deviation is in Table 15.
Dataset	Pre-training	SLF	ALF	AFF
CS-1	CS-3	CS-5	CS-1	CS-3	CS-5	CS-1	CS-3	CS-5
CIFAR-10	ACL [29]	76.57	71.78	62.78	74.04	69.49	61.38	79.15	72.54	65.27
ACL-AIR	78.55	73.33	64.28	76.65	71.38	63.17	79.49	72.95	65.37
DynACL [36]	73.92	69.01	62.51	71.74	66.95	60.87	79.77	72.95	65.60
DynACL-AIR	76.62	70.16	63.29	75.70	69.55	62.67	80.98	74.31	66.75
CIFAR-100	ACL [29]	43.14	38.90	32.27	39.52	35.13	29.30	53.80	45.93	37.61
ACL-AIR	47.85	41.54	34.12	44.80	40.70	35.46	55.12	47.25	39.02
DynACL [36]	44.19	38.08	31.05	39.83	35.61	30.51	54.34	46.71	38.62
DynACL-AIR	45.36	39.21	32.44	42.03	36.48	30.49	55.31	47.33	39.13
In this section, we demonstrate the effectiveness of our proposed AIR in improving ACL [29] and its variants [36, 22, 50] on various datasets including CIFAR-10 [31], CIFAR-100 [31], STL-10 [12], CIFAR-10-C [26], and CIFAR-100-C [26]. Extensive experimental details are shown in Appendix C.
Pre-training. In the main paper, we demonstrate the results of applying our proposed AIR with  $\lambda_{1} = 0.5$  and  $\lambda_{2} = 0.5$  to ACL [29] and DynACL [36] (denoted as "ACL-AIR" and "DynACL-AIR", respectively). We utilized ResNet-18 [25] as the representation extractor following previous self-supervised adversarial training methods [29, 22, 36, 50]. We adopted the same training configuration of ACL [29] using SGD for 1000 epochs with an initial learning rate of 5.0 and a cosine annealing schedule [35]. The batch size  $\beta$  is fixed as 512. The adversarial budget  $\epsilon$  is set as  $8 / 255$ . In the context of DynACL, we took the same data augmentation scheduler and the same scheduler of the hyperparameter  $\omega$  as the setting in the original paper of DynACL [36]. Note that, to reproduce the results of baseline methods, we downloaded the pre-trained weights of ACL [29] on CIFAR-10/CIFAR-100 and DynACL [36] on CIFAR-10/CIFAR-100/STL-10 from their official GitHub as the pre-trained representation extractor. We provide the extensive ablation studies on the hyper-parameters  $(\lambda_{1}$  and  $\lambda_{2})$  in Appendix C.2, the ACL variants (including AdvCL [22] and A-InfoNCE [50]) in Appendix C.3, and the backbone models (including ResNet-50 and Wide ResNet [51]) in Appendix C.4.
Finetuning procedures. We adopted the following three types of finetuning procedures: standard linear finetuning (SLF), adversarial linear finetuning (ALF), and adversarial full finetuning (AFF), to evaluate the learned representations. The former two finetuning procedures freeze the learned representation extractor and only train the linear classifier using natural or adversarial samples, respectively. We took the pre-trained representation extractor as weight initialization and trained the whole model using the adversarial data during AFF. The training configuration of finetuning (e.g., the finetuning epoch and optimizer) exactly follows DynACL [36]. Specifically, we used the official code provided in DynACL [36]'s GitHub for finetuning and illustrate the experimental settings of finetuning in Appendix C as well. For each pre-trained representation extractor, we repeated the finetuning experiments 3 times and report the median results and the standard deviation.
Evaluation protocols. We let "AA" denote the robust test accuracy under AutoAttack (AA) [13] and "SA" stand for the standard test accuracy. To evaluate the robustness against common corruption, we report the mean test accuracy under 15 types of common corruptions [26] with the corruption
Table 3: Cross-task adversarial robustness transferability.  $\mathcal{D}_1\to \mathcal{D}_2$  denotes pre-training and finetuning are conducted on the dataset  $\mathcal{D}_1$  and  $\mathcal{D}_2(\neq \mathcal{D}_1)$ , respectively.
D1→D2	Pre-training	SLF	ALF	AFF
AA (%)	SA (%)	AA (%)	SA (%)	AA (%)	SA (%)
CIFAR-10 → CIFAR-100	ACL [29]	9.98±0.02	32.61±0.04	11.09±0.06	28.58±0.06	22.67±0.16	56.05±0.19
ACL-AIR	11.04±0.06	39.45±0.07	13.30±0.02	36.10±0.05	23.45±0.04	56.31±0.06
DynACL [36]	11.01±0.02	27.66±0.03	11.92±0.05	24.14±0.09	24.17±0.10	55.61±0.17
DynACL-AIR	12.20±0.04	31.33±0.03	12.70±0.03	28.70±0.05	24.82±0.07	57.00±0.13
CIFAR-10 → STL-10	ACL [29]	25.41±0.08	56.53±0.10	27.17±0.09	51.71±0.17	32.66±0.07	61.41±0.13
ACL-AIR	28.00±0.12	61.91±0.13	30.06±0.10	62.03±0.11	34.26±0.09	62.58±0.10
DynACL [36]	28.52±0.09	52.45±0.10	29.13±0.13	49.53±0.17	35.25±0.15	63.29±0.18
DynACL-AIR	29.88±0.04	54.59±0.12	31.24±0.06	57.14±0.09	35.66±0.05	63.74±0.12
CIFAR-100 → CIFAR-10	ACL [29]	18.72±0.07	60.90±0.02	26.92±0.11	57.35±0.07	44.07±0.11	75.19±0.10
ACL-AIR	19.90±0.04	64.89±0.09	27.65±0.06	60.79±0.04	44.84±0.14	75.67±0.13
DynACL [36]	25.23±0.12	59.12±0.10	28.92±0.10	56.09±0.14	47.40±0.23	77.92±0.18
DynACL-AIR	25.63±0.07	59.83±0.08	29.32±0.06	56.65±0.06	47.92±0.12	78.44±0.10
CIFAR-100 → STL-10	ACL [29]	21.77±0.07	46.19±0.05	24.46±0.09	45.40±0.12	28.76±0.07	56.16±0.13
ACL-AIR	22.44±0.04	51.52±0.02	26.55±0.06	53.24±0.09	30.40±0.08	58.45±0.11
DynACL [36]	23.17±0.09	47.54±0.14	26.24±0.13	45.70±0.14	31.17±0.14	58.35±0.18
DynACL-AIR	23.24±0.07	48.20±0.08	26.60±0.05	48.55±0.12	31.42±0.07	58.59±0.10
Table 4: Cross-task common-corruption robustness transferability. The corruption severity (CS) ranging from  $\{1,3,5\}$  (denoted as "CS-\{1,3,5\}") The standard deviation is in Table 16.
D1→D2	Pre-training	SLF	ALF	AFF
CS-1	CS-3	CS-5	CS-1	CS-3	CS-5	CS-1	CS-3	CS-5
CIFAR-10 → CIFAR-100	ACL [29]	31.39	27.76	23.27	27.80	25.09	21.09	52.07	44.22	36.31
ACL-AIR	37.82	32.83	27.06	34.67	30.09	25.07	52.81	44.95	37.01
DynACL [36]	26.74	23.97	20.87	23.70	21.43	18.84	52.87	44.87	36.76
DynACL-AIR	30.35	26.53	22.76	27.67	24.35	20.96	54.00	46.01	37.75
CIFAR-100 → CIFAR-10	ACL [29]	59.65	55.14	49.09	56.15	51.96	47.04	72.94	66.05	59.17
ACL-AIR	63.34	58.14	51.03	59.14	54.23	48.97	73.48	66.83	60.21
DynACL [36]	57.14	52.07	47.03	56.31	52.03	47.40	75.89	69.08	62.02
DynACL-AIR	58.41	53.09	47.94	54.89	49.84	45.08	76.35	69.49	62.44
severity (CS) ranging from  $\{1,3,5\}$  (denoted as "CS-\{1,3,5\}"). Specifically, we used the official code of AutoAttack [13] and RobustBench [15] for implementing evaluations. In Appendix C.5, we provide robustness evaluation under more diverse attacks [13, 14, 2]. In Appendix C.6, we provide the test accuracy under each type of common corruption [26].
4.1 Evaluation of Self-Task Robustness Transferability
Self-task adversarial robustness transferability. Table 1 reports the self-task adversarial robustness transferability evaluated on three datasets where pre-training and finetuning are conducted on the same datasets including CIFAR-10, CIFAR-100, and STL-10. In Appendix C.1, we report the self-task robustness transferability evaluated in the semi-supervised settings. Table 1 demonstrates that our proposed AIR can obtain new state-of-the-art (SOTA) both standard and robust test accuracy compared with existing ACL methods [29, 36]. Therefore, It validates the effectiveness of our proposed AIR in consistently improving the self-task adversarial robustness transferability against adversarial attacks as well as standard generalization in downstream datasets via various finetuning procedures. Notably, compared with the previous SOTA method DynACL [36], DynACL-AIR increases standard test accuracy by  $4.52\%$  (from  $72.90\%$  to  $77.42\%$ ) on the CIFAR-10 dataset via ALF,  $2.05\%$  (from  $43.58\%$  to  $45.63\%$ ) on the CIFAR-100 dataset via ALF, and  $1.17\%$  (from  $46.49\%$  to  $47.66\%$ ) on the STL-10 dataset via SLF.
Self-task common-corruption robustness transferability. Table 2 reports the self-task common-corruption [26] robustness transferability. Specifically, we conducted pre-training on CIFAR-10/CIFAR-100 and then evaluated the test accuracy on CIFAR-10-C/CIFAR-100-C with various corruption severities after various finetuning procedures. Note that the test accuracy under each type of common corruption is reported in Appendix C.6. Table 2 demonstrates that AIR leads to consistent and significant improvement in the robustness against common corruption. In addition, we observe that ACL always achieves much higher test accuracy under common corruptions than DynACL after SLF. We conjecture the reason is that DynACL uses weak data augmentations at the later phase of training, which makes DynACL more likely to overfit the training distribution of CIFAR-10 and thus leads to worse performance under common corruptions via SLF.
Table 5: Self-task adversarial robustness transferability with post-processing. “++” denotes that the pre-trained models are post-processed via LP-AFF [32, 36]. LP-AFF [32] first generates pseudo labels via clustering and then adversariably pre-trains the model using the pseudo labels.
Dataset	Pre-training	SLF	ALF	AFF
AA (%)	SA (%)	AA (%)	SA (%)	AA (%)	SA (%)
CIFAR-10	DynACL++ [36]	46.46†	79.81†	47.95†	78.84†	50.31†	81.94†
DynACL-AIR++	46.99	81.80	48.23	79.56	50.65	82.36
CIFAR-100	DynACL++ [36]	20.07†	52.26†	22.24	49.92	25.21	57.30
DynACL-AIR++	20.61	53.93	22.96	52.09	25.48	57.57
STL-10	DynACL++ [36]	47.21†	70.93†	48.06	69.51	41.84	72.36
DynACL-AIR++	47.90	71.44	48.59	71.45	44.09	72.42
Table 6: Self-task adversarial robustness transferability evaluated on the Imagenette dataset.
Pre-training	ResNet-18	ResNet-50
AA (%)	SA (%)	AA (%)	SA (%)
DynACL [36]	57.15	79.41	58.98	80.74
DynACL-AIR	58.34	80.61	60.10	81.66
4.2 Evaluation of Cross-Task Robustness Transferability
Cross-task adversarial robustness transferability. Table 3 shows the cross-task adversarial robustness transferability where pre-training and finetuning are conducted on the different datasets. We can observe that AIR substantially improves both ACL's and DynACL's robustness against adversarial attacks [13] and standard generalization to other downstream datasets. Particularly, AIR improves the standard and robust test accuracy of ACL via ALF by  $7.52\%$  (from  $28.58\%$  to  $36.10\%$ ) and  $2.21\%$  (from  $11.09\%$  to  $13.30\%$ ), respectively.
Cross-task common-corruption robustness transferability. Table 4 reports the cross-task common-corruption [26] robustness transferability where pre-training is conducted on CIFAR-10/CIFAR-100 and finetuning is conducted on CIFAR-100-C/CIFAR-10-C [26]. The empirical results show that our proposed AIR can consistently improve the accuracy under common corruptions with various severity, which validates the effectiveness of AIR in enhancing the robustness transferability against common corruptions.
4.3 Evaluation of Pre-Trained Models after Post-Processing
Here, we report the self-task adversarial robustness transferability of pre-trained models after post-processing in Table 5. The post-processing method is linear probing and then adversarial full finetuning (LP-AFF) [32] which first generates pseudo labels via clustering and then further adversarially trains the pre-trained model using the pseudo labels. “++” denotes that the pre-trained model is post-processed via LP-AFF. We provide the comparison between DynACL++ and DynACL-AIR++ in Table 5. Note that the superscript  $\dagger$  denotes that the results of DynACL++ are copied from Luo et al. [36]. The results validate that AIR is compatible with the trick of post-processing via LP-AFF [32], which can yield improved performance on various downstream datasets.
4.4 Evaluation on High-Resolution Imagenette Dataset
To the best of our knowledge, there exists no reported result of ACL and DynACL on high-resolution datasets in the existing papers. Therefore, we pre-trained ResNet-18 and ResNet-50 on Imagenette $^3$  of resolution  $256 \times 256$  which is a subset of ImageNet-1K using DynACL and DynACL-AIR, respectively. We compare and report the performance evaluated on Imagenette via SLF in Table 6. Our results validate that our proposed AIR is effective in enhancing the performance on high-resolution datasets. Note that, due to the limitation of our GPU memory, we set the batch size to 256 and 128 for pre-training ResNet-18 and ResNet-50 on Imagenette, respectively. We believe using a larger batch size during pre-training can even further improve the performance [9].
4.5 Evaluation via Automated Robust Fine-Tuning
While evaluating the transferability of a pre-trained model, we observed that the performance on the downstream dataset is sensitive to the hyper-parameters (e.g., the learning rate) of the finetuning
<sup>3</sup>We downloaded the Imagenette dataset from https://github.com/fastai/imagenette.
Table 7: Cross-task adversarial robustness transferability evaluated via AutoLoRa [47].  $\mathcal{D}_1 \rightarrow \mathcal{D}_2$  denotes pre-training and finetuning are conducted on the dataset  $\mathcal{D}_1$  and  $\mathcal{D}_2(\neq \mathcal{D}_1)$ , respectively. "Diff" refers to the gap between the performance achieved by AutoLoRa and that achieved by vanilla finetuning (reported in Table 3).
D1→D2	Finetuning mode	Pre-training	AutoLoRa [47]	Diff
AA (%)	SA (%)	AA (%)	SA (%)
CIFAR-10 → STL-10	SLF	DynACL [36]	29.68	58.24	+0.51	+5.83
DynACL-AIR	29.75	60.53	+0.11	+4.69
ALF	DynACL [36]	31.34	57.74	+1.75	+8.19
DynACL-AIR	31.65	59.44	+0.41	+2.30
AFF	DynACL [36]	35.55	64.16	+0.30	+0.63
DynACL-AIR	35.81	64.28	+0.15	+0.54
CIFAR-100 → STL-10	SLF	DynACL [36]	23.31	50.93	+0.14	+3.39
DynACL-AIR	23.49	51.28	+0.25	+3.08
ALF	DynACL [36]	26.53	51.74	+0.29	+6.04
DynACL-AIR	26.89	49.02	+0.29	+0.47
AFF	DynACL [36]	31.25	58.44	+0.08	+0.09
DynACL-AIR	31.57	58.65	+0.15	+0.21
procedure. It would require extensive computational resources to search for appropriate hyperparameters to achieve a satisfactory performance. To mitigate this issue, we leverage an automated robust finetuning framework called AutoLoRa [47], which can automatically schedule the learning rate and set appropriate hyper-parameters during finetuning. We report the performance achieved by AutoLoRa to further justify the SOTA performance by our proposed AIR in Table 7.
Table 7 shows that AutoLoRa can further enhance the performance of a pre-trained model on the downstream tasks without searching for appropriate hyper-parameters since the value of "Diff" is consistently larger than 0.0. Besides, Table 7 justifies that our proposed AIR is effective in enhancing robustness transferability via various finetuning methods.
This paper leveraged the technique of causal reasoning to interpret ACL and proposed adversarial invariant regularization (AIR) to enhance ACL. AIR can enforce the learned robust representations to be invariant of the style factors. We improved ACL by incorporating the adversarial contrastive loss with a weighted sum of AIR and SIR that is an extension of AIR in the standard context. Theoretically, we showed that AIR implicitly encourages the representational distance between different views of natural data and their adversarial variants to be independent of style factors. Empirically, comprehensive results validate that our proposed method can achieve new state-of-the-art performance in terms of standard generalization and robustness against adversarial attacks and common corruptions on downstream tasks.
[1] Jean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? Advances in Neural Information Processing Systems, 32, 2019.
[2] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-efficient black-box adversarial attack via random search. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII, pages 484-501. Springer, 2020.
[42] Joshua Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592, 2020.
[43] Axel Sauer and Andreas Geiger. Counterfactual generative networks. arXiv preprint arXiv:2101.06046, 2021.
[44] Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. Advances in Neural Information Processing Systems, 33:1513-1524, 2020.
[45] Ruoyu Wang, Mingyang Yi, Zhitang Chen, and Shengyu Zhu. Out-of-distribution generalization with causal invariant transformations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 375-385, 2022.
[46] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L Yuille, and Quoc V Le. Adversarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 819-828, 2020.
[47] Xilie Xu, Jingfeng Zhang, and Mohan Kankanhalli. Autolora: A parameter-free automated robust fine-tuning framework. arXiv preprint arXiv:2310.01818, 2023.
[48] Xilie Xu, Jingfeng Zhang, Feng Liu, Masashi Sugiyama, and Mohan Kankanhalli. Efficient adversarial contrastive learning via robustness-aware coreset selection. In NeurIPS, 2023.
[49] Xu Yang, Cheng Deng, Kun Wei, Junchi Yan, and Wei Liu. Adversarial learning for robust deep clustering. Advances in Neural Information Processing Systems, 33:9098-9108, 2020.
[50] Qiying Yu, Jieming Lou, Xianyuan Zhan, Qizhang Li, Wangmeng Zuo, Yang Liu, and Jingjing Liu. Adversarial contrastive learning via asymmetric infence. In European Conference on Computer Vision, pages 53-69. Springer, 2022.
[51] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016.
[52] Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Axi Niu, Jiu Feng, Chang D Yoo, and In So Kweon. Decoupled adversarial contrastive learning for self-supervised adversarial robustness. In European Conference on Computer Vision, pages 725-742. Springer, 2022.
[53] Cheng Zhang, Kun Zhang, and Yingzhen Li. A causal view on robustness of neural networks. Advances in Neural Information Processing Systems, 33:289-301, 2020.
[54] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
[55] Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan S. Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In International Conference on Machine Learning, pages 11278-11287. PMLR, 2020.
[56] Yonggang Zhang, Mingming Gong, Tongliang Liu, Gang Niu, Xinmei Tian, Bo Han, Bernhard Schölkopf, and Kun Zhang. Adversarial robustness through the lens of causality. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=cZAi1yWpiXQ.
A Mathematical Notations
Table 8: Notation Table
Notation	Description
X	input space
x∈X	data point
T	transformations set
τ∈T	data augmentation fraction
xu	augmented data point via the data augmentation fraction τu(·)
x̂	adversarial data
B~Xβ	a minibatch of β original image samples
Bu	a minibatch of β augmented image samples via the data augmentation fraction τu(·)
B̂u	a minibatch of adversarial counterparts of β augmented samples via the data augmentation fraction τu(·)
U~XN	an unlabeled data set of N image samples
Y	label space on the downstream tasks
yt∈Y	target label on the downstream tasks
YR	the refinement of the target label space
yR∈YR	the proxy label
yRku∈YR	the proxy label of the data point xkuu augmented via τu(·)
Z	projection space
V	feature space where the contrastive loss is applied
hθ: X→Z	the representation extractor parameterized by θ
g: Z→V	the projector
f(·)=g∘hθ(·)	the composition of the representation extractor and the projector
sim(·,·)	the cosine similarity function
KL(·||·)	KL divergence function
s	style variable
c	content variable
λ1	the weight of standard regularization
λ2	the weight of adversarial regularization
Theorem 1 (restated). From the causal view, in ACL, maximizing the conditional probability both  $p(y^{R}|x)$  and  $p(y^{R}|\tilde{x})$  is equivalent to minimizing the learning objective of ACL [29] that is the sum of standard and adversarial contrastive losses.