SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. To begin with, we formulate the proxy label driven  $y^{R}$  by the data augmentation in the context of contrastive learning. We denote the index of  $x_{k}^{i} \in B^{i}$  and  $x_{k}^{j} \in B^{j}$  as  $k i$  and  $k j$ , respectively. For the augmented data  $x_{k}^{i} \in B^{i}$  and  $x_{k}^{j} \in B^{j}$ , we denote their corresponding proxy labels as  $y_{ki}^{R}$  and  $y_{kj}^{R}$ , which is formulated as follows:
$$
y _ {k i} ^ {R} = \mathbf {1} _ {k j}, \quad y _ {k j} ^ {R} = \mathbf {1} _ {k i}, \tag {10}
$$
where  $\mathbf{1}_{ku} \in \{0, 1\}^{2\beta - 1}$  is a one-hot label where  $u \in \{i, j\}$ , the value at index  $ku$  is 1, and the values at other indexes are 0. The one-hot label refers to the refined proxy label of the augmented view of the data point, which directs to its peer view. In the causal view of SCL and ACL, each augmented view of the data has its own proxy label. Therefore, (A)CL aims to minimize the differences in representation output of different views of the same data. Specifically, the conditional probability  $p(y_{ki}^R | x_k^i; \theta)$  and  $p(y_{kj}^R | x_k^j; \theta)$  is formulated as follows:
$$
p \left(y _ {k i} ^ {R} \mid x _ {k} ^ {i}; \theta\right) = p \left(\mathbf {1} _ {k j} \mid x _ {k} ^ {i}; \theta\right) = \frac {e ^ {\operatorname* {s i m} \left(f _ {\theta} \left(x _ {k} ^ {i}\right) , f _ {\theta} \left(x _ {k} ^ {j}\right)\right) / t}}{\sum_ {x \in B ^ {i} \cup B ^ {j} \backslash \left\{x _ {k} ^ {i} \right\}}} e ^ {\operatorname* {s i m} \left(f _ {\theta} \left(x _ {k} ^ {i}\right), f _ {\theta} (x)\right) / t}}, \tag {11}
$$
$$
p \left(y _ {k j} ^ {R} \mid x _ {k} ^ {j}; \theta\right) = p \left(\mathbf {1} _ {k i} \mid x _ {k} ^ {j}; \theta\right) = \frac {e ^ {\operatorname* {s i m} \left(f _ {\theta} \left(x _ {k} ^ {j}\right) , f _ {\theta} \left(x _ {k} ^ {i}\right)\right) / t}}{\sum_ {x \in B ^ {i} \cup B ^ {j} \backslash \{x _ {k} ^ {j} \}} e ^ {\operatorname* {s i m} \left(f _ {\theta} \left(x _ {k} ^ {j}\right) , f _ {\theta} (x)\right) / t}}, \tag {12}
$$
Then, we have the following results:
$$
\begin{array}{l} \underset {\theta} {\arg \min } \sum_ {x _ {k} \in U} \ell_ {\mathrm {C L}} \left(x _ {k} ^ {i}, x _ {k} ^ {j}; \theta\right) (13) \\ = \arg \min  _ {\theta} \sum_ {x _ {k} \in U} - \log \frac {e ^ {\operatorname* {s i m} \left(f _ {\theta} \left(x _ {k} ^ {i}\right) , f _ {\theta} \left(x _ {k} ^ {j}\right)\right) / t}}{\sum_ {x \in B ^ {i} \cup B ^ {j} \backslash \left\{x _ {k} ^ {i} \right\}} e ^ {\operatorname* {s i m} \left(f _ {\theta} \left(x _ {k} ^ {i}\right) , f _ {\theta} (x)\right) / t}} - \log \frac {e ^ {\operatorname* {s i m} \left(f _ {\theta} \left(x _ {k} ^ {j}\right) , f _ {\theta} \left(x _ {k} ^ {i}\right)\right) / t}}{\sum_ {x \in B ^ {i} \cup B ^ {j} \backslash \left\{x _ {k} ^ {j} \right\}} e ^ {\operatorname* {s i m} \left(f _ {\theta} \left(x _ {k} ^ {j}\right) , f _ {\theta} (x)\right) / t}} (14) \\ = \arg \max  _ {\theta} \sum_ {x _ {k} \in U} \log \frac {e ^ {\operatorname* {s i m} \left(f _ {\theta} \left(x _ {k} ^ {i}\right) , f _ {\theta} \left(x _ {k} ^ {j}\right)\right) / t}}{\sum_ {x \in B ^ {i} \cup B ^ {j} \backslash \{x _ {k} ^ {i} \}} e ^ {\operatorname* {s i m} \left(f _ {\theta} \left(x _ {k} ^ {i}\right) , f _ {\theta} (x)\right) / t}} + \log \frac {e ^ {\operatorname* {s i m} \left(f _ {\theta} \left(x _ {k} ^ {j}\right) , f _ {\theta} \left(x _ {k} ^ {i}\right)\right) / t}}{\sum_ {x \in B ^ {i} \cup B ^ {j} \backslash \{x _ {k} ^ {j} \}} e ^ {\operatorname* {s i m} \left(f _ {\theta} \left(x _ {k} ^ {j}\right) , f _ {\theta} (x)\right) / t}} (15) \\ = \arg \max  _ {\theta} \sum_ {x _ {k} \in U} \frac {e ^ {\operatorname* {s i m} \left(f _ {\theta} \left(x _ {k} ^ {i}\right) , f _ {\theta} \left(x _ {k} ^ {j}\right)\right) / t}}{\sum_ {x \in B ^ {i} \cup B ^ {j} \backslash \left\{x _ {k} ^ {i} \right\}}} e ^ {\operatorname* {s i m} \left(f _ {\theta} \left(x _ {k} ^ {i}\right) , f _ {\theta} (x)\right) / t} + \frac {e ^ {\operatorname* {s i m} \left(f _ {\theta} \left(x _ {k} ^ {j}\right) , f _ {\theta} \left(x _ {k} ^ {i}\right)\right) / t}}{\sum_ {x \in B ^ {i} \cup B ^ {j} \backslash \left\{x _ {k} ^ {j} \right\}}} (16) \\ = \arg \max  _ {\theta} \sum_ {x _ {k} \in U} p \left(\mathbf {1} _ {k j} \mid x _ {k} ^ {i}; \theta\right) + \sum_ {x _ {k} \in U} p \left(\mathbf {1} _ {k i} \mid x _ {k} ^ {j}; \theta\right) (17) \\ = \arg \max  _ {\theta} \sum_ {x _ {k} \in U} p \left(y _ {k i} ^ {R} \mid x _ {k} ^ {i}; \theta\right) + p \left(y _ {k j} ^ {R} \mid x _ {k} ^ {j}; \theta\right). (18) \\ \end{array}
$$
Therefore, we can conclude that SCL actually maximizes the conditional probability of the proxy label given the unlabeled data, i.e.,  $p(y^{R}|x)$ .
In the adversarial context, we only need to replace the natural data  $x_{k}^{i}$  and  $x_{k}^{j}$  in Eq. (13) and (18) with the adversarial data  $\tilde{x}_k^i$  and  $\tilde{x}_k^j$  generated in Eq. (3). Therefore, we have the following results:
$$
\begin{array}{l} \underset {\theta} {\arg \min } \sum_ {x _ {k} \in U} \ell_ {\mathrm {C L}} \left(x _ {k} ^ {i}, x _ {k} ^ {j}; \theta\right) + \ell_ {\mathrm {C L}} \left(\tilde {x} _ {k} ^ {i}, \tilde {x} _ {k} ^ {j}; \theta\right) \\ = \arg \max  _ {\theta} \sum_ {x _ {k} \in U} p \left(y _ {k i} ^ {R} \mid x _ {k} ^ {i}; \theta\right) + p \left(y _ {k j} ^ {R} \mid x _ {k} ^ {j}; \theta\right) + p \left(y _ {k i} ^ {R} \mid \tilde {x} _ {k} ^ {i}; \theta\right) + p \left(y _ {k j} ^ {R} \mid \tilde {x} _ {k} ^ {j}; \theta\right), \tag {19} \\ \end{array}
$$
where the adversarial data  $\tilde{x}_k^i$  and  $\tilde{x}_k^j$  are generated in Eq. (3). Therefore, from the causal view, in ACL [29], maximizing both the conditional probability of the proxy label given the natural data (i.e.,  $p(y^{R}|x)$ ) and that given adversarial data (i.e.,  $p(y^{R}|\tilde{x})$ ) is equivalent to minimizing the sum of standard and adversarial contrastive losses.
B.2 Proof of Lemma 2
Lemma 2 (restated). AIR in Eq. (7) can be decomposed into two terms as follows:
Proof. We transform AIR in Eq. (7) as follows:
Eq. (21) and (25) are obtained by using the KL divergence  $\mathrm{KL}(p(x)\| q(x);B) = \sum_{x\in B}p(x)\log \frac{p(x)}{q(x)}$  We obtained Eq. (27) by omitting the term  $\{x\}$  in Eq. (26) for simplicity and finished proving Lemma 2.
B.3 Proof of Proposition 3
Proposition 3 (restated). AIR implicitly enforces the robust representation to satisfy the following two proxy criteria:
$$
(1) \quad p ^ {d o (\tau_ {i})} (y ^ {R} | \tilde {x}) = p ^ {d o (\tau_ {j})} (y ^ {R} | \tilde {x}), \quad (2) \quad p ^ {d o (\tau_ {i})} (\tilde {x} | x) = p ^ {d o (\tau_ {j})} (\tilde {x} | x).
$$
Proof. To enforce the robust representation to satisfy the two proxy criteria shown in Eq. (28), we need to regulate the ACL with the following regularization
$$
\begin{array}{l} \operatorname {K L} \left(p ^ {d o \left(\tau_ {i}\right)} \left(y ^ {R} | \tilde {x}\right) \| p ^ {d o \left(\tau_ {j}\right)} \left(y ^ {R} | \tilde {x}\right); B\right) + \operatorname {K L} \left(p ^ {d o \left(\tau_ {i}\right)} (\tilde {x} | x) \| p ^ {d o \left(\tau_ {j}\right)} (\tilde {x} | x); B\right) (28) \\ = \sum_ {x \in B} p ^ {d o (\tau_ {i})} (y ^ {R} | \tilde {x}) \log \frac {p ^ {d o (\tau_ {i})} (y ^ {R} | \tilde {x})}{p ^ {d o (\tau_ {j})} (y ^ {R} | \tilde {x})} + \sum_ {x \in B} p ^ {d o (\tau_ {i})} (\tilde {x} | x) \log \frac {p ^ {d o (\tau_ {i})} (\tilde {x} | x)}{p ^ {d o (\tau_ {j})} (\tilde {x} | x)} (29) \\ = \sum_ {x \in B} \mathrm {K L} \left(p ^ {d o \left(\tau_ {i}\right)} \left(y ^ {R} | \tilde {x}\right) \| p ^ {d o \left(\tau_ {j}\right)} \left(y ^ {R} | \tilde {x}; \{x \}\right) + \sum_ {x \in B} \mathrm {K L} \left(p ^ {d o \left(\tau_ {i}\right)} (\tilde {x} | x) \| p ^ {d o \left(\tau_ {j}\right)} (\tilde {x} | x); \{x \}\right). \right. (30) \\ \end{array}
$$
Compared with Eq. (25) in the proof of Lemma 2, we can find that AIR also penalizes the same two KL divergence terms while AIR has two extra calibration terms  $p^{do(\tau_i)}(\tilde{x} |x)$  and  $p^{do(\tau_i)}(y^R |\tilde{x})$  that adjust the confidence of each KL divergence term, respectively. We empirically studied the impact of
the calibration term in Appendix C.8 and found that the calibration term is beneficial to improving the performance of ACL. Therefore, it indicates that AIR implicitly enforces the robust representation to satisfy the two proxy criteria shown in Eq. (28).
B.4 Proof of Proposition 4
Proposition 4 (restated). Let  $\mathcal{Y} = \{y_t\}_{t=1}^T$  be a label set of a downstream classification task,  $\mathcal{Y}^R$  be a refinement of  $\mathcal{Y}$ , and  $\tilde{x}_t$  be the adversarial data generated on the downstream task. Assuming that  $\tilde{x}_t \in \mathcal{B}_\epsilon[x]$  and  $\tilde{x} \in \mathcal{B}_\epsilon[x]$ , we have the following results:
$$
p ^ {d o (\tau_ {i})} (y ^ {R} | \tilde {x}) = p ^ {d o (\tau_ {j})} (y ^ {R} | \tilde {x}) \Longrightarrow p ^ {d o (\tau_ {i})} (y _ {t} | \tilde {x} _ {t}) = p ^ {d o (\tau_ {j})} (y _ {t} | \tilde {x} _ {t}) \quad \forall \tau_ {i}, \tau_ {j} \in \mathcal {T},
$$
$$
p ^ {d o (\tau_ {i})} (\tilde {x} | x) = p ^ {d o (\tau_ {j})} (\tilde {x} | x) \Longrightarrow p ^ {d o (\tau_ {i})} (\tilde {x} _ {t} | x) = p ^ {d o (\tau_ {j})} (\tilde {x} _ {t} | x) \quad \forall \tau_ {i}, \tau_ {j} \in \mathcal {T}.
$$
Proof. The proof is inspired by Mitrovic et al. [38].
For  $t\in \{1,\ldots ,T\}$  , we have
$$
\begin{array}{l} p ^ {d o \left(\tau_ {i}\right)} \left(y _ {t} \mid \tilde {x} _ {t}\right) = \int p ^ {d o \left(\tau_ {i}\right)} \left(y _ {t} \mid y ^ {R}\right) p ^ {d o \left(\tau_ {i}\right)} \left(y ^ {R} \mid \tilde {x} _ {t}\right) d y ^ {R} (31) \\ = \int p \left(y _ {t} \mid y ^ {R}\right) p ^ {d o \left(\tau_ {i}\right)} \left(y ^ {R} \mid \tilde {x} _ {t}\right) d y ^ {R} (32) \\ = \int p \left(y _ {t} \mid y ^ {R}\right) p ^ {d o \left(\tau_ {j}\right)} \left(y ^ {R} \mid \tilde {x} _ {t}\right) d y ^ {R} (33) \\ = p ^ {d o \left(\tau_ {j}\right)} \left(y _ {t} \mid \tilde {x} _ {t}\right). (34) \\ \end{array}
$$
Eq. (31) holds due to the fact that  $y^{R}$  is a refinement of  $y_{t}$  according to the definition of the refinement shown in Mitrovic et al. [38]. Eq. (32) holds since the relationship between  $y_{t}$  and  $y^{R}$  is independent of the style, i.e.,  $p^{do(\tau_i)}(y_t|y^R) = p^{do(\tau_j)}(y_t|y^R)$ . Eq. (33) holds because of the assumption that the prediction of adversarial data is independent of the style. Due to that the condition  $p^{do(\tau_i)}(y^R|\tilde{x}) = p^{do(\tau_j)}(y^R|\tilde{x})$  holds for any  $\tilde{x} \in \mathcal{B}_{\epsilon}[x]$ , thereby, this condition also holds for  $\tilde{x}_t \in \mathcal{B}_{\epsilon}[x]$ . Lastly, we obtain that the prediction of adversarial data will be still independent of the style factors on downstream tasks.
Next, due to that the condition  $p^{do(\tau_i)}(\tilde{x} |x) = p^{do(\tau_j)}(\tilde{x} |x)$  holds for any  $\tilde{x}\in \mathcal{B}_{\epsilon}[x]$ , this condition will hold for  $\tilde{x}_t\in \mathcal{B}_{\epsilon}[x]$  as well, i.e.,  $p^{do(\tau_i)}(\tilde{x}_t|x) = p^{do(\tau_j)}(\tilde{x}_t|x)$ . Therefore, the consistency between adversarial and natural data will be still invariant of the style factors on the downstream tasks.
C Extensive Experimental Details and Results
Experimental environments. We conducted all experiments on Python 3.8.8 (PyTorch 1.13) with NVIDIA RTX A5000 GPUs (CUDA 11.6).
Pre-training details of ACL [29]. Following Jiang et al. [29], we leveraged ResNet-18 [25] with the dual batch normalization (BN) [46] as the representation extractor, where one BN is used for the standard branch of the feature extractor and the other BN is used for the adversarial branch, during conducting ACL [29] and its variant DynACL [36]. We pre-trained ResNet-18 models using SGD for 1000 epochs with an initial learning rate 5.0 and a cosine annealing schedule [35]. During pre-training, we set the adversarial budget  $\epsilon$  as  $8 / 255$ , the hyperparameter  $\omega$  as 0.0, and the strength of data augmentation as 1.0. As for the reproduction of the baseline, we used the pre-trained weights published in the GitHub of ACL<sup>45</sup> as the pre-trained representation extractor for finetuning.
Pre-training details of DynACL [36]. The training configurations of DynACL [36] followed ACL [29], except for the strength of data augmentation and the hyperparameter  $\omega$ . We denote the
4Link of pre-trained weights via ACL on CIFAR-10.
5Link of pre-trained weights via ACL on CIFAR-100.
Table 9: The robust/standard test accuracy (%) achieved by ACL-AIR with different  ${\lambda }_{1}$  and  ${\lambda }_{2}$  .
λ2λ1	0.00	0.25	0.50	1.00
0.00	37.39/78.27	38.61/79.48	38.70/79.96	38.76/79.81
0.25	37.55/78.53	38.70/79.53	38.79/79.65	38.73/79.76
0.50	37.51/78.97	38.63/79.72	38.89/80.03	38.82/80.08
1.00	37.04/79.17	38.24/79.64	38.39/80.09	38.77/79.83
Table 10: Performance in semi-supervised settings evaluated on the CIFAR-10 task.
Label ratio	ACL [29]	ACL-AIR	DynACL [36]	DynACL-AIR
AA (%)	SA (%)	AA (%)	SA (%)	AA (%)	SA (%)	AA (%)	SA (%)
1%	45.63	74.84	45.97	76.63	45.78	76.89	46.25	78.57
10%	45.68	76.30	46.17	77.52	47.08	78.22	47.41	79.94
strength of data augmentation and the hyperparameter at epoch  $e$  as  $\mu_{e}$  and  $\omega_{e}$  respectively, where
$$
\mu_ {e} = 1 - \left\lfloor \frac {e}{K} \right\rfloor \cdot \frac {K}{E}, \quad e \in \{0, 1, \dots , E - 1 \} \tag {35}
$$
$$
\omega_ {e} = \nu \cdot \left(1 - \mu_ {e}\right), \quad e \in \{0, 1, \dots , E - 1 \} \tag {36}
$$
in which the decay period  $K = 50$ , the reweighting rate  $\nu = 2/3$ , the total training epoch  $E = 1000$ . In our implementation of DynACL, we only take the dynamic strategy and do not take the trick of the stop gradient operation and the momentum encode [24, 11]. As for the reproduction of the baseline, we downloaded the pre-trained weights published in the GitHub of DynACL $^{678}$  as the pre-trained representation extractor for finetuning.
Details of finetuning procedures. As for SLF and ALF, we fixed the parameters of the representation extractor and only finetuned the linear classifier using the natural training data and adversarial training data respectively. As for AFF, we finetuned all the parameters using the adversarial training data. For all finetuning procedures, we used SGD for 25 epochs for linear finetuning and full finetuning respectively and momentum is set as  $2e - 4$ . The initial learning of linear finetuning is set as 0.01 on CIAFR-10, 0.05 on CIFAR-100, and 0.1 on STL-10. The adversarial budget is fixed as  $8 / 255$  for ALF and AFF. In practice, we used the finetuning code published in the GitHub of DynACL for implementing finetuning procedures.
C.1 Performance in Semi-Supervised Settings [1]
Table 10 reports the performance evaluated in semi-supervised settings [1] during the finetuning procedure. In semi-supervised settings, following ACL [29] and DynACL [36], we first standardly finetuned the pre-trained model using the labelled data and then generated the pseudo labels using the standardly finetuned model. Then, we finetuned the model using the data with pseudo labels as well as the labelled data via AFF. The results validate that AIR  $(\lambda_{1} = 0.5,\lambda_{2} = 0.5)$  can consistently enhance both robust and standard test accuracy of ACL and its variant DynACL in semi-supervised settings.
C.2 Impact of the Hyper-Parameters  $\lambda_{1}$  and  $\lambda_{2}$
We show the performance achieved by ACL-AIR of different  $\lambda_{1} \in \{0.00, 0.25, 0.50, 1.00\}$  and  $\lambda_{2} \in \{0.00, 0.25, 0.50, 1.00\}$  on the CIFAR-10 task in Table 9. We can notice that when only leveraging adversarial regularization ( $\lambda_{1} = 0, \lambda_{2} > 0$ ), both standard and robust test accuracy gain significant improvement, which indicates that adversarial regularization serves an important role in improving the standard generalization and robustness of ACL. By incorporating adversarial and standard regularization together (when both  $\lambda_{1} > 0$  and  $\lambda_{2} > 0$ ), the standard generalization of ACL gets further improved. Table 9 guides us to set  $\lambda_{1} = 0.5$  and  $\lambda_{2} = 0.5$  since it can yield the best performance.
C.3 Incorporating AIR with Two Extra Variants of ACL
Here, we show extra results to validate that our proposed AIR can enhance two extra variants of ACL, including AdvCL [22] and AdvCL with A-InfoNCE [50].
6Link of pre-trained weights via DynACL on CIFAR-10.
7Link of pre-trained weights via DynACL on CIFAR-100.
8Link of pre-trained weights via DynACL on STL-10.
Table 11: Self-task adversarial robustness transferability evaluated on CIFAR-10 using various backbone networks including ResNet-34, ResNet-50, and WRN-28-10.
Network	Pre-training	AA (%)	SA (%)
ResNet-34	DynACL [36]	47.01	78.84
DynACL-AIR	47.56	80.67
ResNet-50	DynACL [36]	47.19	79.65
DynACL-AIR	47.82	81.27
WRN-28-10	DynACL [36]	41.13	71.23
DynACL-AIR	43.86	73.41
Table 12: Self-task adversarial robustness transferability evaluated on CIFAR-10 achieved by AdvCL [22], AdvCL-AIR, A-InfoNCE [50], and A-InfoNCE-AIR.
Pre-training	AdvCL [22]	AdvCL-AIR	A-InfoNCE [50]	A-InfoNCE-AIR
AA (%)	42.58	43.24	42.68	42.84
SA (%)	80.78	81.53	83.18	83.99
Table 13: Self-task adversarial robustness transferability evaluated on the CIFAR-10 dataset via SLF under various attacks including APGD-CE [13], APGD-DLR [13], FAB [14], and Square Attack [2].
Pre-training	APGD-CE (%)	APGD-DLR (%)	FAB (%)	Square Attack (%)
ACL [29]	39.66	41.18	39.27	48.98
ACL-AIR	40.39	40.98	41.01	49.80
DynACL [36]	46.28	46.41	45.56	50.06
DynACL-AIR	47.05	47.29	46.02	50.58
AdvCL [22] leverages an extra contrastive view of high-frequency components and the pseudo labels generated by the clustering method. We leveraged our proposed method IR ( $\lambda_1 = 0.5$ ,  $\lambda_2 = 0.5$ ) to further improve the performance of AdvCL. Yu et al. [50] proposed an asymmetric InfoNCE objective (A-InfoNCE) that treats adversaries as inferior positives that induce weaker learning signals, or as hard negatives exhibiting higher contrast to other negative samples.
We used the code provided by Yu et al. [50] in their GitHub to implement the pre-training of AdvCL as well as A-InfoNCE, and the finetuning procedure. In Table 12, we show that our proposed AIR  $(\lambda_{1} = 0.5,\lambda_{2} = 0.5)$  can consistently improve the robust and standard test accuracy of two extra variants [22, 50].
C.4 Applicability with Different Backbone Networks
In this subsection, we demonstrate that DynACL-AIR can consistently improve the performance on ResNet-34, ResNet-50 [25], and WRN-28-10 [51]. We followed the same training configurations of pre-training and finetuning (SLF) in Section 4 except for the backbone network. Table 11 validates that IR can further enhance the robust and standard test accuracy of DynACL on various backbone networks.
We observe that WRN-28-10 yields a worse performance compared to ResNet-50. We conjecture that it is because we set the batch size to 128 during pre-training due to the limitation of our GPU memory. We believe using a larger batch size during pre-training can further improve the performance of WRN-28-10 according to Chen et al. [9].
C.5 Robustness Evaluation under Various Attacks
In this subsection, we conducted the robustness self-transferability against three strong white-box attacks (APGD-CE [13], APGD-DLR [13] and FAB [14]) and one strong black-box attack (i.e., Square Attack [2]). We evaluate the robustness on the CIFAR-10 dataset via SLF and report the results in Table 13. The results validate that our proposed method can consistently improve robust test accuracy over various adversaries.
C.6 Test Accuracy under Each Type of Common Corruption
We report the test accuracy under each type of common corruption [26] with corruption severity being fixed as 5 in Table 14. We used pre-trained models on CIFAR-10 via DynACL and DynACL-AIR.
<sup>9</sup>GitHub provided by Yu et al. [50].
Table 14: Test accuracy evaluated on CIFAR-10-C under each type of common corruptions with corruption severity being fixed as 5 of CIFAR-10 pre-trained models after SLF and AFF.
Pre-training	Finetuning	Noise	Blur	Weather	Digital
Gaussian	Shot	Impulse	Defocus	Glass	Motion	Zoom	Snow	Frost	Fog	Bright	Contrast	Elastic	Pixel	JPEG
DynACL [36]	SLF	68.14	68.22	62.76	69.73	68.84	67.01	71.86	64.73	62.36	26.24	66.53	17.66	70.64	72.44	73.63
DynACL-AIR	70.10	70.47	64.40	69.79	71.18	67.90	72.10	66.72	63.94	30.16	67.27	18.21	72.43	75.17	76.80
DynACL [36]	AFF	74.02	74.62	66.87	71.44	73.21	69.22	73.68	67.83	66.29	27.59	69.27	18.72	74.60	77.50	79.05
DynACL-AIR	74.54	75.13	67.06	72.63	74.78	70.39	75.66	69.07	67.47	28.74	71.24	19.02	75.85	79.09	80.64
Table 15: Standard deviation of robustness self-transferability against common corruptions reported in Table 2.
Dataset	Pre-training	SLF	ALF	AFF
CS-1	CS-3	CS-5	CS-1	CS-3	CS-5	CS-1	CS-3	CS-5
CIFAR-10	ACL [29]	0.06	0.25	0.13	0.13	0.17	0.10	0.11	0.17	0.21
ACL-AIR	0.03	0.02	0.02	0.07	0.09	0.08	0.16	0.20	0.14
DynACL [36]	0.02	0.01	0.02	0.07	0.16	0.09	0.13	0.20	0.18
DynACL-AIR	0.01	0.03	0.03	0.07	0.11	0.14	0.16	0.11	0.10
CIFAR-100	ACL [29]	0.01	0.09	0.02	0.12	0.14	0.11	0.09	0.04	0.17
ACL-AIR	0.04	0.02	0.02	0.13	0.09	0.04	0.17	0.04	0.16
DynACL [36]	0.03	0.02	0.01	0.14	0.07	0.12	0.08	0.09	0.16
DynACL-AIR	0.08	0.05	0.02	0.08	0.12	0.05	0.16	0.16	0.10
Table 16: Standard deviation of robustness self-transferability against common corruptions reported in Table 4.
D1→D2	Pre-training	SLF	ALF	AFF
CS-1	CS-3	CS-5	CS-1	CS-3	CS-5	CS-1	CS-3	CS-5
CIFAR-10 → CIFAR-100	ACL [29]	0.05	0.02	0.03	0.07	0.09	0.05	0.15	0.10	0.31
ACL-AIR	0.01	0.04	0.04	0.09	0.12	0.10	0.11	0.16	0.15
DynACL [36]	0.04	0.04	0.05	0.11	0.07	0.07	0.19	0.18	0.10
DynACL-AIR	0.03	0.02	0.03	0.12	0.13	0.09	0.18	0.04	0.07
CIFAR-100 → CIFAR-10	ACL [29]	0.03	0.03	0.03	0.08	0.09	0.14	0.08	0.07	0.15
ACL-AIR	0.03	0.02	0.03	0.09	0.15	0.13	0.13	0.14	0.20
DynACL [36]	0.03	0.01	0.02	0.11	0.12	0.14	0.05	0.14	0.19
DynACL-AIR	0.02	0.08	0.02	0.06	0.08	0.09	0.14	0.21	0.17
Table 17: Impact of calibration terms evaluated on the CIFAR-10 task.
Pre-training	Calibration	SLF	ALF	AFF
AA (%)	SA (%)	AA (%)	SA (%)	AA (%)	SA (%)
ACL-AIR	×	38.55	79.80	40.80	77.57	49.51	81.95
ACL-AIR	✓	38.70	79.96	41.09	77.99	49.59	82.30
DynACL-AIR	×	45.09	77.79	46.01	76.12	50.54	82.35
DynACL-AIR	✓	45.23	78.01	46.12	77.01	50.66	82.62
Experimental settings are the same as Section 4. The results validate that IR can consistently improve the test accuracy under each type of common corruption.
C.7 Standard Deviation of Test Accuracy under Common Corruptions
Here, we provide the standard deviation of the test accuracy under common corruptions in Tables 15 and 16.
C.8 Impact of Calibration Terms
In the proof of Proposition 3, we show that there exist calibration terms  $p^{do(\tau_i)}(\tilde{x} |x)$  and  $p^{do(\tau_i)}(y^R |\tilde{x})$  in AIR. Here, we empirically investigate the impact of calibration on the performance in Table 17. The results empirically validate that the calibration term is an essential component of AIR to adjust the confidence of the KL divergence and further enhance the performance of ACL.
One of the limitations is that similar to previous works [29, 22, 50, 36], our proposed AIR is not helpful in improving the efficiency and scalability of ACL methods. Therefore, robust pre-training via
ACL-AIR is still unfriendly to the environment due to emitting much carbon dioxide and consuming much electricity. Besides, applying ACL-AIR to large-scale datasets such as ImageNet-21K [41] is still computationally prohibitive with limited GPU sources.
E Possible Negative Societal Impacts
Our proposed method aims to improve the performance of robust self-supervised pre-training methods [29, 22, 50, 36]. The robust self-supervised pre-training procedure is extremely time-consuming since it needs to spend much time generating adversarial training data, which thus leads to consuming a lot of electricity and emitting lots of carbon dioxide. Unfortunately, our proposed method is not helpful in improving the efficiency of ACL. Consequently, our study could exacerbate the greenhouse effect and be not conducive to environmental protection.