SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \hat {G} _ {\theta} (x | S) \leq \eta \| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {R D}} (U; \theta) \| \| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (\{x \}; \theta) \| + \eta^ {2} \| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {R D}} (U; \theta) \| \| \frac {\partial^ {2}}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (S; \theta) \| \| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (\{x \}; \theta) \| \\ + \eta^ {2} \| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (S; \theta) \| \| \frac {\partial^ {2}}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {R D}} (U; \theta) \| \| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (\{x \}; \theta) \| + \| \frac {\partial}{\partial_ {\theta}} \xi_ {2} (\theta) \| \| \frac {\partial}{\partial_ {\theta}} \mathcal {L} _ {\mathrm {A C L}} (\{x \}; \theta) \| \\ + \left\| \xi_ {1} (\theta) \right\| + \sigma \\ \leq \eta M L _ {1} L _ {2} + \eta^ {2} M L _ {1} k N L _ {4} L _ {2} + \eta^ {2} k N L _ {2} M L _ {3} L _ {2} + \nu_ {2} L _ {2} + \nu_ {1} + \sigma \\ \leq \eta M L _ {2} \left(L _ {1} + \eta k N \left(L _ {1} L _ {4} + L _ {2} L _ {3}\right)\right) + \nu_ {2} L _ {2} + \nu_ {1} + \sigma . \\ \end{array}
$$
Since  $\sigma = \eta ML_2(L_1 + \eta kN(L_1L_4 + L_2L_3)) + \nu_2L_2 + \nu_1 + 1$ , we can obtain that
$$
\hat {G} _ {\theta} (x | S) \leq 2 \eta M L _ {2} \left(L _ {1} + \eta k N \left(L _ {1} L _ {4} + L _ {2} L _ {3}\right)\right) + 2 \nu_ {1} + 2 \nu_ {2} L _ {2} + 1 = 2 \sigma - 1 = \overline {{\hat {G}}} _ {\theta} (S). \tag {15}
$$
Based on the lower bound (Eq. (14)) and the upper bound (Eq. (15)) of the marginal gain function, we can obtain
$$
\begin{array}{l} \frac {\hat {G} _ {\theta} (x \mid A)}{\hat {G} _ {\theta} (x \mid B)} \geq \frac {\hat {G} _ {\theta} (S)}{\hat {\overline {{G}}} _ {\theta} (S)} = \frac {1}{2 \eta M L _ {2} \left(L _ {1} + \eta k N \left(L _ {1} L _ {4} + L _ {2} L _ {3}\right)\right) + 2 \nu_ {1} + 2 \nu_ {2} L _ {2} + 1} \\ = \frac {1}{2 \sigma - 1} = 1 - \alpha^ {*} \in (0, 1). \tag {16} \\ \end{array}
$$
Therefore, we have proved  $\hat{G}_{\theta}(S)$  is  $\alpha^{*}$ -submodular. According to Lemma 1,  $\hat{G}_{\theta}(S)$  is  $\gamma$ -weakly submodular where  $\gamma > \gamma^{*} = \frac{1}{2\sigma - 1}$ .
Theorem 2 (restated) Given a fixed parameter  $\theta$ , we denote the optimal solution of Eq. (5) as  $G_{\theta}^{*} = \sup_{S \subseteq X, |S| / |X| = k} G_{\theta}(S)$ . Then,  $\hat{S}^{*}$  in Eq. (6) found via greedy search satisfies
$$
G _ {\theta} (\hat {S} ^ {*}) \geq G _ {\theta} ^ {*} - \left(G _ {\theta} ^ {*} + k N \sigma\right) \cdot e ^ {- \gamma^ {*}}.
$$
Proof. We provide the lemma that describes the optimality guarantee of the greedy search algorithm in the problem of maximizing a monotone  $\alpha$ -approximate submodular function subject to cardinality constraints.
Lemma 2 ([37, 38]). Given a monotone and  $\alpha$ -approximate submodular function  $G(S)$ , the greedy search algorithm achieves a  $1 - e^{-(1 - \alpha)}$  approximation factor for the problem of maximizing  $G(S)$  subject to cardinality constraints.
Note that according to Theorem 1,  $\hat{G}_{\theta}(S)$  is a monotone and  $\alpha^{*}$ -approximate submodular function. Therefore, according to Lemma 2, we have
$$
\hat {G} _ {\theta} \left(\hat {S} ^ {*}\right) \geq \left(1 - e ^ {- \gamma^ {*}}\right) \hat {G} _ {\theta} ^ {*}, \tag {17}
$$
where  $\hat{G}_{\theta}^{*} = \sup_{S\subseteq X,|S| / |X| = k}\hat{G}_{\theta}(S)$  and  $\hat{S}^*$  is found via greedy search. Since  $\hat{G}_{\theta}(S) = G_{\theta}(S) + |S|\sigma$ , we have
$$
\hat {G} _ {\theta} ^ {*} = \sup  _ {S \subseteq X, | S | / | X | = k} (G _ {\theta} (S) + | S | \sigma) = \sup  _ {S \subseteq X, | S | / | X | = k} G _ {\theta} (S) + k | X | \sigma = G _ {\theta} ^ {*} + k N \sigma . \tag {18}
$$
Recall that  $\hat{G}_{\theta}(\hat{S}^{*}) = G_{\theta}(\hat{S}^{*}) + kN\sigma$ . Therefore, by transforming Eq. (17), we can obtain
$$
G _ {\theta} \left(\hat {S} ^ {*}\right) \geq G _ {\theta} ^ {*} - \left(G _ {\theta} ^ {*} + k N \sigma\right) \cdot e ^ {- \gamma^ {*}}. \tag {19}
$$
A.3 Derivation of Eq. (7)
Here, we provide the details of deriving Eq. (7) using Taylor expansion as follows:
$$
\begin{array}{l} \hat {G} _ {\theta} (B | S) = \hat {G} _ {\theta} (S \cup B) - \hat {G} _ {\theta} (S) (20) \\ = G _ {\theta} (S \cup B) + | S \cup B | \sigma - \left(G _ {\theta} (S) + | S | \sigma\right) (21) \\ = G _ {\theta} (S \cup B) - G _ {\theta} (S) + \beta \sigma (22) \\ = - \mathcal {L} _ {\mathrm {R D}} (U; \theta - \eta \nabla_ {\theta} \mathcal {L} _ {\mathrm {A C L}} (S \cup B; \theta)) + \mathcal {L} _ {\mathrm {R D}} (U; \theta - \eta \nabla_ {\theta} \mathcal {L} _ {\mathrm {A C L}} (S; \theta)) + \beta \sigma (23) \\ = - \mathcal {L} _ {\mathrm {R D}} (U; \theta - \eta \nabla_ {\theta} \mathcal {L} _ {\mathrm {A C L}} (S; \theta) - \eta \nabla_ {\theta} \mathcal {L} _ {\mathrm {A C L}} (B; \theta)) \\ + \mathcal {L} _ {\mathrm {R D}} (U; \theta - \eta \nabla_ {\theta} \mathcal {L} _ {\mathrm {A C L}} (S; \theta)) + \beta \sigma (24) \\ = - \mathcal {L} _ {\mathrm {R D}} (U; \theta_ {S} - \eta \nabla_ {\theta} \mathcal {L} _ {\mathrm {A C L}} (B; \theta)) + \mathcal {L} _ {\mathrm {R D}} (U; \theta_ {S}) + \beta \sigma (25) \\ \approx - \left(\mathcal {L} _ {\mathrm {R D}} (U; \theta_ {S}) - \eta \nabla_ {\theta} \mathcal {L} _ {\mathrm {R D}} (U; \theta_ {S}) ^ {\top} \nabla_ {\theta} \mathcal {L} _ {\mathrm {A C L}} (B; \theta) + \xi\right) + \mathcal {L} _ {\mathrm {R D}} (U; \theta_ {S}) + \beta \sigma (26) \\ \approx \eta \nabla_ {\theta} \mathcal {L} _ {\mathrm {R D}} (U; \theta_ {S}) ^ {\top} \nabla_ {\theta} \mathcal {L} _ {\mathrm {A C L}} (B; \theta) + \beta \sigma , (27) \\ \end{array}
$$
where  $B$  is a minibatch whose batch size is  $\beta \in \mathbb{N}$ ,  $S$  is the subset, and  $S$  and  $B$  are disjoint subsets. Eq. (20) holds according to the definition of marginal gain in Definition 1. Eq. (21) holds according to the definition of the proxy set function  $\hat{G}_{\theta}(S)$  in Theorem 1. Eq. (22) holds because  $S$  and  $B$  are disjoint. Eq. (23) holds according to the definition of set function in Eq. (4). Eq. (24) holds by splitting the ACL loss on  $S \cup B$  into the sum of the ACL loss on  $S$  and the ACL loss on  $B$ . Eq. (25) holds by letting  $\theta_{S} = \theta - \eta \nabla_{\theta} \mathcal{L}_{\mathrm{ACL}}(S; \theta)$ . In Eq. (26), we transform  $\mathcal{L}_{\mathrm{RD}}(U; \theta_{S} - \eta \nabla_{\theta} \mathcal{L}_{\mathrm{ACL}}(B; \theta))$  via Taylor expansion where  $\xi$  is the remainder. Eq. (27) holds by omitting the remainder. By replacing  $\theta_{S}$  with  $\theta - \eta \nabla_{\theta} \mathcal{L}_{\mathrm{ACL}}(S; \theta)$  in Eq. (27), we can obtain Eq. (7).
B Extensive Experimental Details and Results
B.1 An Extra Trick to Enable Efficient RCS on Large-Scale Datasets with Limited GPU Memory
RCS on large-scale datasets such as ImageNet-1K [1] needs to calculate the loss gradient for each minibatch of training data as the first step (Line 5-7 in Algorithm 1). Then, the greedy search is conducted to iteratively select the minibatch that has the largest marginal gain and add this minibatch into the final coreset (Line 9-20 in Algorithm 1). Saving the gradients of all the minibatches on GPU needs a large amount of GPU memory. When the GPU memory is limited, there exists an issue that we are unable to save all the gradients on GPU.
A direct solution is to transfer the loss gradients from GPU to CPU and save all the gradients on CPU. When we need to use the loss gradient of a minibatch, we can transfer the gradient from CPU to GPU
for calculation. Unexpectedly, the transition between CPU and GPU still consumes a large amount of time in practice. Thus, we do not take this solution.
To solve this issue, we split the entire training set into several training subsets. Practically, we take every 100 minibatches of training data as a training subset. Then, we conduct RCS on each training subset respectively and collect the coresets from each training subset together as the final coresets for robust pre-training. In this way, we can enable RCS to efficiently search for the coresets from large-scale datasets with limited GPU memory. We apply this trick to the experiments regarding ACL on ImageNet-1K in Section 4.2 and SAT on ImageNet-1K in Section 4.3.
B.2 Complementary Experimental Details
We conducted all experiments on Python 3.8.8 (PyTorch 1.13) with 4 NVIDIA RTX A5000 GPUs (CUDA 11.6).
Experimental details of ACL [14]. Following Jiang et al. [14], we leveraged the dual BN [61], where one BN is used for the standard branch of the feature extractor and the other BN is used for the adversarial branch, for the implementation of ResNet-18 and WRN-28-10 backbone models.
As for ACL using ResNet-18 on CIFAR-10 and CIFAR-100, we pre-trained ResNet-18 models using SGD for 1000 epochs with an initial learning rate 5.0 and a cosine annealing schedule [62]. During pre-training, we set the adversarial budget  $\epsilon_{\mathrm{ACL}} = 8 / 255$ , the step size  $\alpha_{\mathrm{ACL}} = 2 / 255$ , and the PGD step  $T_{\mathrm{ACL}} = 5$  for generating adversarial training data.
As for ACL using WRN-28-10 on ImageNet-1K, we pre-trained WRN-28-10 on ImageNet-1K of  $32 \times 32$  resolution via ACL [14] with RCS and Random, respectively. We set  $\beta = 512$ ,  $T_{\mathrm{ACL}} = 5$ ,  $\epsilon_{\mathrm{ACL}} = 8/255$ ,  $\rho_{\mathrm{ACL}} = 2/255$ ,  $T_{\mathrm{RCS}} = 1$ . The temperature parameter  $t$  was set as 0.1, following Chen et al. [13]. The model was trained using SGD for 200 epochs with an initial learning rate 0.6 and a cosine annealing scheduler. We set warmup epoch  $W = 200$ , and then CS was executed every  $I = 10$  epochs. We pre-trained WRN-28-10 with the subset fraction  $\mathbf{k} = 0.05$  and then finetuned it on CIFAR10 and CIFAR-100.
Experimental details of DynACL [17]. The pre-training settings of DynACL [17] followed ACL [14], except for the strength of data augmentation and the hyperparameter  $\omega$ . We denote the strength of data augmentation and the hyperparameter at epoch  $e$  as  $\mu_{e}$  and  $\omega_{e}$  respectively, where
$$
\mu_ {e} = 1 - \left\lfloor \frac {e}{K} \right\rfloor \cdot \frac {K}{E}, \quad e \in \{0, 1, \dots , E - 1 \},
$$
$$
\omega_ {e} = \nu \cdot (1 - \mu_ {e}), \quad e \in \{0, 1, \dots , E - 1 \},
$$
in which the decay period  $K = 50$ , the reweighting rate  $\nu = 2/3$ , the total training epoch  $E = 1000$ . In our implementation of DynACL, we only take the dynamic strategy and do not take the trick of the stop gradient operation and the momentum encode [63, 64].
Experimental details of finetuning. As for linear finetuning (SLF and ALF), we fixed the parameter of the encode and only trained the linear classifier using SGD with an initial learning rate 0.01 divided by 10 at Epoch 10 and 20 for 25 epochs. As for AFF, we trained the whole model for 25 epochs using SGD. The adversarial training data is generated via PGD with 10 PGD steps, adversarial budget 8/255, and step size 2/255. Practically, we exactly used the finetuning code provided in the GitHub of DynACL [17].
Baseline models that are pre-trained on the entire set. As for the pre-trained ResNet-18 on the entire set of CIFAR-10 and CIFAR-100 via ACL and DynACL, we used their released pre-trained weights in their GitHub<sup>3456</sup> to reproduce their results.
As for the pre-trained WRN-28-10 on the entire set of ImageNet-1K, due to the extremely long pre-training time of ACL on the entire set (as shown in Table 3), we do not provide the results of ACL on the full set of ImageNet-1K. We only provide the results of ACL on the coreset in Table 1 and 2.
3Link of pre-trained weights via ACL on CIFAR-10.
4Link of pre-trained weights via ACL on CIFAR-100.
5Link of pre-trained weights via DynACL on CIFAR-10.
6Link of pre-trained weights via DynACL on CIFAR-100.
Table 3: Running time (hours) on the entire training set of various models via ACL, DynACL, SAT, and TRADES.
Pre-training on the entire set	ResNet-18 on CIFAR-10	ResNet-50 on ImageNet-1K	WRN-28-10 on ImageNet-1K
ACL [14]	42.8	-	650.2
DynACL [17]	42.9	-	-
SAT [34]	5.2	286.1	341.7
TRADES [35]	5.5	-	-
Table 4: Self-task adversarial robustness transferability of ACL with RCS evaluated on the STL-10 dataset. The number after the dash line denotes subset fraction  $k \in \{ 0.05,0.1,0.2\}$  .
Pre-training	Runing time (hours)	SLF
SA (%)	RA (%)
ACL-Entire	74.4	71.16	33.21
ACL with Random-0.05	10.9	62.86	22.00
ACL with RCS-0.05	13.1	66.77	26.99
ACL with Random-0.1	13.9	64.54	24.16
ACL with RCS-0.1	16.4	68.17	28.66
ACL with Random-0.2	19.4	66.61	27.09
ACL with RCS-0.2	22.6	70.26	30.09
Table 5: Self-task adversarial robustness transferability of DynACL with RCS evaluated on the STL-10 dataset. The number after the dash line denotes subset fraction  $k \in \{ 0.05,0.1,0.2\}$  .
Pre-training	Runing time (hours)	SLF
SA (%)	RA (%)
DynACL-Entire	74.7	69.63	46.51
DynACL with Random-0.05	10.9	55.05	27.38
DynACL with RCS-0.05	13.2	63.75	37.01
DynACL with Random-0.1	14.1	58.77	30.72
DynACL with RCS-0.1	16.6	67.42	40.88
DynACL with Random-0.2	19.7	62.79	34.94
DynACL with RCS-0.2	22.9	69.14	43.27
As for pre-training via SAT [34] on the entire set of ImageNet-1K, we use WRN-28-10 pre-trained on ImageNet-1K of  $32 \times 32$  resolution provided by Hendrycks et al. [58] and ResNet-50 ( $\epsilon = 4.0$ ) pre-trained on the entire set of ImageNet-1K of  $224 \times 224$  resolution provided by Salman et al. [48].
Table 3 demonstrates the approximated training time on the entire set of various models via ACL [14], DynACL [17], SAT [34], and TRADES [35], respectively. All the running time was evaluated on NVIDIA RTX A5000 GPUs. We used 1 GPU for robust pre-training on CIFAR-10 and CIFAR-100, and 4 GPUs for pre-training on ImageNet-1K.
B.3 Self-Task Adversarial Robustness Transferability Evaluated on the STL-10 Dataset
Following the setting of [17], we applied our proposed method RCS to speed up ACL and DynACL on STL-10 [26] and report the results in Tables 4 and 5. The results validate that RCS is effective in speeding up both ACL and DynACL while being able to learn useful robust representations.
B.4 Experimental Details of Figure 1
In Figure 1, we report the cross-task robustness transferability of pre-trained ResNet-18 via ACL with RCS (subset fraction  $k = 0.1$ ) from CIFAR-10 to CIFAR-100 and STL-10 via SLF. The experimental settings are the same as Section 4.1.
B.5 Robustness Evaluated under Various Attacks
In this subsection, we report the adversarial robustness evaluated under three strong white-box attacks (APGD-CE [19], APGD-DLR [19] and FAB [55]) and one strong black-box attack (i.e., Square Attack [56]). We evaluate the cross-task adversarial robustness transferability from CIFAR-100 to STL-10 via ALF under various attacks and report the results in Table 6. The results validate that RCS can consistently improve robust test accuracy over various adversaries compared with Random and almost maintain the robustness transferability while significantly speeding up the pre-training.
Table 6: Adversarial robustness evaluated under various attacks including APGD-CE [19], APGD-DLR [19], FAB [55], and Square Attack [56]. The number after the dash line denotes subset fraction  $k = 0.1$ .
Pre-training	Running time (hours)	APGD-CE (%)	APGD-DLR (%)	FAB (%)	Square Attack (%)
ACL	42.8	29.70	25.55	24.60	28.61
ACL with Random-0.1	8.3	27.29	23.70	22.69	26.39
ACL with RCS-0.1	9.3	29.10	26.77	25.62	29.15
DynACL	42.9	31.35	27.41	26.76	28.61
DynACL with Random-0.1	8.3	26.56	22.44	21.44	25.39
DynACL with RCS-0.1	9.4	29.23	26.89	25.99	29.50
Figure 5: Average value of the KL divergence between natural test data and its adversarial variant.
B.6 Analysis of the Coreset Selected by RCS
Quantitative analysis of the coreset selected by RCS. Here, we provide comprehensive quantitative analyses of the coreset selected by RCS in Figures 5 and 6 to help understand the effectiveness of RCS in maintaining transferability.
In Figure 5, we demonstrate the average value of the KL divergence between natural data point  $x$  and its PGD-20 adversarial variant on the CIFAR-10 test set (i.e.,  $\ell_{\mathrm{RD}}(x)$ ). Specifically, we collected the average value of the KL divergence between natural data point  $x$  and its PGD-20 adversarial variant on the CIFAR-10 test set (i.e.,  $\ell_{\mathrm{RD}}(x)$ ) and show it in the leftmost panel of Figure 5. It validates that RCS helps minimize RD, thus helping ACL obtain useful and adversarily robust representations. It validates that RCS helps minimize RD, thus helping ACL obtain useful and adversarily robust representations.
In Figure 6, we demonstrate the imbalance ratio and the maximum mean discrepancy (MMD) between the coreset and the whole training set. Note that the imbalance ratio is the ratio of the sample size of the largest majority class and that of the smallest minority class. MMD is a classical measurement of the distance between two distributions. The left panel of Figure 6 shows that the corset selected by RCS is almost class-balanced since the imbalance ratio of RCS is slightly higher than 1.0. The right panel of Figure 6 shows that RCS yields a lower MMD between the entire training set and the selected coreset compared to Random. Therefore, our quantitative analysis demonstrates that RCS selects a coreset that is closer to the entire training set than Random, thus helping to maintain transferability.
Visualization analysis of the coreset selected by RCS. We count the frequency of each training sample in the CIFAR-10 dataset being selected into the coreset. In Figure 7, we visualize the top-5 most-frequently selected (MFS) training data and the top-5 least-frequently selected (LFS) training data. Figure 7 shows that, compared to LFS data, MFS data are images whose backgrounds are more complex and are more difficult to be differentiated from the subject. Recent work [65, 43, 66] has shown that exempting the representations from the nuisance style factors such as the background factor can improve the robustness against distribution shifts. RCS prefers to select images of complex backgrounds to help the model learn representations that are independent of the background factors, thus helping maintain robustness against adversarial perturbations.
B.7 Comparison Between ACL with FGSM and ACL with RCS
To the best of our knowledge, previously no one has studied how to incorporate FGSM [24, 18] with ACL to speed up ACL with FGSM. Previous studies [24, 25] only leveraged the trick for speeding up supervised AT where labels are required. Here, we take the primitive step to study the performance of FGSM in efficient ACL.
We demonstrate the comparison between ACL with FGSM and ACL with RCS on the CIFAR-10 task, which validates that our proposed RCS is more efficient and effective than FGSM [24].
Figure 6: We show the imbalance ratio of the coreset (left panel) and the maximum mean discrepancy (MMD) between coreset and the full training set (right panel) at each epoch. We conducted ACL with RCS and ACL with Random on the CIFAR-10 dataset using  $k = 0.2$ . The coreset is selected every  $I = 20$  epoch. We used a Gaussian filter ( $\sigma = 3$ ) to smooth the line.
Most-Frequently Selected (MFS) Data
Least-Frequently Selected (LFS) Data
Figure 7: We show the most-frequently selected data (upper panel) and the least-frequently selected data from the CIFAR-10 training data.
Table 7: The self-task adversarial robustness transferability achieved by ACL with FGSM and ACL with RCS on the CIFAR-10 dataset. The number after the dash line denotes the subset fraction  $k = 0.2$ .
Pre-training	Runing time (hours)	SLF	ALF
SA (%)	RA (%)	SA (%)	RA (%)
ACL-Entire	42.8	78.87	39.19	75.74	40.56
ACL with FGSM	21.7	86.70	0.00	60.04	0.11
ACL with RCS-0.2	13.0	75.96	37.21	74.32	38.68
DynACL-Entire	42.9	75.39	45.05	72.90	45.65
DynACL with FGSM	21.8	83.89	1.02	70.83	3.27
DynACL with RCS-0.2	13.1	75.16	44.32	72.10	45.75
We conducted pre-training on CIFAR-10 using ResNet-18 via both ACL and DynACL on the entire set with FGSM (denoted as "ACL with FGSM" and "DynACL with FGSM" respectively) and then conducted the linear evaluation on CIFAR-10. Following Fast-AT [24], we used one-step PGD with the adversarial budget as 8/255 and step size as 10/255 for FGSM. Other training and finetuning configurations (e.g., optimizer and learning rate scheduler) are the same as in Section 4.1.
In Table 7, we show the comparison between RCS (subset fraction  $k = 0.2$ ) and FGSM in speeding up ACL. We can observe that FGSM cannot effectively learn useful robust representations. In addition, RCS even consumes less running time than FGSM. Therefore, our proposed method RCS is more efficient and effective than FGSM [24] in speeding up ACL.
Table 8: Self-task adversarial robustness transferability achieved by ACL with RCS using various distance functions evaluated on the CIFAR-10 dataset. The number after the dash line denotes subset fraction  $k \in \{0.05, 0.1, 0.2\}$ .
Pre-training	Distance function	Runing time (hours)	SLF
SA (%)	RA (%)
ACL-Entire	-	42.8	78.87	39.19
ACL with Random-0.05	-	6.5	67.45	22.96
ACL with RCS-0.05	JS	7.7	72.39	32.43
ACL with RCS-0.05	OT	7.9	72.49	32.36
ACL with RCS-0.05	KL	7.6	72.56	32.49
ACL with Random-0.1	-	8.3	70.68	27.19
ACL with RCS-0.1	JS	9.4	74.43	34.23
ACL with RCS-0.1	OT	9.5	74.39	34.26
ACL with RCS-0.1	KL	9.3	74.67	34.30
ACL with Random-0.2	-	11.8	72.01	29.87
ACL with RCS-0.2	JS	13.2	75.75	37.18
ACL with RCS-0.2	OT	13.3	75.59	37.06
ACL with RCS-0.2	KL	13.0	75.96	37.21
Table 9: Self-task adversarial robustness transferability achieved by ACL with RCS using various warmup epochs evaluated on the CIFAR-10 dataset.
Pre-training	Warmup epoch	Runing time (hours)	SLF
SA (%)	RA (%)
ACL-Entire	-	42.8	78.87	39.19
ACL with Random	100	11.8	72.01	29.87
ACL with RCS	100	13.0	75.96	37.21
ACL with Random	200	15.2	73.38	30.59
ACL with RCS	200	16.3	76.43	38.33
ACL with Random	300	18.7	73.91	31.62
ACL with RCS	300	19.6	78.18	39.01
B.8 Efficient ACL via RCS with Various Distance Functions  $\Phi$
We pre-trained ACL with RCS using various distance functions on CIFAR-10 and then report the performance via linear evaluation on CIFAR-10 in Table 8. Other training settings exactly keep the same as Section 4.1, except for the distance functions.
It validates that efficient ACL via RCS with various distance functions can consistently obtain better transferability compared with Random. Especially, RCS with KL divergence often achieves better transferability to downstream tasks among different distance functions. Therefore, we use KL divergence for RCS in all the experiments in the main paper.
B.9 Efficient ACL via RCS with Various Warmup Epochs  $W$
This section provides the results of RCS with different warmup epochs  $W \in \{100, 200, 300\}$  for efficient ACL. We set the subset fraction  $k$  as 0.2. Other training settings follow Section 4.1. Table 9 shows that as the warmup epoch of RCS increases, the performance evaluated on the CIFAR-10 task becomes better while consuming more running time. Besides, RCS still consistently achieves better performance than Random, which validates the effectiveness of RCS in speeding up ACL while maintaining robustness transferability.
B.10 Efficient ACL via RCS with Various Epoch Intervals  $I$
We pre-trained ResNet-18 on CIFAR-10 via ACL with RCS using  $I \in \{10, 20, 50\}$ ,  $k = 0.2$ , and  $W = 100$ . Then, we evaluate the performance in the CIFAR-10 task via SLF. Table 10 shows that a larger  $I$  leads to more pre-training time since the frequency of conducting coreset selection becomes larger. Meanwhile, a larger  $I$  can lead to higher robust and standard test accuracy in downstream tasks. This is because a larger  $I$  enables the coreset to be updated more timely to adapt to the latest state of the model and select data points that can help the model improve its robustness.
Table 10: Impact of  $I$  for RCS evaluated on the CIFAR-10 task.
I	Runing time (hours)	SLF
SA (%)	RA (%)
10	15.6	76.31	38.17
20	13.0	75.96	37.21
50	12.2	75.87	35.54