SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 11: Impact of batch size for coreset selection evaluated on the CIFAR-10 task.
Pre-training	Batch size for RCS	Runing time (hours)	SLF
SA (%)	RA (%)
ACL-Entire	-	42.8	78.87	39.19
ACL with Random	-	11.8	72.01	29.87
ACL with RCS	64	13.6	76.21	37.48
ACL with RCS	128	13.3	76.15	37.41
ACL with RCS	256	13.1	75.89	37.17
ACL with RCS	512	13.0	75.96	37.21
Table 12: Compatibility with AdvCL [15] evaluated on the CIFAR-10 task via SLF.
Pre-training	Runing time (hours)	SLF
SA (%)	RA (%)
AdvCL-Entire	57.8	80.89	42.36
AdvCL with Random	11.0	73.67	33.61
AdvCL with RCS	13.5	77.93	38.89
B.11 Efficient ACL via RCS with Various Batch Sizes
In this subsection, we show the impact of the batch size during coreset selection. We trained ResNet-18 via ACL with RCS on CIFAR-10 and then linearly finetuned ResNet-18 models on CIFAR-10. The batch size for RCS is selected from  $\{64, 128, 256, 512\}$  and the batch size for ACL keeps as 512. The subset fraction keeps as 0.2. Other training settings exactly follow Section 4.1. We report the standard and robust test accuracy in Table 11.
We can find that as the batch size for RCS decreases, the running time becomes larger. It is because there is a larger number of batches needed to calculate the loss gradients during RCS when the batch size becomes smaller. Besides, we observe that the test accuracy on the downstream tasks seems to gradually increase as the batch size decreases. Especially, ACL with RCS using 64 batch size gains consistent improvement compared with ACL with RCS using 512 batch size, which indicates that a smaller batch size for RCS is useful to improve the performance but consumes more running time. Therefore, there could be a trade-off between the running time and the transferability. We leave how to further improve the efficiency and effectiveness in maintaining the transferability of RCS as the future work.
B.12 RCS for Accelerating Another Variant AdvCL [15]
Fan et al. [15] proposed a variant of ACL method, called "AdvCL", that leverages a standardly pretrained model on ImageNet-1K to generate pseudo-labels for CIFAR-10 training data via  $K$ -means clustering. Based on ACL [14], the loss function of AdvCL is composed of a weighted sum of the adversarial contrastive loss and an ensemble of the CE loss between the adversarial data and its pseudo label over different choices of cluster number. By simply replacing the loss function of the ACL  $\mathcal{L}_{\mathrm{ACL}}(\cdot)$  with that of AdvCL in Algorithm 1 and 2, we can apply RCS for efficient AdvCL.
We pre-trained ResNet-18 via AdvCL with RCS on CIFAR-10 using SGD for 1000 epochs with an initial learning rate 0.5 and a cosine annealing scheduler. We set  $\beta = 512$ ,  $T_{\mathrm{ACL}} = 5$ ,  $\epsilon_{\mathrm{ACL}} = 8/255$ ,  $\rho_{\mathrm{ACL}} = 2/255$ , and  $T_{\mathrm{RCS}} = 3$ . The pre-training settings of AdvCL exactly follow Fan et al. [15]. We take the same configuration of RCS as that of RCS for ACL in Section 4.1, i.e., warmup epoch  $W = 100$ , the epoch interval for conducting RCS  $I = 20$ , and the subset fraction  $k = 0.1$ . Then, we evaluate the performance on the CIFAR-10 task via SLF.
Table 12 shows that RCS can speed up AdvCL [15] as well. Besides, RCS is a principled method that helps AdvCL to obtain effective robust representations since AdvCL with RCS always achieves a higher test accuracy compared with AdvCL with Random. Therefore, the experimental results
Algorithm 3 RCS for Supervised AT
1: Input: Labeled training set  $D$ , validation set  $U$ , batch size  $\beta$ , model  $g \circ f_{\theta}$ , learning rate for RCS  $\eta$ , subset fraction  $k \in (0,1]$ , loss function of supervised AT method  $\mathcal{L}_{supervised}(\cdot)$   
2: Output: Coreset  $S$   
3: Initialize  $S \gets \emptyset$ ,  $N \gets |D|$   
4: Split entire set into minibatches  $\{B_m\}_{m=1}^{[N/\beta]}$   
5: for each minibatch  $B_m \subset D$  do  
6: Compute gradient  $q_m \gets \nabla_\theta \mathcal{L}_{supervised}(B_m; \theta)$   
7: end for  
8: // Conduct greedy search via batch-wise selection  
9: for  $1, \ldots, [kN/\beta]$  do  
10: Compute gradient  $q_U \gets \nabla_\theta \mathcal{L}_{\mathrm{RD}}(U; \theta)$   
11: Initialize best_gain = -∞  
12: for each minibatch  $B_m \subset D$  do  
13: Compute marginal gain  $\hat{G}(B_m|S) \gets \eta q_U^\top q_m$   
14: if  $\hat{G}(B_m|S) > \text{best_gain then}$   
15: Update  $s \gets m$ , best_gain  $\leftarrow \hat{G}(B_m|S)$   
16: end if  
17: end for  
18: Update  $S \gets S \cup B_s$ ,  $D \gets D \setminus B_s$   
19: Update  $\theta \gets \theta - \eta q_s$   
20: end for
Algorithm 4 Efficient Supervised AT via RCS
1: Input: Labeled training set  $D$ , validation set  $U$ , total training epochs  $E$ , learning rate  $\eta'$ , batch size  $\beta$ , warmup epoch  $W$ , epoch interval for executing RCS  $I$ , subset fraction  $k$ , learning rate for RCS  $\eta$ , loss function of supervised AT method  $\mathcal{L}_{supervised}(\cdot)$   
2: Output: robust pre-trained feature extractor  $f_{\theta}'$   
3: Initialize parameters of model  $g \circ f_{\theta}'$   
4: Initialize training set  $S \gets D$   
5: // Warmup training for  $W$  epochs; Training on the coreset for  $(E - W)$  epochs  
6: for  $e = 0$  to  $(E - 1)$  do  
7: if  $e\% I == 0$  and  $e \geq W$  then  
8:  $\theta \gets \text{copy}(\theta')$   
9:  $S \gets \text{RCS}(D, U, \beta, g \circ f_{\theta}, \eta, k, \mathcal{L}_{supervised}(\cdot)) //$  by Algorithm 3  
10: end if  
11: for batch  $m = 1, \dots, \lceil |S| / \beta \rceil$  do  
12: Sample a minibatch  $B_m$  from  $S$   
13: Update  $\theta' \gets \theta' - \eta' \nabla_{\theta'} \mathcal{L}_{supervised}(B_m; \theta')$   
14: end for  
15: end for
validate that our proposed RCS can be a unified and effective framework for accelerating ACL as well as its variants [15, 17].
B.13 RCS for Accelerating Supervised AT
In this section, we first provide the algorithm of RCS for supervised AT including fast adversarial training (Fast-AT) [24], free adversarial training (Free-AT) [25], standard adversarial training (SAT) [34] and TRADES [35]. Then, we show the comparison between RCS and ACS [33] in speeding up supervised AT on CIFAR-10 and the effectiveness of RCS in efficient SAT on ImageNet-1K [1].
Algorithm of RCS for supervised AT. Prior to introducing the algorithm of RCS for SAT and TRADES, we first provide the preliminaries of SAT and TRADES.
Given a labeled training set  $D = \{(x_i, y_i)\}_{i=1}^N$ , where data  $x_i \in \mathcal{X}$  and label  $y_i \in \mathcal{Y} = \{0, 1, \dots, C-1\}$ , a feature extractor  $f_\theta : \mathcal{X} \to \mathcal{Z}$ , a randomly initialized classifier  $g : \mathcal{Z} \to \mathbb{R}^C$ ,
Table 13: Comparison between Fast-AT [24] with ACS [33] and Fast-AT with RCS on CIFAR-10.
Training method	Running time (minutes)	SA (%)	RA under PGD-20 (%)	RA under AutoAttack (%)
Fast-AT on the entire set	25.0	86.20	45.80	41.04
Fast-AT with ACS	12.8	82.71	45.71	40.92
Fast-AT with RCS	12.0	83.47	45.89	41.06
Table 14: Compatibility with Free-AT [25].
Training method	Running time (minutes)	SA (%)	RA under PGD-20 (%)	RA under AutoAttack (%)
Free-AT on the entire set	116.5	84.18	49.05	45.14
Fast-AT with RCS	60.6	82.39	49.24	45.15
the loss function of SAT is
$$
\mathcal {L} _ {\mathrm {S A T}} (D; \theta) = \sum_ {i = 1} ^ {N} \left\{\max  _ {\tilde {x} _ {i} \in \mathcal {B} _ {\epsilon} [ x _ {i} ]} \ell \left(g \circ f _ {\theta} \left(\tilde {x} _ {i}\right), y _ {i}\right) \right\}, \tag {28}
$$
where  $\ell$  is the Cross-Entropy (CE) loss and  $\tilde{x}_i$  is adversarial training data generated by PGD within the  $\epsilon$ -ball centered at  $x_i$ .
The loss function of TRADES is
$$
\mathcal {L} _ {\text {T R A D E S}} (D; \theta) = \sum_ {i = 1} ^ {N} \left\{\ell \left(g \circ f _ {\theta} \left(x _ {i}\right), y _ {i}\right) + c \cdot \max  _ {\tilde {x} _ {i} \in \mathcal {B} _ {\epsilon} \left[ x _ {i} \right]} K L \left(g \circ f _ {\theta} \left(\tilde {x} _ {i}\right), g \circ f _ {\theta} \left(x _ {i}\right)\right) \right\}, \tag {29}
$$
where  $\ell$  is the CE loss,  $KL(\cdot ,\cdot)$  is the KL divergence,  $c > 0$  is a trade-off parameter, and  $\tilde{x}_i$  is adversarial training data generated by PGD within the  $\epsilon$ -ball centered at  $x_{i}$ . We set  $c = 6$ , following Zhang et al. [35]. Note that the parameters of  $g$  are updated during supervised AT. Here we omit the parameters of  $g$  since we only use the parameters of the feature extractor  $f_{\theta}$  in downstream tasks.
The RCS problem for supervised AT is formulated as follows:
$$
S ^ {*} = \underset {S \subseteq D, | S | / | D | = k} {\arg \max } - \mathcal {L} _ {\mathrm {R D}} (U; \theta - \eta \nabla_ {\theta} \mathcal {L} _ {\text {s u p e r v i s e d}} (S; \theta)), \tag {30}
$$
in which we replace the ACL loss  $\mathcal{L}_{\mathrm{ACL}}(\cdot)$  in Eq. (5) with the supervised AT loss  $\mathcal{L}_{\mathrm{supervised}}(\cdot)$  (e.g.,  $\mathcal{L}_{\mathrm{SAT}}(\cdot)$  and  $\mathcal{L}_{\mathrm{TRADES}}(\cdot)$ ). Due to that  $\mathcal{L}_{\mathrm{RD}}(\cdot)$  only needs data and does not need any label, RCS is applicable to supervised AT, no matter if the validation set is unlabeled or labeled. By leveraging greedy search, we show the algorithm of RCS for supervised AT in Algorithm 3 and efficient supervised AT via RCS in Algorithm 4.
B.13.1 Comparison Between RCS and ACS [33] in Speeding Up Supervised AT including Fast-AT [24], SAT [34] and TRADES [35]
Comparison between Fast-AT with RCS and Fast-AT [24] with ACS [33]. We conducted Fast-AT, Fast-AT with ACS, and Fast-AT with RCS on CIFAR-10. The experimental setting of Fast-AT and Fast-AT with ACS exactly follows that in Section 4.3 of Dolatabadi et al. [33]. That is, we trained ResNet-18 on CIFAR-10 via Fast-AT ( $\epsilon = 8/255$ ,  $\alpha = 10/255$ ) using SGD with 0.9 momentum for 60 epochs with the initial learning rate of 0.1 and divided by 10 at Epoch 40 and 50. As for the RCS for speeding up Fast-AT, we set the subset fraction as  $k = 0.5$ , warmup epoch as  $W = 10$ , and epoch interval for executing RCS  $I = 10$ . We report the standard test accuracy and robust test accuracy evaluated by PGD-20 ( $\epsilon = 8/255$ ,  $\alpha = 8/2550$ ) and AutoAttack in Table 13. All experiments are conducted on one RTX A5000 GPU. Table 13 validate that RCS, without using labels, is more efficient and effective in speeding up Fast-AT.
Compatibility with Free-AT [25]. Further, we conducted Free-AT [25] and Free-AT with RCS on CIFAR-10. We trained ResNet-18 on CIFAR-10 via Free-AT ( $\epsilon = 8/255$ ) using SGD with 0.9 momentum for 60 epochs with the initial learning rate of 0.01 and divided by 10 at Epoch 40 and 50. We keep the configurations of RCS for speeding up Free-AT the same as above. We report the standard test accuracy and robust test accuracy evaluated by PGD-20 and AutoAttack in Table 14.
Therefore, RCS, without using label information, can further speed up both Fast-AT [24] and Free-AT [25] while almost maintaining the adversarial robustness. In addition, RCS without using labels is more efficient than ACS while achieving similar adversarial robustness compared with ACS. Therefore, it validates the effectiveness of RCS in efficient supervised AT.
Table 15: Comparison between SAT [34] with ACS [33] and SAT with RCS on CIFAR-10. Here, "RA" stands for the robust test accuracy under PGD-20 attacks following the setting of Dolatabadi et al. [33]. The number after the dash line denotes subset fraction  $k \in \{0.05, 0.1, 0.2\}$ .
Training method	Running time (minutes)	SA (%)	RA (%)
SAT on the entire set	314	82.49	52.11
SAT with Random-0.05	92	69.46	32.66
SAT with ACS-0.05	142	78.44	49.31
SAT with RCS-0.05	106	78.83	49.46
SAT with Random-0.1	100	73.69	35.42
SAT with ACS-0.1	151	79.08	50.43
SAT with RCS-0.1	115	80.17	50.54
SAT with Random-0.2	119	76.89	35.72
SAT with ACS-0.2	172	80.71	50.80
SAT with RCS-0.2	131	81.89	50.87
Table 16: Comparison between TRADES [35] with ACS [33] and TRADES with RCS on CIFAR-10. Here, "RA" stands for the robust test accuracy under PGD-20 attacks following the setting of Dolatabadi et al. [33]. The number after the dash line denotes subset fraction  $k \in \{0.05, 0.1, 0.2\}$ .
Training method	Running time (minutes)	SA (%)	RA (%)
TRADES on the entire set	332	82.72	54.35
TRADES with Random-0.05	111	74.56	40.12
TRADES with ACS-0.05	152	77.23	49.59
TRADES with RCS-0.05	123	77.52	50.28
TRADES with Random-0.1	122	75.80	42.19
TRADES with ACS-0.1	168	78.00	50.46
TRADES with RCS-0.1	136	78.59	50.83
TRADES with Random-0.2	146	77.41	44.07
TRADES with ACS-0.2	193	79.19	51.52
TRADES with RCS-0.2	161	79.82	51.84
Efficient SAT [34] and TRADES 29 via RCS on CIFAR-10. We trained ResNet-18 on CIFAR-10 via SAT [34] using SGD with 0.9 momentum for 120 epochs with the initial learning rate of 0.1 divided by 10 at Epoch 30 and 60. For PGD configurations during SAT, we set  $\beta = 128$ ,  $T_{\mathrm{SAT}} = 10$ ,  $\epsilon_{\mathrm{SAT}} = 8/255$ ,  $\rho_{\mathrm{SAT}} = 2/255$ . During RCS, we set  $T_{\mathrm{RCS}} = 3$ . We took  $W = 30$  epochs for warmup, and then CS was conducted every  $I = 10$  epochs with different subset fractions  $k \in \{0.05, 0.1, 0.2\}$ .
We trained ResNet-18 on CIFAR-10 via TRADES using SGD with 0.9 momentum for 100 epochs with the initial learning rate of 0.1 divided by 10 at Epoch 60 and 90. For PGD configurations during TRADES, we set  $T_{\mathrm{TRADES}} = 10$ ,  $\epsilon_{\mathrm{TRADES}} = 8 / 255$ ,  $\rho_{\mathrm{TRADES}} = 2 / 255$ . Other training settings follow SAT.
Here, we used ACS [33] and random selection (dubbed as "Random") as the baseline. The implementation of ACS exactly follows the public GitHub of ACS [33]. We use the same subset fraction and warmup epoch for both ACS and Random as RCS.
In Tables 15 and 16, we used the robust test accuracy of adversarial data generated via PGD-20 to evaluate adversarial robustness. It shows that RCS always achieves higher robust and standard test accuracy while consuming running time compared with Random and ACS [33] among various subset fractions. It validates the effectiveness of our proposed RCS in maintaining the adversarial robustness of supervised adversarial training.
B.13.2 RCS for Efficient Supervised Robust Pre-Training on ImageNet-1K
Effectiveness of RCS in speeding up STA on ImageNet-1K [1] and maintaining robustness transferability [58] of SAT. Hendrycks et al. [58] pointed out that the ImageNet-1K adversarially pre-trained models can improve the adversarial robustness on downstream tasks.
Following Hendrycks et al. [58], we pre-trained WRN-28-10 on ImageNet-1K of  $32 \times 32$  resolution using SAT [34]. The model was trained using SGD with 0.9 momentum for 100 epochs with an initial learning rate of 0.1 and a cosine decay scheduler. For PGD configurations during SAT, we
Table 17: Cross-task adversarial robustness transferability of adversarially pre-trained WRN-28-10 from ImageNet-1K to CIFAR-10. Here, "RA" stands for robust test accuracy under PGD-20 attacks following the setting of Hendrycks et al. [58]. The number after the dash line denotes subset fraction  $k \in \{0.05, 0.1, 0.2\}$ .
Pre-training	Runing time (hours)	ALF	AFF
SA (%)	RA (%)	SA (%)	RA (%)
Standard training on entire set	66.7	10.12	10.04	84.73	51.91
SAT [58] on entire set	341.7	85.90	50.89	89.35	59.68
SAT with Random-0.05	53.6	69.59	31.58	85.55	53.53
SAT with RCS-0.05	68.6	79.72	44.44	87.99	56.87
SAT with Random-0.1	70.2	73.28	33.86	86.78	54.95
SAT with RCS-0.1	81.9	81.92	45.10	88.87	57.69
SAT with Random-0.2	103.4	75.46	39.62	86.64	56.46
SAT with RCS-0.2	121.9	83.94	46.88	89.54	58.13
Table 18: Cross-task adversarial robustness transferability of adversarially pre-trained WRN-28-10 from ImageNet-1K to CIFAR-100. Here, "RA" stands for robust test accuracy under PGD-20 attacks following the setting of Hendrycks et al. [58]. The number after the dash line denotes subset fraction  $k \in \{0.05, 0.1, 0.2\}$ .
Pre-training	Runing time (hours)	ALF	AFF
SA (%)	RA (%)	SA (%)	RA (%)
Standard training on entire set	66.7	11.05	0.00	54.81	27.81
SAT [58] on entire set	341.7	63.35	32.08	61.56	35.64
SAT with Random-0.05	53.6	41.1	19.69	61.12	31.03
SAT with RCS-0.05	68.6	58.31	27.95	64.95	33.74
SAT with Random-0.1	70.2	50.57	20.99	63.6	31.97
SAT with RCS-0.1	81.9	59.92	28.11	66.12	34.41
SAT with Random-0.2	103.4	51.05	24.19	61.97	33.58
SAT with RCS-0.2	121.9	61.90	29.01	67.40	35.91
set  $\beta = 256$ ,  $T_{\mathrm{SAT}} = 10$ ,  $\epsilon_{\mathrm{SAT}} = 8 / 255$ ,  $\rho_{\mathrm{SAT}} = 2 / 255$ . We set  $T_{\mathrm{RCS}} = 3$  during RCS. We took  $W = 10$  epochs for warmup, and then CS was executed every  $W = 10$  epochs. We adversarially pretrain WRN-28-10 with different subset fractions  $k \in \{0.05, 0.1, 0.2\}$  and then conducted adversarial finetuning on CIFAR-10 and CIFAR-100.
To evaluate the cross-task adversarial robustness transferability of pre-trained WRN-28-10 models via SAT, we adversarially trained the pre-trained models on CIFAR-10 and CIFAR-100 for 30 epochs using SGD with the initial learning rate 0.01 and a cosine annealing scheduler. The learning objective of robust finetuning exactly follows  $\mathcal{L}_{\mathrm{Robust}}(\cdot)$  shown in Section 2.2. Note that we used the adversarially pre-trained WRN-28-10 on the entire set released in the GitHub of Hendrycks et al. [58] to reproduce the results of the baseline.
The results shown in Table 17 and 18 validate that RCS can significantly accelerate SAT on ImageNet-1K. Besides, SAT with RCS always obtains much better robustness transferability than SAT with Random and consistently achieves better transferability than standard pre-training as well. Therefore, it validates that RCS can be an effective method to speed up supervised robust pre-training as well.
Effectiveness of RCS in speeding up SAT on ImageNet-1K [1] and maintaining standard transferability [48] of SAT. Salman et al. [48] empirically discovered that adversarially pre-trained models on ImageNet-1K can yield higher standard test accuracy on downstream tasks than standardly pre-trained models. Following Salman et al. [48], we pre-trained ResNet-50 models on ImageNet-1K of  $224 \times 224$  resolution using SAT [34]. The model was trained using SGD with 0.9 momentum for 90 epochs with an initial learning rate 0.1 divided by 10 at Epoch 30 and 60. For PGD configurations during SAT, we set  $\beta = 256$ ,  $T_{\mathrm{SAT}} = 3$ ,  $\epsilon_{\mathrm{SAT}} = 4/255$ ,  $\rho_{\mathrm{SAT}} = \frac{3\epsilon_{\mathrm{SAT}}}{T_{\mathrm{SAT}}}$ . We set  $T_{\mathrm{RCS}} = 1$  during RCS. We took  $W = 10$  epochs for warmup, and then CS was conducted every  $I = 10$  epochs. We adversarially pre-trained ResNet-50 with different subset fractions  $k \in \{0.05, 0.1, 0.2\}$  and then conducted standard finetuning on downstream tasks i.e., CIFAR-10 and CIFAR-100.
To evaluate the cross-task standard transferability of pre-trained ResNet-50 via SAT on ImageNet-1K of  $224 \times 224$  resolution, we conducted standard linear finetuning (SLF) and standard full finetuning (SFF) on downstream tasks (i.e., CIFAR-10 and CIFAR-100) for 150 epochs using SGD. As for SFF,
Table 19: Cross-task standard transferability [48] of adversarially pre-trained ResNet-50 from ImageNet-1K to CIFAR-10 and CIFAR-100, respectively. We report the standard test accuracy (\%) via standard linear finetuning (SLF) and standard full finetuning (SFF). The number after the dash line denotes subset fraction  $k \in \{0.05, 0.1, 0.2\}$ .
Pre-training	Runing time (hours)	CIFAR-10	CIFAR-100
SLF	SFF	SLF	SFF
Standard training [48] on entire set	-	78.84	97.41	57.09	84.21
SAT [48] on entire set	286.1	93.53	98.09	77.29	86.99
Fast-AT [24] on entire set	10.4	90.91	97.54	73.35	83.33
SAT with Random-0.05	38.7	85.72	95.27	69.29	82.34
SAT with RCS-0.05	48.2	92.68	97.65	75.35	84.71
SAT with Random-0.1	45.8	87.14	95.60	71.23	83.62
SAT with RCS-0.1	55.4	92.92	97.82	75.41	85.22
SAT with Random-0.2	70.3	87.69	96.10	72.05	84.14
SAT with RCS-0.2	79.8	93.48	98.06	76.39	85.44
we finetuned all the parameters of the encoder. We set the initial learning rate of 0.001 for standard full finetuning and 0.01 for standard partial finetuning. The learning rate is divided by 10 at Epoch 50 and 100. Here, we use the same data augmentation as Salman et al. [48] to resize the images of CIFAR-10 and CIFAR-100 to  $224 \times 224$ . Note that we used the standardly pre-trained ResNet-50 on the entire set and adversarially pre-trained ResNet-50 on the entire set released in the GitHub of Salman et al. [48] to reproduce the results of baselines.
Table 19 shows that RCS substantially accelerates SAT on ImageNet-1K while consistently achieving higher standard test accuracy on downstream tasks than standard pre-training. Besides, SAT with RCS always obtains much higher standard test accuracy than SAT with Random. It validates that RCS is an effective method for efficient supervised AT.
In addition, we also provide a comparison between the standard transferability of Fast-AT [24] and SAT with RCS. We downloaded the pre-trained ResNet-50 vis Fast-AT released in the GitHub of Wong et al. [24]. Although SAT with RCS consumes more running time than Fast-AT, SAT with RCS obtains significantly higher standard test accuracy on downstream tasks, which validates the effectiveness of RCS in maintaining the transferability of supervised robust pre-training methods.
Therefore, our proposed RCS can be a unified and effective framework for speeding up both supervised and unsupervised robust pre-training while maintaining the transferability of pre-trained models.
C Possible Negative Societal Impacts
Our paper aims to speed up robust pre-training by decreasing the number of training data, which decreases the running time of pre-training and should be friendly to the environment. However, to implement all the experiments in this project, we have conducted the pre-training procedure at least 264 times and the finetuning procedure at least 2376 times, which definitely emitted a bunch of CO2 and consumed a large amount of electricity. Therefore, we hope that our project can help improve the efficiency of robust pre-training and make pre-training more environmental-friendly in the future.