Efficient Adversarial Contrastive Learning via Robustness-Aware Coreset Selection
Authors: Xilie Xu
Jingfeng Zhang
Feng Liu
Masashi Sugiyama
Mohan Kankanhalli
Venue: NeurIPS 2023 spotlight (2023)
Keywords: robust pre-training
adversarial contrastive learning
coreset selection

Abstract: Adversarial contrastive learning (ACL) does not require expensive data annotations but outputs a robust representation that withstands adversarial attacks and also generalizes to a wide range of downstream tasks. However, ACL needs tremendous running time to generate the adversarial variants of all training data, which limits its scalability to large datasets. To speed up ACL, this paper proposes a robustness-aware coreset selection (RCS) method. RCS does not require label information and searches for an informative subset that minimizes a representational divergence, which is the distance of the representation between natural data and their virtual adversarial variants. The vanilla solution of RCS via traversing all possible subsets is computationally prohibitive. Therefore, we theoretically transform RCS into a surrogate problem of submodular maximization, of which the greedy search is an efficient solution with an optimality guarantee for the original problem. Empirically, our comprehensive results corroborate that RCS can speed up ACL by a large margin without significantly hurting the robustness transferability. Notably, to the best of our knowledge, we are the first to conduct ACL efficiently on the large-scale ImageNet-1K dataset to obtain an effective robust representation via RCS. Our source code is at https://github.com/GodXuxilie/Efficient_ACL_via_RCS.

Efficient Adversarial Contrastive Learning via Robustness-Aware Coreset Selection
Xilie Xu' X Jingfeng Zhang Feng Liu', Masashi Sugiyama Mohan Kankanhalli
School of Computing. National University of Singapore
2 RIKEN Center for Advanced Intelligence Project (AIP)
School of Computer Science, The University of Auckland
School of Computing and Information Systems, The University of Melbourne
Graduate School of Frontier Sciences, The University of Tokyo jingfeng. zhang@auckland ac nz fengliu. ml @gmail sugi @k u-tokyo.ac acc.yyooc....... jjp mohan@comp. nus edu.sg mus.edu.ss.
xuxilie@comp nus, edu. ag
Abstract
Adversarial contrastive learning (ACL) does require expensive data annotations but outputs robust representation that withstands adversarial attacks and also generalizes to a wide range of downstream tasks, However, ACL needs tremendous rumning time to generate the adversarial variants of all training data, which limits scalability to large datasets. speed up ACI., this paper proposes robustness- aware coreset selection (RCS) method. RCS does not require label information and searches for an informative subset that minimizes a representational divergence, which is the distance of the representation between natural data and their virtual adversarial variants. The vanilla solution of RCS via traversing all possible subsets is computationally prohibitive. Therefore, we theoretically transform RCS into a surrogate problem of submodular maximization, of which the greedy search is an efficient solution with optimality guarantee for the original problem. Empirically, our comprehensive results corroborate that RCS can speed up ACL by large margin without significantly hurting the robustness transferability. Notably, to the best of our knowledge, we are the first to conduct ACL efficiently on the large- scale ImageNet-1K KK dataset obtain effective robust representation via RCS. Our source code is at aalhtpss/ggtuuubboom/oooo Cooeuuubbooommiiiouullll/eiiiinn ACC_ ACL_via_RCCS RCS)
Introduction
The pre-trained models can be easily finetuned to downstream applications, recently attracting
increasing attention [1 2 3 Notably, vision Transformer [5] pre-trained on ImageNet-1K K
1
can achieve state-of-the-un performance on many downstream computer-vision applications 66 7
Foundation models [8] trained on large-scale unlabeled data (such GPT 9 and CLAP (10) can be adapted to a wide range of downstream tasks. Due to the prohibitively high cost of annotating large-scale data, the pre-trained models commonly powered by the techniques of unsupervised learning [1 [2] in which contrastive learning (CL) is the most effective learning style to obtain the
generalizable feature representations [3
Adversarial Cl. (ACI.) 14 16 17 that incorporate adversarial data with contrastive loss, can
yield robust representation that is adversarially robust. Compared with the standard CL. [3), ACL can output robust representation that can transfer some robustness to the downstream tasks against
The first two authors have made equal contributions.
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Robustness transferalilitty from CIFAR- CIFAR, 100
Robustmess transferalillity from CIEAR-IO STL-10
Robust test seemarr
More efficient!
Robust test geeurary
28-
More efficient!
27-
10
26
25
More robust!
More robust!
24-
23-
27-
21
45
25
20
15
10
45
25
20
15
10
Ramning time (houns)
Ramning time (hours)
ACL or entire set DynACL on enrize oct
ACL with rundom: selection
ACL with ICC (ocns)
DynACL with random sclection
DyaACL with BCS (ours)
Figure We learn representation using CIFAR-10 66 dataset (without requiring labels) via ACL [4] and DynACL [17]. Then, we evaluate the representation's 's robustness transferability to CIFAR-I00 100 [6] and STL-I0 [26] (using labels during finetuning) via standard linear finetuning. We demonstrate the rumning time of robust pre-training w.r.t. different coreset selection (CS) strategies and report the robust test accuracy under AutoAttack [19]. Our proposed RCS (star shapes) always achieves higher robust test accuracy than random selection (triangles) while consuming much less running time than pre-traine tr-inriiing on the entire set (squares). Experimental details are in Appendix [B.4]
adversarial attacks [8 19 via finetuning. The robustmess transferability is of great importance to a pre-trained model's practicality in safety-critical downstream tasks (20 21 22 23].
However, ACL is computationally expensive, which limits its scalability. At each training epoch, ACL first needs to conduct several backward propagations (BPs) on all training data to generate their adversarial variants and then train the model with those adversarial data. Even if use the techniques of fast adversarial training [24] 25 that significantly reduce the need for conducting BPs per data, ACL still encounters the issue of the large scale of the training that is commonly used for pre-training a useful representation.
Coreset selection (CS) [27
that selects small yet informative subset can reduce the need for the whole training set, but CS cannot be applied to ACL directly. Mirzasoleiman et al. [29] and
28
32 have significantly accelerated the standard training by selecting an informative training subset. Recently, Dolatabadi et al. 33
Killamsetty et al. 30
proposed an adversarial CS method 24 by selecting representative subset that can accurately align the gradient of the full set. However, the existing CS methods require the label information of training data, which are applicable the ACL that learns from the unlabeled data.
to accelerate the standard adversarial training
334
35
To accelerate the ACL., this paper proposes a robustness-awarc coreset selection (RCS) that selects coreset without requiring the label information but still helps ACL to obtain effective robust representation. RCS searches for a coresct that minimizes the representational divergence (RD) of a representation. RD measures the representation difference between the natural data and their virtual adversarial counterparts 36), in which virtual adversarial data can greatly alter the output distribution in the sense of feature representation.
Although solving the RCS via traversing all possible subsets is simple, it is computationally pro- hibitive. Therefore, we transform the RCS into proxy problem of maximizing a set function that theoretically shown monotone and 7-weakly submodular (see Theorem subject to cardinality constraints 37 38 Then, we can apply the greedy search (37 to efficiently find coreset that can minimize the RD. Notably, Theorem 2 provides the thcoretical guarantee of optimality of our greedy-search solution.
Empirically, we demonstrate RCS can indeed speed up ACL 14 and its variants 17 As shown in Figure [ RCS can speed up both ACL [14] and DynACL [7 by 4 times and obtain an effective representation without significantly hurting the robustness transferability (details in Section 4.1). Notably, to the best of our knowledge, we are the first to apply ACL on the large-scale ImageNet-1K K N dataset to obtain a robust representation via RCS (see Section 4.2 Besides, we
empirically show that RCS is compatible with standard adversarial training
34 35|24| as well in
Section 4.3 Our comprehensive experiments corroborate that our proposed RCS a unified and principled framework for efficient robust learning.

---

2
Background and Preliminaries
In this section, we introduce related work and preliminaries about contrastive learning
13
4
2.1 Related Work
Contrastive learning (CL), CL approaches are frequently used to leverage large unlabeled datasets for learning useful representations. Previous unsupervised methods that map similar samples to similar representations [3 have the issue of collapsing to a constant representation. CL addresses the
representational collapse by introducing the negative samples [2. Chen et al. [13] presented SimCLR that leverages the contrastive loss for learning useful representations and achieved significantly improved accuracy on the standard suit of downstream tasks. Recently, adversarial contrastive learning (ACL) [40 14,
[7 43) that incorporates adversarial training with the contrastive loss 1 has become the most effective unsupervised approaches learn robust representations. Jiang et al. [4] showed that ACL could exhibit better adversarial robustness on downstream tasks compared with standard CL (i.e., SimCLR [3). Luo et al. [17] proposed to dynamically schedule the strength of data augmentations to narrow the gap between the distribution of training data and that of test data, thus enhancing the performance of ACL. Xu et al. [3] improved ACL from the lens of causality results.
41 15 I6, 42
45 and proposed an adversarial invariant regularization that achieves new SOTA
However, ACL needs large computational resources to obtain useful representations from large-scale datasets. ACL needs to spend a large amount of running time generating adversarial training data during pre-training. Fast adversarial training (Fast-AT) (24] uses the fast gradient descent method (FGSM) [8] accelerate the generation procedure of adversarial training data, thus speeding up robust training. A series of recent works 46 47 followed this line and further improved the performance of fast AT. However, the time complexity of ACL is still proportional the size of the training set. Besides, the large-scale training sets (e.g., ImageNet- IK [I] contains over 1 million training images) further contribute to the inefficiency of adversarially pre-training procedures 48 To enable efficient ACI,, we propose a novel CS method that can speed up ACL by decreasing the amount of training data.
Coreset selection (CS). CS aims to select a small yet informative data subset that can approximate
certain desirable characteristics (e.g., the loss gradient) of the entire set [28
Several studies
have shown that CS effective for efficient standard training in supervised 22
30 and semi-
supervised (32 settings. Dolatabadi al. (33) proposed adversarial corcset selection (ACS) that selects representative coresets of adversarial training data that can estimate the adversarial loss gradient on the entire training set. However, ACS is only adapted to supervised AT [34 (35). Thus, superv
ACS requires the label information and is not applicable to ACL. In addition, previous studies did not explore the influence of CS on the pre-trained model's transferability. To this end, we propose novel CS method that does not require label information and empirically demonstrate that our proposed method can significantly accelerate ACL while only slightly hurting the transferability.
2.2 Preliminaries
Here, we introduce the preliminaries of contrastive learning [3 14 17]. input space X' with the infinity distance metric doo (2, doe (xe, x²) S <) be the closed ball of radius > 0 centered at x E X.
Let (X, dow) be the x' lloo, and [e[x] [x]] {o X
We first introduce the standard contrastive loss [3. Let fo : X Z be feature extractor parameterized by 0. : Z - V be a projection head that maps representations to the space where the contrastive loss is applied, and T; Tj X - X' be two transformation operations randomly sampled from a pre- drfined transformation set T. Given a minibatch B ~ X consisting (zx),, (Ix) Vark B} consisting of 23 samples. We take ha)-) = g fo(-) and xa = (rxx for any Ig ~ X' and u E (1.j}. The standard contrastive loss of a positive pair (at, .1)) is as follows:
ACL 14] and DynACL [17].
of [3 samples, denote the augmented minibatch B'
( ho)
(et (4;)= -- Σ log
Σ
wE(
>
where sim(p. =p [Tq/lpll|aa|| is the cosine similarity function and t > 0 temperature parameter.
3

---

Given an unlabeled set X ~ XN consisting of N samples, the loss function of ACL 14 is
LACL (X; Σx= AACL (xz; where
(ACU(TT::) 0) = max
((10++11100 (1 (1+w) ( #f:) + (1 w) lci (a 01;0),
in which E [0, 1] is hyperparameter and ii. and dig are adversarial data generated via projected
gradient descent (PGD) 34] within the e-balls centered at oi and z². Note that ACL [4] fixes
while DynACL [7] dynamically schedules w according to its dynamic augmentation scheduler that gradually anneals from a strong augmentation
weak one. We leave the details of the data augmentation scheduler [17] in Appendix B.2 due to the limited space.
Given an initial positive pair (z<(0) ).)))) PGD step E N, step size p > 0, and adversarial budget 2 0, PGD iteratively updates the pair of data from = to T - 1 as follows:
(+((+1) = 11g, [at (o0::00((t (zt((t) + p sign V 20 (x x
()(++1) -Ils. (x5.00 (15.(7) + sign V a lee (w+(T)) xd(()))
where [x] projects the data into the e-ball around the initial point x.
ACL is realized by conducting one step of inner maximization on generating adversarial data via PGD and one step of outer minimization updating / by minimizing the ACL loss on generated adversarial data, alternatively. Note that the parameters of the projection head g are updated as well during ACL. Here omit the parameters of g for notational simplicity since we only the parameters of feature extractor fo on downstream tasks after completing ACL.
3 Robustness-Aware Coreset Selection
In this section, we first introduce the representational divergence (RD), which is quantified by the distance of the representation between natural data and their virtual adversarial variants, as the measurement of the adversarial robustness of representation. Then, we formulate the learning objective of the Robustness-awaz Coreset Selection (RCS). Next, we theoretically show that our proposed RCS can efficiently solved via greedy search. Finally, we give the algorithm of efficient ACL via RCS.
3.1
Representational Divergence (RD)
Adversarial robustness is the most significant property of adversarially pre-trained models, which could lead to enhanced robustness transferability, Inspired by previous studies [35 55) we measure the adversarial robustness of the representation in an unsupervised manner using the representational divergence (RD). Given natural data point X' and a model fe X V composed of feature extractor fo and projector head g. RD this data point εDD (z; 8) is quantified by the distance between the representation of the natural data and that of its virtual adversarial counterpart (36], i.c.
LND (a; 0) = d(g o fo(3), g fo(x)) s.t. f =
arg max d(g fo(a') g fo())),
x'EB. [x]
in which
PGD method [5]] generate adversarial data ă within the e-ball centered at and d(..:) V x + R is a distance function, such as the (JS) divergence [52]. and the optimal transport (OT) distance [50]. We denote the RD on the unlabeled validation set as LRD (U: = Σx, eu ÊRD (z,; 0). The smaller the RD is, the representations are of less sensitivity adversarial perturbations, thus being more robust.
(KL) divergence [35),
the
3.2 Learning Objective of Robustness-Aware Coreset Selection (RCS)
Our proposed RCS aims to select an informative subset that can achieve the minimized RD between natural data and their adversarial counterparts, thus expecting the selected coresct to be helpful in improving the adversarial robustness of representations. Therefore, given an unlabeled training set X ~ N and unlabeled validation set U X (M < N), proposed Robustness-aware Coreset Selection (RCS) searches for a coreset 5* such that
S*
arg min
CRD(U; arg min (SCL(S;

---

Note that the subset fraction k E (0, controls the size of the coreset, i.e.. |St| KN. RCS only needs calculate RD on the validation set (i.e., (U)) and ACL loss on the subset (i.e., LACI. (S)). which can be computed in an unsupervised manner. Thus, RCS is applicable to ACI. on unlabeled datasets, and compatible with supervised AT on labeled datasets as well, such as Fast-AT [24].
SAT [3 and TRADES [35] (details in Appendix [B.3). Intuitively, the coreset S found by RCS can make the pre-trained model via minimizing the ACL loss CACL(S") achieve the minimized CRD(U) thus helping attain adversarially robust presentations, which could be beneficial to the robustness transferability to downstream tasks.
To efficiently solve the inner minimization problem of Eq. 22. we modify Eq. ( by conducting one-step gradient approximation as the first step, which keeps the same practice as [31 30 i.c.
S*
arg min sgx, S1/IK|Sk
(RD(U; 0 (S:)cc
where 7j 0 is the learning rate. We define set function 2 + R as follows:
Go(S x) a LuD(U; (V
We fix the size of the coreset, i.e., | k|X| being constant. Then, the optimization problem in Eq. (2) can be reformulated as
S arg max Go( SCX,[S|/XX|=
(5)
3.3 Method-Greed Search
Solving Eq. (5) is formulated as maximizing set function subject to cardinality |S| constraint
53
31 [1]
naive solution to this problem to traverse all possible subsets of size kN and select the subset S which has the largest value Go But this naive solution is computationally prohibitive since it needs to calculate the loss gradient VoLACL (S;;) for (EN) times. Fortunately, Das and Kempe (37] and Gatmiry and Gomez- Rodriguez 38) proposed the greedy search algorithm to efficiently solve this kind of problem approximately if the set function is monotone and "- --wakkly submodular. Note that the greedy search algorithm shown in Algorithm oonly needs to calculate the loss gradient [ + [ < (AN) times where is batch size.
Definition 1 (Monotonicity and 7-weakly submodularity [37 (44). Given a set function G : 2 - R, the marginal gain of G is defined as Gxx))A) & G(A U {x}) G(A) any A C X and x E X \ A. The sel function G
The set function is monotone if (ar 2 0 for any C and x E
called w-weakly submodular if E.eB G( Z (GAA B) G(A)) for some y E [0, 1] and any disjoint subset A, B S X.
Assumption 1. The first-order gradients and the second-order gradients of lRD and lACL are bounded w.r.t. 0, ie.,
① 0g (uD(=;(0]l S L,, 0g where Lt, La, Lg, and La are positive constants.
02 (xcL(x; S L2, 0g (nD(2; S Ls, fACL (x; La,
0²
Next, we theoretically show that proxy set function Go(S) is monotone and 7-weakly submodular in Theorem 1
Theorem 1. We define a proxy set function Go(S) & Gs(S) + Slo. where o =1+ + 21 + v2L2 LL 11 + 0+, and 12 > 0 are positive constants. Given Assumption (S(S) is monotone and 7-weakly submodular where 7 -
The proof is in Appendix A.1
We construct proxy optimization problem in Eq. 5) as follows:
S
arg max
Ge(S))
(6)
SCX|iS|/IX|=
can be leveraged to approximately solve the proxy problem Eq. (6) and can provide the following optimality guarantee of the optimization problem in Eq. (5)
According to Theorem a greedy search algorithm 37
38
5

---

Algorithm Robustness-aware Coreset Selection (RCS)
1: Input: Unlabeled training set X, unlabeled validation set U, batch size B, model g fe. learning rate for RCS 7/, subset fraction & E (0, 1]
2: Output: Coreset S
3: Initialize + 0, + [X]
4: Split entire training set into minibatches {Bm [N/8]
5: for each minibatch Bm C X do
6: Compute gradient Im (Bmi
7: end for
8: // Conduct greedy search via batch-wise selection
9: for I, 1,...., / do
Compute gradient qu t VOLRD(U;0)
11: Initialize best beit_gainnt = -0
for each minibatch Bm X do Compute marginal gain G(B,,.15 t nbn Im // refer Eq. if G Bm (S) > best__gaie then
13:
15:
Update +- m, best_ best_gaiinn____e gain t- ((mm)s)
16:
end if
17:
end for
18:
Update S S-SUBB B2, - X\A.
19:
Update 0 + -17ss
20:
end for
Theorem 2. Given a fixed parameter 0, we denote the optimal solution of Eq. suPsçx,sS//XI== Go(S). Then, S in Eq. (6) found via greedy search satisfies
5 as G
Go 2 G3 (Gg kNo) e
Remark. The proof is in Appendix A.2 Theorem 2 indicates that the greedy search for solving the
proxy problem in Eq. (6) can provide guaranteed fower-bound of the original problem in Eq. ( which implies that RCS via greedy search can help ACL to obtain the minimized RD and robust representations. We also empirically validate that ACL the coreset selected by RCS achieve lower RD compared with ACL. on the randomly selected subset Figure 5 (in Appendix B.6), which empirically supports that RCS via greedy search effective in achieving the minimized RD.
Algorithm of RCS. Therefore, RCS. Algorithm ittraaes the batch-wise selection [ | times. At each iteration, finds the
greedy search algorithm via batch-wise selection for
minibatch B that has the largest gain Ga (B[S) based on the parameters updated by the previously selected coreset, and then adds this minibatch into the final coreset. RCS via greedy search needs to calculate the gradient for each minibatch [N/3] times (Line 6 in Algorithm] I and the gradient on the validation set [N/33] times (Line 10 in Algorithm T}. total, RCS needs to calculate the loss gradient [N/3] + [ < (+%) times, which is significantly more efficient than the native solution. We approximate the marginal gain function using the Taylor expansion as follows:
(B ~
VoLACI.
3a,
where B is a minibatch and 3 is the batch size. The derivation of Eq. 7 is in Appendix A.3 It enables efficiently calculate the marginal gain for each minibatch (Line 13 Algorithm I We omit this term in Algorithm since 30 is a constant. Intuitively, RCS selects the data whose training loss gradient and validation loss gradient are of the most similarity. In this way, the model can minimize the RD (validation loss) after updating its parameters by minimizing the ACL loss (training loss) on the coreset selected by RCS, thus helping improve the adversarial robustness the representation.
3.4 Efficient ACL via RCS
We show an efficient ACL procedure via RCS Algorithm ACI with RCS trains the model on the previously selected coreset for I N epochs, and for every epochs new coresct is selected.

---

Algorithm 2 Efficient ACL via RCS
1: Input:
Unlabeled training set X, unlabeled validation set U, total training epochs E, learning rate 1/,, batch size 3, warmup epoch W, epoch interval for executing RCS I, subset fraction learning rate for RCS 17
2: Output: Adversarially pre-trained feature extractor for
3: Initialize parameters of model g fø
4: Initialize training set <- X
5: // Warmup training for W epochs; Training on the coreset for (E W) epochs
6: for e === 0 to (E 1) do and W then
7: if' e%
8:
0 - copy(0"
9:
S RCS(X, 3, k) // by Algorithm
10:
end if
for batch m I.... [||//8] do
12: Sample minibatch Bm from S Update 6' 0' v/ CAcI,
14: end for
end for
The pre-training procedure is repeated until the required epoch E E N is reached. We provide four tricks with three tricks as follows and one trick that enables efficient RCS on large-scale datasets with limited GPU memory in Appendix B.1
Warmup on the entire training set. We take W epochs train the model on the entire training set the warmup. Warmup training enables the model to have a good starting point to provide informative gradients used in RCS. For example, we use 10% of the total training epochs for warmup.
Last-layer gradients. It is computationally expensive to compute the gradients deep model due to a tremendous number of parameters in the model. To tackle this issue, we utilize a last-layer gradient approximation, by only considering the loss gradients of the projection head during RCS.
full layers of
Adversarial data approximation. Calculating adversarial data during CS is extremely time- consuming since generating adversarial data needs to iteratively perform BP I times. We let TACL be PGD steps, EACL be the adversarial budget, and PACL be the step size for PGD during ACL. Similarly, TRCS, fncs, and Pres denote PGD configurations during RCS. To mitigate the above issue, we decrease TRCS (i.e., 0 < TRCS S TACL) and PRCS adversarial data during CS. Note that the same adversarial budget is used for ACL and RCS, i.c., ERCS EACL.
Pace PAcI- -Acu. for efficiently generating Tgcss
4 Experiments
In this section, we first validate that our proposed RCS can significantly accelerate ACL 14 and its variant (17[55) on various datasets [6 26 with minor degradation in transfcrability to downstream tasks. Then, we apply RCS to large-scale dataset, i.c., ImageNet-IK [1 to demonstrate that RCS enhances the scalability of ACL. Lastly, we demonstrate extensive empirical results. Extensive experimental details are in Appendix B.2 B.2
Efficient pre-training configurations. using ResNet-18 backbone networks. The pre-training settings of ACL and DynACL exactly follow their original paper and we provide the details in Appendix B.2 For the hyperparameters
We leverage RCS to speed up ACL 14 and DynACL 17
RCS, we set 512, 77 0.01, and TRCS = 3. We took W 100 epochs for warmup. and then CS was executed every I = 20 epoch. We used different subset fractions & E (0.05, 0.1, 0.2} for CS. The KL divergence was used the distance function calculate CRD(3 (.) for all the experiments in Section 4 We used the pre-trained weights released in ACL's and DynACL's GitHub as the pre-trained encoder reproduce their results on the entire set. We repeated all the experiments using different random seeds three times and report the median results to exclude the effect of randomization.
Finetuning methods. We used the following three kinds of finetuning methods to evaluate the learned representations: standard linear finetuning (SLF), adversarial linear finetuning (ALF), and
7

---

SLF on CIEAB-W0
SUI oe CEPAR--00
ALE de CIEAR-100
ALE on CIPAR-IO 3C3
AFF us CIFAR-HO
AFF us CIEAR-UM
Sharkee Lei
Sclext
16-
4.
Spoodiop nito
Spending miti
Speed-up molo
Speod-ppesst extt
Sgood--p up rallo
Sgeed-rp min
STL:10
STL-10
S1L-30
STL-10
STL-10
STL.1
Stankod best orcuur
anteel Lox. serimay
Sa Sarkre bett perntey
Robo lot UOCG
1
x
12
2 8
Rabrot tast
22
19
8
Speederp
ScLoes Spend-tpp
Spacietp
Figure 2: Cross-task adversarial robustness transferability from CIFAR-10 10 to CIFAR-100 (upper row) and STL-10 (bottom row). The number after the dash line denotes subset fraction l E {0.05, 0.2. ACL with RCS and DynACL with RCS correspond to the red and orange solid lines, respectively. ACL with Random and DynACL with Random correspond to the blue and green dotted lines, respectively.
SLF CIFMR-30
SLF oa CIFAR-10
ALF au UIFAR-10
ALF on CIEMRUI
AFF on CHPAR-10
APP on CIRFA8--1
Stndee le
beok
22
2
Spood up atto
Speodiug mit0
Speei-up mico
Spool-up.cat callo
Spood up sallo
Speed uD mioo
SLF in S11-10
SLF on STT-16
S11-10
ST1-10
STTL.10
STL-10
Stanled tot seouner
Botea Los AECINA
Stan Lan to ansuncy
Bubert fet mermyy
Wanninn tot serreyy
Hot tel GECuuCy
18
18
3 3
22
Speedirp
Spestep
Spectup
Speclsrp
ACL BCS44
Figure 3: Cross-task adversarial robustness transferability from CIFAR- 100 to CIFAR-10 (upper row) and STL-10 (bottom row). The number after the dash line denotes subset fraction k E 0.1,
adversarial full finetuning (AFF). SLF and ALF will keep the learned encoder frozen and only finetune the linear classifier using natural or adversarial samples, respectively. AFF leverages the pre-trained encoder as weight initialization and trains the whole model using the adversarial data. The finetuning settings exactly follow DynACL [17. More detials in Appendix B.2
Evaluation metries. The reported robust test accuracy (dubbed as "RA"") is evaluated via AutoAt- tack (AA) 19 and the standard test accuracy (dubbed as "SA") is evaluated on natural data, In practice, we used the official code of Auto/Attack [19] for implementing evaluations. In Appendix B.5
we provide robustness evaluation under more diverse attacks 19 55 56].
2) with the random selection strategy (dubbed as "Random") to obtain the coreset S during ACL the baseline. The implementation of Random exactly follows that in the coreset and data selection library 31
Baseline. We replaced RCS (Line 9 in Algorithm
Speed-up ratio. The speed-up ratio denotes the ratio of the running time of the whole pre-training procedure on the entire set (dubbed as "Entire") to that of ACL with various CS strategies (i.e., Random and RCS). We report the running time of Entire in Table 3
4.1 Effectiveness of RCS in Efficient ACL
Cross-task adversarial robustness transferability. Figures 2 and 3 demonstrate the cross-task adversarial robustness transferability from CIFAR-10 and CIFAR-100 to downstream tasks, respec- tively. First, we observe that ACI/DynACL, with RCS (red/orange lines) always achieves significantly higher robust and standard test accuracy than ACL/DynACL with Random (blue/green lines) among
8

---

SLF du CHEAR-10
SLP ua CIPAR-10 10
ALF un CIPAN--1O
ALF on CHEAR-10
AFF us CIFAR-1O
AFF cu. CIEAB-10
Sulaad occusay
3ulvet bot EXNEERO
St
425-
412
Suerber Les
17.5-
8 2
1 %
25.2
225
35
3
4
6
Speec-t milio
Speot-up ratlo
Speetiup mico
Speed up rstlo
Speed-ng mih3
Spood ratlo
CIEMB-10
CHEAR-10)
CIBAR-00
Sharkin best orcausy
Rdh lod acsuaay
Bobrot tat artraay
Sta Aurl fest orrmny
Rob lot UNEORNY
10 R 5
5 11 2
Speederp
Spaniotpo
Figure 4: Self-task adversarial robustness transferability evaluated on the CIFAR- 10 (left three panels) and CIFAR- 100 (right three panels) dataset. The number after the dash line denotes subset fraction k E (0.05, 0.1,0.2).
various subset fractions while achieving a similar speed-up ratio to Random. Besides, compared with ACL/DynACL on the entire set (black/brown squares), ACL/DynACL with RCS (red/orange lines) can almost maintain robust and standard test accuracy on downstream tasks. Notably, RCS-0.2 almost maintains robust test accuracy on various downstream tasks via various finetuning methods. validates the effectiveness of our principled method RCS in maintaining robustness transferability while speeding up pre- pre-training.
Self-task adversarial robustness transferability. Figure 4 shows the self-task adversarial ro- bustness transferability evaluated on CIFAR-10 and CIFAR-TOW 100 datasets where pre-training and finetuning were conducted on the same datasets. In Appendix B.3 we provide the performance on the S0L-110 26 task in Tables 4 and 5 Apparently, ACI/DDnACI. with RCS (red/orange lines) always yields better performance across tasks than ACL/DynACL with Random (blue/green lines). Besides, RCS (solid lines) can almost maintain robust test accuracy via various finetuning methods compared with Entire (squares) while having high speed-up ratio. validates that RCS is effective in efficiently learning robust representations.
4.2 Benchmarks on ImageNet-11K
To the best of our knowledge, we are the first to apply ACL [4] on ImageNet-1K efficiently via RCS. As shown in Table I ACL on the entire set of ImageNet- 1K will need about 650.2 hours using 4 NVIDIA RTX A5000 GPUs, which is extremely time-consuming and unmanageable for us to conduct ACL on the entire set. Thus, we do not provide the results of ACL on the entire set.
Tables T and 2 demonstrate the cross-task adversarial robustness transferability of pre-trained WideResNet 57 with width 10 and depth 28 (WRN-28-100 10) from ImageNet- IK of 32 x 32 resolution to CIFAR-I0 and CIFAR- 100, respectively. Experimental details of pre-training are in Appendix [B.2 The two tables demonstrate that ACL with RCS consistently leads to better robust and standard test accuracy on downstream tasks than ACL with Random. Besides, we can observe that ACL with RCS often obtains better robustness transferability compared with standard CL. Surprisingly, ACL with RCS even consumed less running time (111.8 hours) than standard CL (147.4 44 hours) while achieving better transferability. It validates that RCS effectively enhances the scalability of ACL, and efficient ACL via RCS can yield useful and robust representations.
4.3 Extensive Experimental Results
Ablation study. We provide the results of RCS using various distance functions in Appendix B.8 RCS with various warmup epochs W in Appendix [B.9 RCS with various epoch intervals for executing RCS in Appendix 10, RCS with various batch sizes during CS in Appendix B.1 B. and RCS for another variant Ad [15 in Appendix B.12
Comparison between ACL with FGSM and ACL with RCS. To the best of our knowledge, no work has studied how to use FGSM (one-step PGD) [24] to speed up ACL. In Table demonstrate

---

Table 1: Cross-task adversarial robustness transferability from ImageNet-1K CIFAR- 10.
Runing (hours)
SLF
ALF
AFF
Pre-training
(%)
RA (%)
SA (%)
(%)
(%)
RA (%)
Standard CL
147.4
84.3610.17
1.01
10.00
10000
86.63
19.7
ACL.on on entire set
650.2
ACL with Random
94.3
68.751006
59.57.1.28 10.8
84.755.0.8 27.14
12.121
ACL with RCS
111.8
70.02=0.12
22.45=0013
63.94+021
31.13=017
85.23+0023
=012
Table 2: Cross-task adversarial robustness transferabilitty from ImageNet- to CIFAR-100.
Runing time
SLF
ALF
AFF
Pre-training
(hours)
SA (%)
RA (%)
SA (%)
RA (%)
SA (%)
RA (%)
Standurd CL
147.4
57.34 20.3
0.01
9.32=000
0.06:0001
61.33
25.11
ACL on entire set
650.2
ACL with Random
94.3
38.531005
10.50.1o.3
28.44.0023
11,931021
59.63
25,4610026
ACI. with RCS
111.8
40,28=0.1
14.55+0.10
33.15+0.6
14.89=0.16
60.25+0018
28.24
that FGSM cannot effectively learn robust representations and consumes more running time than RCS. Therefore, our proposed RCS is more efficient and effective in speeding up ACL than FGSM.
following Hendryeks et al. 58 and Salman et al. [48), we apply RCS to speed up pre-training WRN-28-10 and ResNet-50 via ImageNet- K [ and demonstrate that SAT with RCS can almost maintain transferability. Therefore, RCS can be a unified and effective framework for efficient robust pre-training.
RCS for efficient supervised robust pre-training. In Appendix B.1..2
SAT 34
Comparison between RCS and ACS [33] in speeding up supervised AT including Fast-AT [24), SAT (34), and TRADES (35]. We provide the comparison between RCS and ACS (33 in Ap-
pendix 13.1 which validates that our proposed RCS, without using labels during CS, is more
efficient and effective than ACS in terms of speeding up supervised AT'
24 34 35
Analysis of the coreset selected by RCS. We provide comprehensive quantitative and visualization analyses the coreset selected by RCS Appendix [B.6 The analyses demonstrate that RCS helps minimize RD and tends to select coreset that is closer to the entire training set compared to Random, thus helping ACL obtain useful and adversarially robust representations.
Conclusion
This paper proposed robustness-awan coreset selection (RCS) framework for accelerating robust pre-training. RCS found an informative subset that helps minimize the representational divergence between natural data and their adversarial counterparts. We theoretically showed that RCS can be efficiently solved by greedy search approximately with an optimality guarantce. RCS does not require label information and is thus applicable to ACL as well as supervised AT. Our experimental results validated that RCS can significantly speed up both ACL and supervised AT while slightly hurting the robustness transferability.
One of the limitations is that RCS still requires spending particular amount of running time during coreset selection in calculating the loss gradients, although we theoretically propose the greedy-search algorithm with an optimality guarantee make RCS efficient. We leave how further improve efficiency of RCS, such as by leveraging better submodular function optimization methods [59] 60). as the future work.
Acknowledgements
This research is supported by the National Research Foundation, Singapore under its Strategic Capability Research Centres Funding Initiative, Australian Research Council (ARC) under Award No. DP230101540 and NSF and CSIRO Responsible Al Program under Award No. 2303037. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
10

---

References
[1] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Iece, 2009.
[2] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21K pretraining for the masses, arXiv preprint arXiv: 104. 10972, 2021.
[3] Jiahua Dong, Yang Cong, Gan Sun, Bineng Zhong, and Xiaowei Xu. What can be transferred: Unsupervised domain adaptation for endoscopic lesions segmentation. In CVPR, 2020.
[4] Jiahua Dong, Yang Cong, Gan Sun. Zhen Fang, and Zhengming Ding. Where and how to transfer: Knowledge aggregation-induee adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1(2):1118, 2021.
transferability perception for unsupervised domain
doi: 10.1 10.1009/PAAM..2011...3 2206.8000......
[5] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
[6] Alex Krizhevsky. Learning multiple layers features from tiny images. Technical report, 2009.
[7] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 191000467 2019.
[8] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0722 2021.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-19011 2020.
[10] Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap: Learning audio concepts from natural language supervision. arXiv preprint arXiv:2206.04769, 2022.
[11] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[12] Phuc H Kh-Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 8: 19999977199934,, 2020.
[13] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1599-1607. PMLR, 2020.
[14] Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. Advances in Neural Information Processing Systems, 33:16199-16210, 2020.
[15] Lijie Fan, Sijia Liu, Pin- Yu Chen, Gaoyuan Zhang, and Chuang Gan. When does contrastive learning preserve adversarial robustness from pretraining to finetuning? Advances in Neural Information Processing Systems, 34:21480-21492 22492, 2021.
[16] Qiying Yu, Jieming Lou, Xianyuan Zhan, Qizhang Li, Wangmeng Zuo, Yang Liu, and Jingjing Liu. Adversarial contrastive learning via asymmetric infonce. In European Conference on Computer Vision, pages 53-69. Springer, 2022.
[17] Rundong Luo, Yifei Wang, and Yisen Wang. Rethinking the effect of data augmentation in adversarial contrastive learning. In The Eleventh International Conference on Learning Representations, 2023. URI.https: httpt:t/t/:::oneeeee http::///ppereiieeeee wwt.://ooummiddootoee oddmmm?ddDDmmuuNNyyyyx.
[18] Ian J. Goodfellow, Jonathon Shiens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In Yoshua Bengio and Yann LeCun, editors, The Third International Conference on Learning Representations, 2015.
11

---

[19] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning. PMLR, 2020.
pages
[20] Varun H Buch, Irfan Ahmed, and Mahiben Maruthappu. Artificial intelligence in medicine: current trends and future possibilities. British Journal of General Practice, 68(668):143-144, 2018.
[21] Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security, pages 99-112. Chapman and Hall/CRC, 2018.
[22] Xu Yang, Cheng Deng, Kun Wei, Junchi Yan, and Wei Liu. Adversarial learning for robust deep clustering. Advances in Neural Information Processing Systems, 33:9098-9108, 2020.
[23] Zhiyuan Dang. Cheng Deng, Xu Yang, Kun Wei, and Heng Huang. Nearest neighbor matching for deep clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13693-13702, 2021.
[24] Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In ICLR, 2020.
[25] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry& S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In NeurlPS, 2019.
[26] Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[27] Pankaj Agarwal, Sariel dele,e, Kasturi R Varadarajan, et al. Geometric approximation via coresets. Combinatorial and computational geometry, 52(1):1-30, 2005.
[28] Dan Feldman. Core-sets: Updated survey. In Sumpling techniques for supervised or unsuper- vised taskx, pages 23-44. Springer, 2020.
[29] Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pages 6950-6960.
PMIR, 2020.
[30] Krishnateja Killamsetty, S Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad- match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pages 5464-5474. PMLR, 2021.
[31] Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh Iyer. Glister: Generalization based data subset selection for efficient and robust learning. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11108118. 2021.
[32] Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust supervieee learning. Advances in Neural Information Processing Systems, 24:11488-14501, 2021.
[33] Hadi M Dolatabadi, Sarah Erfani, and Christopher Leckie. Adversarial coreset selection for efficient robust training. arXiv preprint MXXi:22009007885. 2022.
[34] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[35] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML. 2019.
[36] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and supe-vupe learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-18 1993, 2018.
[37] Abhimanyu Das and David Kempe. Submodular meets spectral: Greedy algorithms for subset selection, sparse approximation and dictionary selection. In ICML, 2011.
[38] Khashayar Gatmiry and Manuel Gomez-Rodriguez. The network visibility problem. ACM Transactions on Information Systems (TOIS), 40(2):1-42, 2021.
12

---

[39] Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2, page O. Lille, 2015.
[40] Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial supervidd contrastive learning. Advances in Neural Information Processing Systems, 33:2983-209 2994,, 2020.
[41] Chih-Hui Ho and Nuno Nvasconcelos. Contrastive learning with adversarial examples. Advances in Neural Information Processing Systems, 33: 13:17081-17099 2020.
[42] Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Axi Niu, Jiu Feng, Chang D Yoo, and In So Kweon. Decoupled adversarial contrastive learning for self-supe adversarial robustness. In European Conference on Computer Vision, pages 725-742. Springer, 2022.
[43] Xilie Xu, Jingfeng Zhang, Feng Liu, Masashi Sugiyama, and Mohan Kankanhalli, Enhancing adversarial contrastive learning via adversarial invariant regularization. In NeurlPS, 2023.
[44] Judea Pearl. Causality. Cambridge university press, 2009.
[45] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: founda- tions and learning algorithms. The MIT Press, 2017.
[46] Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast ad- versarial training. Advances in Neural Information Processing Systems, 33:16048-1609 2020.
[47] Yihua Zhang, Guanhua Zhang, Prashant Khanduri, Mingyi Hong, Shiyu Chang, and Sijia Liu. Revisiting and advancing fast adversarial training through the lens of bi-level optimization. In International Conference on Machine Learning, pages 26693-26712. PMLR, 2022.
[48] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do ad- versarially robust imagenet models transfer better? Advances in Neural Information Processing Systems, 33:3533-3545, 2020.
[49] Xiaojun Xu, Jacky Y Zhang. Evelyn Ma, Hyun Ho Son, Sanmi Koyejo, and Bo Li. Adversarially robust models may not transfer better: Sufficient conditions for domain transferability from the view of regularization. In International Conference on Machine Learning, pages 24770-24802.
PMLR, 2022.
[50] Haichao Zhang and Jianyu Wang. Defense against adversarial attacks using feature scattering- based adversarial training. Advances in Neural Information Processing Systems, 32, 2019.
[51] Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In Interna- tional Conference on Machine Learning pages 1 11778-18 1287. PMLR, 2020.
[52] Sravanti Addepalli, Samyak Jain, and Venkatesh Babu. Efficient and effective augmentation strategy for adversarial training. arXiv preprint 22Xiv:2210.153114 2022.
[53] Richard M Karp. Reducibility among combinatorial problems. In Complexity of computer computations, pages 85-103. Springer, 1972.
[54] Richard Santiago and Yuichi Yoshida. Weakly submodular function maximization using local submodularity ratio. In Yixin Cao, Sin-Wing Cheng, and Minming Li, editors, 31st International Symposium on Algoritimss and Computation, ISAAC 2020, December 14-18. 2020, Hong Kong, China (Virtual Conference). volume 181 of LiPles, pages 64:1-64:17. Schloss Dagstuhl Leibniz-Zentrum für Informatik, 2020.
[55] Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with a fast adaptive boundary attack. In International Conference Machine Learning, pages 2196-2205. PMLR, 2020.
[56] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: a query-efficient black-box adversarial attack via random search. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII, pages 484-501. Springer, 2020.
[57] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 16155 146, 2016.
[58] Dan Hendryeks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. In International Conference on Machine Learning, pages 2712- 2721. PMLR, 2019.
13

---

[59] Andreas Krause and Daniel Golovin. Submodular function maximization. Tractability, (71-104):3, 2014.
[60] Daniel Golovin and Andreas Krause. Adaptive submodularity: Theory and applications in active learning and stochastic optimization. Journal of Artificial Intelligence Research, 42: 427-486, 2011.
[61] Cihang Xie, Mingxing Tan, Boging Gong, Jiang Wang, Alan L Yuille, and Quoe V Le. Adver- sarial examples improve image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 819-828, 2020.
[62] llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: /608. 03983, 2016.
[63] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach to self-supervised learning, Advances in neural information processing systems, 33:21271-21284, 2020.
[64] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640-9649, 2021.
[65] Jovana Mitrovic, Brian McWilliams, Jacob C Walker, Lars Holger Buesing, and Charles Blundell. Representation learning via invariant causal mechanisms. In International Con- ference on Learning Representations, 2021. URL httpht://ppeeeeeeee..eeee net/ net/ffouuffiid 9p2ekP904Rs
[66] Peter Bühlmann. Invariance, causality and robustness. 2020.
14

---

A Theoretical Analysis
A.1 Proof of Theorem 1
Theorem 1 (restated) We define a proxy set function Go(S) - Ga(S) + |S where o =
1+ V1 + 12l2 nM Le(Li M L2L3)), V1 0+, and 22 are positive constants.
Given Assumption (S) monotone and 7-weakly submodular where 7 > 7 2
Proof. We first provide proof of monotonicity of (S)). We use Taylor expansion times to convert the marginal gain function Go(x(S) as follows,
Go(z|S) = Go(S (x}) Go(S)
a CACL(S:0)
a (aco(((())))) + CRD(U;0- - "0 (ACL(S;0) + -o
a
-URD(UU;
=
0g
Taylor expansion
à (S:
0
(U:
-1 0g
+o
0g
d
Taylor expunsion
0
[RD(U:0)
0 (UD(U:0)
0 (ACI(S:0)+
0 (aci(12):0)
11 0g
0g
dg
(0) +o
1l (U;(U:0)
a (A
- ~22
LRD CRD (U:
02 CACL (S: R
CACI((x):
0g
Og
0
0g
0g
11 0 (S:
02
θ (Ac((1)):0) + (2) (acc((=):0))
0
CRD (U:0)
0g
0g
0g
0g 0g
+ [1) + o,
(8)
where (i( 0 and (0) + are the remainder terms of Taylor series. Recall that LRD (U:
Ex.ev (DD (r;;; 0), LACL () Ex,s ( 0), |U] M, and [X] = N. According to
Assumption we have the following results:
0
11
(U:(U:))] s 1 = ML1;
(9)
0g
0²
(RD(U: (U;0)] < L = MLx;
(10)
0,
||
CACL (S:0)) < 1 < A1X KNL2
(11)
0g
02
(S: < م S - = KNL4.
(12)
0g
We assume that (0)] S v and ( < V2 where vs - 0+ and u2 > are two positive constant. We can transform Eq. as follows,
Go(x)S) 2 |(Go(())S1
- (ACL(())00)
-1²1 2
02
LACL (S: (S:0)11 2
2 7|| [RD(U:0)] 0)1
CRD (UD(U;)) (1;0)11
Oy
On
8g
21 LACL(S:0)Il (nD(U:))]I
LACL(1e)) (1e):0)1 - -1-00
(1ll
(1)
9
0p
0g
03
0g
[1)1 to
2 12 - -12 -
V2l 1/1 +
2 -1/MIa(T1 -1 +
- 12 111
Since o = 7L + + LzLx)) + + + 1, we can obtain that
(x(s) 2
(13)
15

---

Eq.
13 shows that (S) is monotone, i.e., Go(z]S) > 0.
Then, we provide proof of 7-weakly submodularity of Go(S). The sketch of proof is that we first prove (S) is a-submodular. Then, according to Lemma we can conclude that Go(S) is also 7-weakly submodular.
Lemma (Proposition 4 of Gatmiry and Gomez-Rodriguez (38). The set function G(S) 2* R
that is a-submodular is 7-weakly submodular with the submodularity ratio 2 1 - a.
The following the definition of a-submodularity.
Definition 2 (o-submodularity
38 A function is called a-submodularr if the marginal gain of
adding an element x to set
a times greater than or equals to the gain of adding an element
set B where A C B. i.e.
Go(z|A) (1 (1-a)Go(z1B))
A,B|ACB
According to Eq.
13
we have obtained the lower bound of the marginal gain
Go(z|S) 2 (S).
(14)
We obtain the upper bounder of marginal gain based Eq. as follows,
a
2² R 0g (U:(U: 0g (AcL(S: Cacu(fx);
Go(x|S) nnl mll
(U: (u:
FACE (fx)
n
0g
0g
0g
0² (uD(U:
i LACL (1=)
0 E2(0)||| ((c((())
12 + 0g
(S:
0g
0g
8g
0g
+ (0)1 to
< MMMLI L2 + M
+ to
$ (L, I L2la)) L2 + ta.
Since (LL + +KN + L2L3)) + 12 1/2 1, obtain that
Go(z|S) S 2T (L L2L3)) + 2v1 2v2 + 2o 1= Go (15)
Based on the lower bound (Eq. 14 and the upper bound (Eq. 15 of the marginal gain function, we can obtain
(a(x)AA) 2
Cg(
Go(z|B) Go(S)
M + 7 + L2Ln)) + 201 + 22 +
2x =1- (0.
(16)
Therefore, we have proved GS) is a submodulaa.. According to Lemma [ submodular where 7 > 7" T
Gg(S) is 7-weakly
A.2 Proof of Theorem
Theorem 2 (restated) Given fixed parameter 0, denote the optimal solution of Eq. G SUDscxxs///Ii== Go(S). Then, S in Eq. (6) found via greedy search satisfies
5
as
Go G (G) -No)
Proof. We provide the lemma that describes the optimality guarantee of the greedy search algorithm the problem of maximizing a monotone a-approximate submodular function subject to cardinality constraints.
16

---

Lemma 2 (37 38)) Given a monotone and a-approximate submodular function G(S), the greedy search algorithm achieves 1 - e -1-a) approximation factor for the problem of maximizing G(S) subject cardinality constraints.
Note that according to Theorem Therefore, according to Lemma
Gg(S) is monotone and a' -apprxxmmatee submodular function. we have
Go(S*) 2 (1 - yG
(17)
where G3 = sUpsEx////IXI= Go(S) and S is found via greedy search. Since Go( Ga(5)
[S we have
G3
sup SEX./S/ =
(Go(S) 1 SCX,/S//XX|
sup
Go(S) + A/X G3 KNo..
(18)
Recall that G5() Go(S*) + kNo. Therefore, by transforming Eq. we can obtain
Go(S*) 2 G3 (G) kNo) e
(19)
A.3
Derivation of Eq.
Here, provide the details of deriving Eq.
using Taylor expansion as follows:
(B|S) G)(SUB) GG(S)
(20)
Go((S |SU B/o
(21)
Go(SU Ga(S) + 3o
(22)
LuDD(U
(anl))) + Bo - (S
(23)
+ ( Bo
(24)
-RDDUU; - VooooVoooCLLBBBB + (U: 0g) 8o
7)
(25)
(CRD(U:6g) - VoLLD VOLACL (B:0) + E) + LRD + 3o (26)
V/VoLa (U;0s) A (B:0) -3a
(27)
where a minibatch whose batch size is 3 E N, S is the subset, and S and B are disjoint subsets.
Eq. (20) holds according the definition marginal gain in Definition I Eq. (21) holds according
to the definition of the proxy set function (S) in Theorem Eq. 22 holds because S and B
are disjoint. Eq. (23) holds according to the definition of set function in Eq. 44) Eq. (24) holds by
splitting the ACL loss on S U B into the sum of the ACL loss on S and the ACL loss on B. Eq. (25
holds by letting As = 0 - 1) o (S;6). In Eq. (26). we transform EnD(Ui As V CACL B:
via Taylor expansion where is the remainder. Eq. (27) holds by omitting the remainder. By replacing
Os with 0 vo 8) in Eq. (27). we can obtain Eq.
B Extensive Experimental Details and Results
B.1 An Extra Trick to Enable Efficient RCS on Large-Scale Datasets with Limited GPU Memory
RCS on large-scale datasets such as ImageNet-IK ( needs to calculate the loss gradient for each minibatch of training data as the first step (Line 5-7 in Algorithm DT. Then, the greedy search is conducted to iteratively select the minibatch that has the largest marginal gain and add this minibatch into the final coreset (Line 9-20 in Algorithm[I). Saving the gradients of all the minibatches GPU needs a large amount of GPU memory. When the GPU memory limited, there exists an issue that unable save all the gradients GPU.
direct solution to transfer the loss gradients from GPU CPU and save all the gradients on CPU. When need to use the loss gradient of minibatch, we can transfer the gradient from CPU to GPU
17

---

for calculation. Unexpectedly, the transition between CPU and GPU still consumes a large amount of time in practice. Thus, we do not take this solution.
To solve this issue, we split the entire training set into several training subsets. Practically, we take every 100 minibatches of training data as training subset. Then, we conduct RCS each training subset respectively and collect the coresets from each training subset together as the final coresets for robust pre-training. In this way, we can enable RCS efficiently search for the coresets from large-scale datasets with limited GPU memory. We apply this trick to the experiments regarding ACL on ImageNet- 1K in Section 4.2 and SAT on ImageNet- IK in Section
B.2 Complementary Experimental Details
We conducted all experiments on Python 3.8.8 (PyTorch 1.13) with 4 NVIDIA RTX A,000 GPUs (CUDA 11.6).
Following Jiang et al. [4], we leveraged the dual BN [6], where one BN is used for the standard branch of the feature extractor and the other BN is used for the adversarial branch, for the implementation of ResNet-18 and WRN-28- 10 backbone models.
Experimental details of ACL [4].
As for ACL using ResNet- 18 on CIFAR-10 and CIFAR-100, we pre-trained 18 models using SGD for 1000 epochs with an initial learning rate 5.0 and a cosine annealing schedule [62]. During pre-training, we set the adversarial budget CACL = 8/255, the step size OACL 2/ 255, and the PGD step TACL= = 5 for generating adversarial training data.
As for ACL using WRN-28-10 10 on ImageNet-IK, we pre-trained WRN-28-10 on ImageNet- IK 32 32 resolution via ACL [4] with RCS and Random, respectively. We set B = 512, TACL EACL 8/255. PACL = 2/ 255 TRCs = 1. The temperature parameter was set as 0.1, following Chen al. [13]. The model was trained using SGD for 200 epochs with an initial learning rate 0.6 and a cosine annealing scheduler. We set warmup epoch W = 200, and then CS was executed every 10 epochs. We pre-trained WRN-28-10 with the subset fraction k 0.05 and then finetuned on CIFAR10 and CIFAR-100.
Experimental details of DynACL 177. The pre-training settings of DynACL 17 followed ACL [14,, except for the strength of data augmentation and the hyperparameter w. We denote the strength of data augmentation and the hyperparameter at epoch e as Ma and We respectively, where
e
K
Me = 1 - |
e e 1,...... E 1--1 1}.
K
E'
We = v (1 111--)) pe);
EE (0, E. 1)1,
1}.
in which the decay period K 50, the reweighting rate 2/3, the total training epoch F 1000. In our implementation of DynACL, we only take the dynamic strategy and do not take the trick of the stop gradient operation and the momentum encode 663 64
Experimental details of finetuning. As for linear finetuning (SLF and ALF), we fixed the parameter of the encode and only trained the linear classifier using SGD with an initial learning rate 0.01 divided by 10 at Epoch 10 and 20 for 25 epochs. As for AFF, we trained the whole model for 25 epochs using SGD. The adversarial training data is generated via PGD with 10 PGD steps, adversarial budget 8/255. and step size 2/255. Practically, we exactly used the finetuning code provided in the GitHub of DynACL [17].
Baseline models that are pre-trained on the entire set. As for the pre- pre-trained ResNet- 18 on the entire set of CIFAR- 10 and CIFAR-100 via ACL and DynACL, we used their released pre-trained weights in their GitHut to reproduce their results.
As for the pre-trained WRN-28-10 on the entire set of ImageNet-IK, due to the extremely long pre-training time of ACL on the entire set (as shown in Table we do not provide the results of ACL on the full set of ImageNet-1K: We only provide the results of ACL on the coreset in Table T and
Link of pre-trained weights via ACL on CIFAR-10.
I Link of pre-trained weights via ACI on CIFAR-TOR 100.
Link of pre-trained weights via DynACI on CIFAR-10.
Link of pre-trained weights via DynACL on CIFAR-IÚS 100.
18

---

Table 3: Running time (hours) on the entire training set of various models via ACL, DynACL, SAT, and TRADES.
Pre-training the entire
ResNet-18 CIFAR-10
ResNet-50 ImageNet-IK
WRN-28-10 ImageNet-IK,
ACL [4]
42.8
650.2
DynACL 117
42,9
SAT 34
5.2
286.1
341.7
TRADES 335
Table 4: Self-task adversarial robustness transferability of ACL with RCS evaluated on the STL-10 dataset. The number after the dash line denotes subset fraction {0.05, 0.2).
Runing time
SLF
Pre-training
(hours)
SA (%)
RA (%)
ACL-Entire
74.4
71.16
33.21
ACL with Random-0.05
10.9
62.86
22.00
ACL. with RCS-0.05
13.1
66.77
26.99
ACL with Random-0.1
13.9
64.54
24.16
ACL with RCS-0.1
16.4
68.17
28.66
ACL with Random-0.2
19.4
66.61
27.09
ACL with RCS-0.2
22.6
70.26
30.09
Table 5: Self-task adversarial robustness transferability of DynACL with RCS evaluated on the
STL-10 dataset. The number after the dash line denotes subset fraction {0.05, 0.1, 0.2).
Runing time (hours)
SLF
Pre-training
SA (%)
RA (%)
DynACL-Entire
74.7
69.63
46.51
DynACL with Random-0.05
10.9
55.05
27.38
DynACL with RCS-0.05
13.2
63.75
37.01
DynACL with Random-0.1
14.1
58.77
30.72
DynACL with RCS-0.1
16.6
67.42
40.88
DynACL with Random-0.2
19.7
62.79
34.94
DynACL with RCS-0.2
22.9
69.14
43.27
As for pre-training via SAT 34 on the entire set of ImageNet-IK IK, we use WRN-288 10 pre-trainedd on ImageNet- 1K of 32 x 32 resolution provided by Hendrycks et al. [58] and ResNet-50 (c 4.0) pre-trained on the entire set of ImageNet- 1 K of 224 x 224 resolution provided by Salman et al. [48]
Table demonstrates the approximated training time on the entire set of various models via ACI 14] DynACL [T7, SAT 34, and TRADES (35), respectively. All the running time was evaluated on NVIDIA RTX A5000 GPUs. We used I GPU for robust pre-training CIFAR-10 and CIFAR-100, and GPUs for pre-training on ImageNet-IK.
B.3 Self-Task Adversarial Robustness Transferability Evaluated on the STL-10 Dataset
Following the setting of [17], applied our proposed method RCS to speed up ACL and DynACL on STL-10 (26) and report the results Tables 4 and The results validate that RCS is effective in speeding up both ACL and DynACL while being able to learn useful robust representations.
B.4 Experimental Details Figure
In Figure I we report the cross-task robustness transferability of pre-trained ResNet-18 via ACL with RCS (subset fraction l: 0.1) from CIFAR-I0 CIFAR-100 and STL-10 via SLF. The experimental settings are the same as Section 4.1
B.5
Robustness Evaluated under Various Attacks
In this subsection, we report the adversarial robustness evaluated under three strong white-box attacks (APGD-CE [19) APGD-DLR [9 and FAB (55]) and one strong black-box attack (i.e., Square Attack (56)) We evaluate the cross-task adversarial robustness transferability from CIFAR-100 to STL-10 via ALF under various attacks and report the results in Table The results validate that RCS can consistently improve robust test accuracy over various adversaries compared with Random and almost maintain the robustness transferability while significantly speeding up the pre-training.
19

---

Table 6: Adversarial robustness evaluated under various attacks including APGD-CE 19] APGD-
DLR
19
FAB [55], and Square Attack [56].
The number after the dash line denotes subset fraction
k == 0.1.
Pre-training
Running time (hours)
APGD-CE
APGD-DLR (36)
FAB (%)
Square Attack (56
ACI.
42.8
29.70
25,35
24.60
28.61
ACL with Random-0.1
27.29
23.70
22.69
26.39
ACL with RCS-0.1
9.3
29.10
26,77
25.62
29.15
DynACL
42.9
31.35
27,41
26.76
28.61
DynACL with Random-0.1
8.3
26.56
22.44
21.44
25.39
DynACL RCS-0.1
29.23
26.89
25.99
29.50
Quantitative analysis
0.35-
ACLertiire
KL. divergence
0.30
05
0.25
RCS-0.05
Random-0.11
0.20
RCS--0
0.15
Randdom-tt
0.10-
ACL speed-up ratio
Figure 5: Average value of the KI. divergence between natural test data and its adversarial variant.
B.6 Analysis of the Coreset Selected by RCS
Quantitative analysis of the coreset selected by RCS. Here, provide comprehensive quantita- tive analyses of the coreset selected by RCS in Figures and 6 to help understand the effectiveness of RCS in maintaining transferability.
Figure we demonstrate the average value of the KI. divergence between natural data point x and its PGD-20 adversarial variant the CIFAR- 10 test set (i.e., fRD (a:)). Specifically, we collected the average value of the KL divergence between natural data point æ and its PGD-20 adversarial variant on the CIFAR-10 test set (i.e., ÉDD (z)) and show it in the leftmost panel of Figure 5 It validates that RCS helps minimize RD, thus helping ACL obtain useful and adversarially robust representations. It validates that RCS helps minimize RD, thus helping ACL obtain useful and adversarially robust representations.
In Figure 6 we demonstrate the imbalance ratio and the maximum mean discrepancy (MMD) between the coreset and whole training set. Note that the imbalance ratio the ratio of the sample size of the largest majority class and that of the smallest minority class. MMD is classical measurement of the distance between two distributions. The left panel of Figure 6 shows that the corset selected by RCS is almost bass-balanced since the imbalance ratio of RCS is slightly higher than 1.0. The right panel of Figure [6 shows that RCS yields a lower MMD between the entire training set and the selected coreset compared to Random. Therefore, our quantitative analysis demonstrates that RCS selects a coreset that is closer to the entire training set than Random, thus helping to maintain transferability.
Visualization analysis the coreset selected by RCS. We count the frequency each training sample in the CIFAR- 10 dataset being selected into the coreset. In Figure visualize the top-5 most-frequently selected (MFS) training data and the top-5 least-frequently selected (LFS) training data. Figure 7 shows that, compared LFS data, MFS data images whose backgrounds are more
complex and are more difficult to be differentiated from the subject. Recent work 66
43 66 has
shown that exempting the representations from the nuisance style factors such the background factor can improve the robustness against distribution shifts. RCS prefers to select images of complex backgrounds to help the model learn representations that are independent of the background factors, thus helping maintain robustness against adversarial perturbations.
B.7 Comparison Between ACL with FGSM and ACL with RCS
To the best our knowledge, previously no one has studied how to incorporate FGSM 24 [8] with ACL speed up ACL with FGSM. Previous studies (24 25 only leveraged the trick for speeding up supervised AT where labels are required. Here, we take the primitive step to study the performance of FGSM in efficient ACL..
We demonstrate the comparison between ACL with FGSM and ACL with RCS on the CIFAR-10 task, which validates that our proposed RCS is more efficient and effective than FGSM 24]
20

---

Imbalance Ratio
MMD
x10
1.14
Imbalance Ratio
1.13
Raudom-0.2
8.05
RCS-0.2
1.12
1.11
MMD
7.95
Random-0.2
1.10
7.90
RCS-0.2
1.09
7.85
1.08
1.07
200
400
600
800
1000
200
400
600
800
1000
Epoch
Epoch
Figure 6: We show the imbalance ratio the coreset (left panel) and the maximum mean discrepancy (MMD) between coreset and the full training set (right panel) at each epoch. We conducted ACL with RCS and ACL with Random on the CIFAR-10 dataset using = 0.2. The coreset is selected every I = 20 epoch. We used Gaussian filter (a 3) to smooth the line.
Most- fiet--Freuentlly Selected (MFS) Data
Least- Fresu-Fre Selected (LFS) Data
Figure 7: We show the most-frequently selected data (upper panel) and the least-frequently selected data from the CIFAR-10 10 training data.
Table 7: The self-task adversarial robustness transferability achieved by ACI with FGSM and ACL with RCS on the CIFAR-10 10 dataset. The number after the dash line denotes the subset fraction 0.2.
Runing time (hours)
SLF
ALF
Pre-training
SA (%)
RA (%)
SA (%)
RA (%)
ACL-Entire
42.8
78.87
39.19
75.74
40.56
ACL with FGSM
21.7
86.70
0.00
60.04
0.11
ACL with RCS-0.2
13.0
75.96
37.21
74.32
38.68
DynACL-Entire
42.9
75.39
45.05
72.90
45.65
DynACL with FGSM
21.8
83.89
1.02
70.83
3.27
DynACL with RCS-0.2
13.1
75.16
44.32
72.10
45.75
We conducted pre-training CIFAR-10 using ResNet-18 via both ACL and DynACL the entire set with FGSM (denoted as"ACI with FGSM" and "DynACL, with FGSM" respectively) and then conducted the linear evaluation CIFAR-10. Following Fast-AT 24 the adversarial budget as 8/255 and step size as 10/255 for FGSM. Other training and finetuning configurations (e.g., optimizer and learning rate scheduler) are the same as in Section 4.1
used one-step PGD with
Table 7 we show the comparison between RCS (subset fraction = 0.2) and FGSM in speeding up ACL. We observe that FGSM cannot effectively learn useful robust representations. In addition, RCS even consumes less running time than FGSM. Therefore, our proposed method RCS is more efficient and effective than FGSM 24 in speeding up ACL.
21

---

Table 8: Self-task adversarial robustness transferability achieved by ACL with RCS using various distance functions evaluated on the CIFAR-10 dataset. The number after the dash line denotes subset fraction & E {0.05, 0.1,0.2)
Runing time (hours)
SLF
Pre-training
Distance function
SA (%)
RA (%)
ACL-Entire
42.8
78.87
39.19
ACL with Random-0.05
6.5
67.45
22.96
ACL with RCS-0.05
JS
7.7
72.39
32.43
ACL with RCS-0,05
OT
7.9
72.49
32.36
ACL with RCS-0.05
KL
7.6
72.56
32.49
ACL with Random-0.1
8.3
70.68
27.19
ACL with RCS-0.1
IS
9,4
74.43
34.23
ACL with RCS-0.1
OT
9.5
74.39
34.26
ACL with RCS-0.1
KL
9.3
74.67
34.30
ACL with Random-0.2
11.8
72.01
29.87
ACL. with RCS-0.2
JS
13.2
75.75
37.18
ACL with RCS-0.2
OT
13.3
75.59
37.06
ACL with RCS-0.2
KL
13.0
75.96
37.21
Table 9: Self-task adversarial robustness transferability achieved by ACL with RCS using various warmup epochs evaluated on the CIFAR-10 dataset.
Runing time (hours)
SLF
Pre-training
Warmup epoch
SA (%)
RA (%)
ACL-Entire
42.8
78.87
39.19
ACL with Random
100
11.8
72.01
29.87
ACL with RCS
100
13.0
75.96
37.21
ACL with Random
200
15.2
73.38
30.59
ACL with RCS
200
16.3
76.43
38.33
ACL with Random
300
18.7
73.91
31.62
ACL with RCS
300
19.6
78.18
39.01
B.8 Efficient ACL via RCS with Various Distance Functions
We pre-trained ACL with RCS using various distance functions CIFAR-10 and then report the performance via linear evaluation on CIFAR-10 in Table Other training settings exactly keep the same as Section 4.1 except for the distance functions.
validates that efficient ACL via RCS with various distance functions can consistently obtain better transfcrability compared with Random. Especially, RCS with KL divergence often achieves better transferability downstream tasks among different distance functions. Therefore, we use KI. divergence for RCS in all the experiments in the main paper.
B.9 Efficient ACL via RCS with Various Warmup Epochs W
This section provides the results of RCS with different warmup epochs W E (100, 200, 300} for
efficient ACL. We set the subset fraction l 0.2. Other training settings follow Section 4.1 Table 9 shows that as the warmup epoch of RCS increases, the performance evaluated on the CIFAR-10 task becomes better while consuming more running time. Besides, RCS still consistently achieves better performance than Random, which validates the effectiveness of RCS speeding up ACL while maintaining robustness transferability.
B.10 Efficient ACL via RCS with Various Epoch Intervals I
We pre-trained ResNet-18 on CIFAR-I0 via ACL with RCS using E {10, 20, 50), k 0.2, and W 100. Then, we evaluate the performance in the CIFAR-I0 task via SLF. Table shows that larger leads to more pre-training time since the frequency of conducting coreset selection becomes larger. Meanwhile, larger I can lead to higher robust and standard test accuracy in downstream tasks. This is because a larger enables the coreset to be updated more timely to adapt to the latest state of the model and select data points that help the model improve its robustness.
22

---

Table 10: Impact of for RCS evaluated on the CIFAR-10 task.
Runing time
SLF
(hours)
SA (%)
RA (%)
10
15.6
76.31
38.17
20
13.0
75.96
37.21
50
12.2
75.87
35.54
Table 11: Impact of batch size for coreset selection evaluated on the CIFAR-10 task.
Runing time (hours)
SLF
Pre-training
Batch size for RCS
SA (%)
RA (%)
ACL-Entire
42.8
78.87
39.19
ACL with Random
1 11.8
72.01
29.87
ACL with RCS
64
13.6
76.21
37.48
ACL with RCS
128
13.3
76.15
37.41
ACL with RCS
256
13.1
75.89
37.17
ACL with RCS
512
13.0
75.96
37.21
Table 12: Compatibility with AdvCL 15]
evaluated on the CIFAR-I0 task SLE;
Runing time (hours)
SLF
Pre-training
SA (%)
RA (%)
AdyCL-Entire
57.8
80.89
42.36
AdvCL with Random
11.0
73.67
33.61
AdvCL with RCS
13.5
77.93
38.89
B.11 Efficient ACL via RCS with Various Batch Sizes
In this subsection, we show the impact of the batch size during coreset selection. We trained ResNet- 18 via ACL with RCS on CIFAR-10 10 and then linearly finetuned ResNet- models on CIFAR-10. The batch size for RCS is selected from {64, 128, 256, 512} and the batch size for ACL keeps as 512. The subset fraction keeps as 0.2. Other training settings exactly follow Section standard and robust test accuracy in Table TT
We report the
We can find that as the batch size for RCS decreases, the running time becomes larger. is because there is a larger number of batches needed to calculate the loss gradients during RCS when the batch size becomes smaller. Besides, we observe that the test accuracy on the downstream tasks seems to gradually increase as the batch size decreases. Especially, ACI. with RCS using 64 batch size gains consistent improvement compared with ACL with RCS using 512 batch size, which indicates that smaller batch size for RCS is useful to improve the performance but consumes more running time. Therefore, there could be a trade-off between the running time and the transferability. We leave how to further improve the efficiency and effectiveness in maintaining the transferability of RCS as the future work.
B.12 RCS for Accelerating Another Variant AdvCL [15
Fan et al. [15] proposed variant of ACL method, called "AdvCL", that leverages standardly pre- trained model on ImageNet- K to generate pseudo-labels for CIFAR-10 training data via K-means clustering. Based ACL [4] the loss function of AdvCL is composed of weighted sum of the adversarial contrastive loss and an ensemble of the CE loss between the adversarial data and its pseudo label over different choices of cluster number. By simply replacing the loss function of the ACI (.C. with that of AdvCL. in Algorithm and 2
we can apply RCS for efficient AdvCI..
We pre-trained ResNet-18 18 via AdvCL with RCS CIFAR-I0 using SGD for 1000 epochs with an initial learning rate 0.5 and a cosine annealing scheduler. We set / = 512, T TACL = 5, EACL = 8/255, PACL 2/255, and TRCS 3. The pre-training settings of AdvCL exactly follow Fan et al. [15] We take the same configuration of RCS as that of RCS for ACL in Section 4.1 i.e., warmup epoch W = 100, the epoch interval for conducting RCS I 20, and the subset fraction 0.1. Then, evaluate the performance on the CIFAR- task via SLF.
Table 12 shows that RCS can speed up AdvCL [15
well. Besides, RCS is principled method that helps AdvCL to obtain effective robust representations since AdvCL with RCS always achieves higher test accuracy compared with AdvCL with Random. Therefore, the experimental results
23

---

Algorithm 3 RCS for Supervised AT
1: Input: Labeled training D, validation set U, batch size B, model g fo, learning rate for
RCS 1/. subset fraction E (0, 1], loss function of supervised AT method Ssupereised()
2: Output: Coreset S
3: Initialize S + 0, N + [D]
4: Split entire set into minibatches {Bm)
[N/9] m=1
5: for each minibatch Bm C D do
6: Compute gradient 9m VeLeupercised (Bm; (Bm,)
7: end for
8: // Conduct greedy search via batch-wise selection do Compute gradient qu t VOLRD(U;O) 11: Initialize best beit_gainttta = -00 for each minibatch Bm D) do Compute marginal gain Gi G.(B...5 t nu 4m if G (Bm (S) > best__gait then
9: for 1, ,....,,Lk...
15:
Update +- m, best_ luiitggainnn +- G(Bm)))
end if
17:
end for
18:
Update S S--UBB B2, DE DV B,
19: Update 0 + 1170
end for
Algorithm 4 Efficient Supervised AT via RCS
1: Input: Labeled training set D, validation set U, total training epochs E, learning rate ry', batch size 3. warmup epoch W, epoch interval for executing RCS subset fraction k, learning rate for RCS loss function of supervised AT method Lsupervised
2: Output: robust pre-trained feature extractor f'
3: Initialize parameters of model g S6
4: Initialize training set <- D
5: // Warmup training for W epochs; Training on the coreset for (E W) epochs
6: for
to (E 1) do
7: if e%% = and W then
8:
- copy(8") S (RCS(D,U,,,)) fo. n, Ci suppervised // by Algorithm 3
9:
10: end if
11: for batch l.... [|S|/3] do
12: Sample minibatch Bn, from
Update O' 0' Vo Laupernised (Bm; 0')
14: end for
15:
end for
validate that our proposed RCS can be unified and effective framework for accelerating ACL as well as its variants [7
B.13 RCS for Accelerating Supervised AT
In this section, we first provide the algorithm of RCS for supervised AT including fast adversarial train- ing (Fast-AT) [24]. free adversarial training (Free-AT) (25). standard adversarial training (SAT) B4
and TRADES 35 Then, we show the comparison between RCS and ACS [33 in speeding up
supervised AT CIFAR-10 and the effectiveness of RCS in efficient SAT on ImageNet-I1 1K (
Algorithm of RCS for supervised AT. Prior introducing the algorithm of RCS for SAT and TRADES, we first provide the preliminaries of SAT and TRADES.
Given labeled training set D {(I,, 9.) where data z; X' and label ys y {0, 1. I...,,C........... C 1}. feature extractor fo X + Z, randomly initialized classifier g Z - RC

---

Table 13: Comparison between Fast-AT
24
with ACS [3
and ATst-AT with RCS on CIFAR-10.
Training method
Running time (minutes)
SA (%)
RA under PGD-20 (96)
RA under AutoAttack (%)
Fast-AT on the entire set
25.0
86,20
45.80
41.04
Fast-AT with ACS
12.8
82.71
45.71
40.92
Fast-AT with RCS
12.0
83.47
45,89
41.06
Table 14: Compatibility with Free-AT [25).
Training method Free-Al on the entire set Fast-AT with RCS
Running time (minutes)
SA (%)
RA under PGD-20 (%)
RA under AutoAttack (%)
116.5
84.18
49.05
45.14
60.6
82.39
49.24
45.15
the loss function of SAT is
SATT (D:0) Σ
max Elg fo(8:).
(28)
where é is the Cross-Entropy (CE) loss and x; is adversarial training data generated by PGD within the e-ball centered at Tz.
The loss function of TRADES is
LTRADES (D:0) = Σ Elgo so( te. max KL(g fo( lo(w:)). [o(1))))) [..
(29)
where is the CE loss, (I.):)) is the KL divergence, c > O is a trade-off parameter, and I; adversarial training data generated by PGD within the e-ball centered æ;. We set c following Zhang et al. (35]. Note that the parameters of g are updated during supervised AT. Here we omit the parameters of g since we only use the parameters of the feature extractor fg in downstream tasks.
The RCS problem for supervised AT is formulated as follows:
S arg max - -LRD (UR 0 - -VVooosuperiise (S;0))
(30)
SCD
in which we replace the ACL loss LACL (-) in Eq. with the supervised AT loss ssupervised (-) (e.g.. (-AT(-) and (TRADES(.).. Due to that (RD(-) only needs data and does not need any label, RCS is applicable to supervised AT, no matter if the validation
unlabeled or labeled. By
leveraging greedy search, we show the algorithm of RCS for supervised AT in Algorithm and efficient supervised AT via RCS in Algorithm 4
B.13.1 Comparison Between RCS and ACS 33 in Speeding Up Supervised AT including Fast-AT (24), SAT 34 and TRADES 33
Comparison between Fast-AT with RCS and Fast-AT [24] with ACS (33] We conducted Fast- AT, Fast-AT with ACS, and Fast-AT with RCS on CIFAR-10. The experimental setting of Fast-AT and Fast-AT with ACS exactly follows that in Section 4.3 of Dolatabadi et al. [33. That is, trained ResNet-18 on CIFAR- 10 via Fast-AT' (c = 8/255, a = 10/255) using SGD with 0.9 momentum for 60 epochs with the initial learning rate of 0.1 and divided by 10 Epoch 40 and 50. As for the RCS for speeding up Fast-AT, we set the subset fraction as k: = 0.5, warmup epoch as W = 10, and epoch interval for executing RCS 10. We report the standard test accuracy and robust test accuracy evaluated by PGD-20 (e 8/255, a 8/2550) and AutoArtack in Table 13 All experiments are conducted one RTX A5000 GPU. Table 13 validate that RCS, without using labels, more efficient and effective in speeding up Fast-AT.
Compatibility with Free-AT (25). Further, we conducted Free-AT (25 and Free-AT with RCS on CIFAR-10. We trained ResNet-18 on CIFAR-10 via Free-AT (e = 8/255) using SGD with 0.9 momentum for 60 epochs with the initial learning rate of 0.01 and divided by 10 at Epoch 40 and 50. We keep the configurations of RCS for speeding up Free-AT the same as above. We report the standard test accuracy and robust test accuracy evaluated by PGD-20 and AutoAttack in Table [4
Therefore, RCS, without using label information, can further speed up both Fast-AT 24] and Free- AT 25 while almost maintaining the adversarial robustness. In addition, RCS without using labels more efficient than ACS while achieving similar adversarial robustness compared with ACS. Therefore, it validates the effectiveness of RCS in efficient supervised AT.
25

---

with ACS [33 and SAT with RCS on CIFAR-10. Here, "RA" stands for the robust test accuracy under PGD-20 attacks following the setting of Dolatabadi
Table 15: Comparison between SAT +.
et al. 33 The number after the dash line denotes subset fraction k E (0.05, 0.1, 0.2).
Training method SAT on the entire set
Running time (minutes)
SA (%)
RA (%)
314
82.49
52.11
SAT with Random-0.05
92
69.46
32.66
SAT with ACS-0.05
142
78.44
49.31
SAT with RCS-0.05
106
78.83
49.46
SAT with Random-O.
100
73.69
35.42
SAT with ACS-0.1
151
79.08
50.43
SAT with RCS-0.1
115
80.17
50.54
SAT with Random-0.2
119
76.89
35.72
SAT with ACS-0.2
172
80.71
50.80
SAT with RCS-0.2
131
81.89
50.87
Table 16: Comparison between TRADES 35) with ACS 33 and TRADES with RCS on CIFAR- 10. Here, "RA" stands for the robust test accuracy under PGD-20 attacks following the setting of Dolatabadi et al. 33 The number after the dash line denotes subset fraction k: E {0.05, 0.1, 0.2].
Training method
Running time (minutes)
SA (%)
RA (%)
TRADES on the entire set
332
82.72
54.35
TRADES with Random-0.05
111
74.56
40.12
TRADES with ACS-0.05
152
77.23
49.59
TRADES with RCS-0.05
123
77.52
50.28
TRADES with Random-0.1
122
75.80
42.19
TRADES with ACS-0.1
168
78.00
50.46
TRADES with RCS-0.1
136
78.59
50.83
TRADES with Random-0.2
146
77.41
44.07
TRADES with ACS-0.2
193
79.19
51.52
TRADES with RCS-0.2
161
79.82
51.84
Efficient SAT 34 and TRADES 29 via RCS on CIFAR-10. We trained ResNet-18 on CIFAR-
10 via SAT (3 using SGD with 0.9 momentum for 120 epochs with the initial learning rate of
0.1 divided by 10 at Epoch 30 and 60. For PGD configurations during SAT,
set 3 128.
TSAT = 10, ESAT = 8/255, PSAT =
2/255. During RCS, we set TRCS = 3. We took = 30 epochs for warmup, and then CS was conducted every 10 epochs with different subset fractions E (0.05,0.14 0.1, 0.2).
We trained ResNet-18 on CIFAR- 10 via TRADES using SGD with 0.9 momentum for 100 epochs with the initial learning rate of 0.1 divided by 10 at Epoch 60 and 90. For PGD configurations during TRADES, we set TTRADES 10, éTRADES 8/255, PTRADES 2/255. Other training settings follow SAT.
Here, we used ACS 33 and random selection (dubbed as "Random") ") as the baseline. The imple- mentation of ACS exactly follows the public GitHub of ACS [3]. We use the same subset fraction and warmup epoch for both ACS and Random as RCS.
In Tables 15 and 16. used the robust test accuracy of adversarial data generated via PGD-20 to evaluate adversarial robustness. shows that RCS always achieves higher robust and standard test accuracy while consuming running time compared with Random and ACS [33] among various subset fractions. It validates the effectiveness of our proposed RCS in maintaining the adversarial robustness of supervised adversarial training.
B.13.2 RCS for Efficient Supervised Robust Pre-Training on ImageNet-1K
Effecitveness of RCS in speeding up STA on ImageNet-1K I and maintaining robustness transferability (58) of SAT. Hendrycks et al. (58] pointed out that the ImageNet- IK adversarially pre-trained models can improve the adversarial robustness on downstream tasks.
Following Hendrycks et al. 55 we pre-trained WRN-28- 10 on ImageNet-IK of 32 x 32 resolution using SAT 34. The model was trained using SGD with 0.9 momentum for 100 epochs with an initial learning rate of 0.1 and cosine decay scheduler. For PGD configurations during SAT,
26

---

Table 17: Cross-task adversarial robustness transfcrability of adversarially pre-trained WRN-28-10 from ImageNet- 1K to CIFAR-IO. Here, "RA" stands for robust test aceuracy under PGD-20 attacks The number after the dash line denotes subset fraction
following the setting of Hendrycks et al. 58 k E {0.05, 0.1, 0.2).
Runing time (hours)
ALF
AFF
Pre-training
SA (%)
RA (%)
SA (%)
RA (%)
Standard training on entire set
66.7
10.12
10.04
84.73
51.91
SAT (58] entire set
341.7
85.90
50.89
89.35
59.68
SAT with Random-0.05
53,6
69.59
31.58
85,55
53.53
SAT with RCS-0.05
68.6
79.72
44.44
87.99
56.87
SAT with Random-0.1
70.2
73.28
33.86
86.78
54.95
SAT with RCS-0.1
81.9
81.92
45.10
88.87
57.69
SAT with Random-O.2
103.4
75.46
39.62
86.64
56,46
SAT with RCS-0.2
121.9
83.94
46.88
89.54
58.13
Table 18: Cross-task adversarial robustness transferability of adversarially pre-trained WRN-28-10 from ImageNet- 1K to CIFAR- 100. Here, "RA" stands for robust test accuracy under PGD-20 attacks following the setting of Hendrycks et al. [58]. The number after the dash line denotes subset fraction k {0.05, 0.1, 0.2).
Runing time (hours)
ALF
AFF
Pre-training
SA (%)
RA (%)
SA (%)
RA (%)
Standard training on entire set
66.7
11.05
0.00
54.81
27.81
SAT (58] entire set
341.7
63.35
32.08
61.56
35.64
SAT with Random-0.05
53.6
41.1
19.69
61.12
31.03
SAT with RCS-0.05
68.6
58.31
27.95
64.95
33.74
SAT with Random-0.1
70.2
50.57
20.99
63.6
31.97
SAT with RCS-0.1
81.9
59.92
28.11
66.12
34.41
SAT with Random-0.2
103.4
51.05
24.19
61.97
33.58
SAT with RCS-0.2
121.9
61.90
29.01
67.40
35.91
set 3 = 256, TSAT = 10, ESAT = 8/255, PSAT = 2/255. We set TRCS 3 during RCS. We took W = 10 epochs for warmup, and then CS was executed every W 10 epochs. We adversarially pre- train WRN-28-10 10 with different subset fractions is E (0.05, 0.1, 0.2) and then conducted adversarial finetuning on CIFAR- and CIFAR-100.
To evaluate the cross-task adversarial robustness transfcrability of pre-trained WRN-28-10 models via SAT, we adversarially trained the pre-trained models on CIFAR-IO and CIFAR-100 for 30 epochs using SGD with the initial learning rate 0,01 and a cosine annealing scheduler. The learning objective of robust finetuning exactly follows CRobust (-) shown in Section 2.2 Note that we used the adversarially pre- trentrennd WRN-28- 10 on the entire set released in the Gut of Hendrycks et al. [58 to reproduce the results of the baseline.
The results shown in Table 17 and validate that RCS can significantly accelerate SAT on ImageNet- 1K. Besides, SAT with RCS always obtains much better robustness transferability than SAT with Random and consistently achieves better transferability than standard pre-training as well. Therefore, validates that RCS can be an effective method speed up supervised robust pre-training as well.
Effecitveness of RCS in speeding up SAT on ImageNet- I and maintaining standard transferability 48) of SAT. Salman et al. [48] empirically discovered that adversarially pre-trained models on ImageNet-Ile can yield higher standard test accuracy on downstream tasks than standardly pre-trained models. Following Salman et al. [48], pre- pre-trained ResNet-50 models on ImageNet- IK of 224 x 224 resolution using SAT (34. The model was trained using SGD with 0.9 momentum for 90 epochs with an initial learning rate 0.1 divided by 10 at Epoch 30 and 60. For PGD configurations during SAT, we set 3 = 256, TSAT = 3, <SAT = 4/255, PSAT = 3EsAT
TEAT We set TRcs = during
RCS. We took W = 10 epochs for warmup, and then CS was conducted every / = 10 epochs. We adversarially pre-trained ResNet-50 with different subset fractions i {0.05, 0.1, 0.2] and then conducted standard finetuning on downstream tasks i.e., CIFAR- and CIFAR-
To evaluate the cross-task standard transferability of pre-trained ResNet-50 via SAT on ImageNet-1K of 224 x 224 resolution, we conducted standard linear finetuning (SLF) and standard full finetuning (SFF) on downstream tasks (i.c., CIFAR-10 and CIFAR-100) for 150 epochs using SGD, As for SEF,
27

---

Table 19: Cross-task standard transferability 48 of adversarially pre-trained ResNet-50 from
ImageNet- 1K to CIFAR-10 and CIFAR-I00, respectively. We report the standard test accuracy (%) via standard linear finetuning (SLF) and standard full finetuning (SFF). The number after the dash line denotes subset fraction l E {0.05, 0.1, 0.2).
Runing time (hours)
CIFAR-10
CIFAR-I00
Pre-training
SLF
SFF
SLF
SFF
Standard training [48] on entire set
78.84
97.41
57.09
84.21
SAT 488] on entire set
286.1 1
93.53
98.09
77.29
86.99
Fast-AT [24] on entire set
10.4
90.91
97.54
73.35
83.33
SAT with Random-0.05
38.7
85.72
95.27
69.29
82.34
SAT with RCS-0.05
48.2
92.68
97.65
75.35
84.71
SAT with Random-0.1
45.8
87.14
95.60
71.23
83.62
SAT with RCS-0.1
55.4
92.92
97.82
75.41
85.22
SAT with Random-0.2
70.3
87.69
96.10
72.05
84.14
SAT with RCS-0.2
79.8
93.48
98.06
76.39
85.44
we finetuned all the parameters of the encoder. We set the initial learning rate of 0.001 for standard full finetuning and 0.01 for standard partial finetuning. The learning rate divided by 10 at Epoch 50 and 100. Here, we use the same data augmentation as Salman et al. 48 to resize the images of CIFAR-10 and CIFAR-100 to 224 x 224. Note that we used the standardly pre-trained ResNet-50 on the entire set and adversarially pre-trained ResNet-50 on the entire set released in the GitHub of Salman al. 48 to reproduce the results of baselines.
Table 19ble|19shows shows that RCS substantially accelerates SAT on ImageNet-IK K while consistently achieving higher standard test accuracy on downstream tasks than standard pre-training. Besides, SAT with RCS always obtains much higher standard test accuracy than SAT with Random. It validates that RCS is an effective method for efficient supervised AT.
In addition, we also provide a comparison between the standard transfcrability of Fast-AT [24] and SAT with RCS. We downloaded the pre-trained ResNet-50 vis Fast-AT released in the GitHub of Wong et al. [24]. Although SAT with RCS consumes more running time than Fast-AT, SAT with RCS obtains significantly higher standard test accuracy downstream tasks, which validates the effectiveness of RCS maintaining the transferability of supervised robust pre-training methods.
Therefore, our proposed RCS can be unified and effective framework for speeding up both supervised and unsupervised robust pre-training while maintaining the transferability of pre-trained models.
C Possible Negative Societal Impacts
Our paper aims to speed up robust pre-training by decreasing the number of training data, which decreases the running time of pre-trainn pre-training and should be friendly to the environment. However, to implement all the experiments in this project, have conducted the pre-training procedure least 264 times and the finetuning procedure at least 2376 times, which definitely emitted a bunch of CO2 and consumed large amount of electricity. Therefore, hope that project can help improve the efficiency of robust gre-training and make pre- trainini more environmental-fiie ffrenndll in the future.
28


Reviews:
[{"review_id": "a90AhGltHa", "reviewer": "Reviewer_fJND", "summary": "This paper proposes a coreset selection for efficient adversarial self-supervised learning. By selecting a coreset every epoch that can minimize the representation divergence for training, it maintains similar robustness performance despite a learning speed that is more than three times faster.", "questions": "None", "limitations": "The authors have well described their limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It is technically sound. The fact that adversarial self-supervised learning (ASL) can be trained more than three times faster using coreset selection is especially remarkable.\n- The proposed claims are well supported through theoretical analysis and experiments.", "weaknesses": "- The biggest weakness of the proposed method is that there isn't a significant difference compared to random selection. It utilizes more computation than random selection, but the gain in performance is small. It would be nice if the difference in computation compared to random selection could be explained.\n- Moreover, the originality of the proposed method seems more like a simple application of applying the RD loss to previous work [1,2] rather than a new combination. If the author could explain this more clearly, I would be willing to revise my score.\n- Furthermore, the one-step gradient approximation, warmup, last layer gradients, and adversarial example data approximation also seem to be techniques proposed in previous work, so there seems to be no originality in this regard.\n- There seems to be a lack of justification that a coreset with a small representational divergence is sufficient to gain robustness. And there's not enough explanation on how the results can achieve comparable robustness.\n- It would be helpful if there were explanations, perhaps utilizing representation visualization, as to what samples become the coreset selection, and why it helps with robustness.\n\n[1] Kilamsetty et al., RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning\n\n[2] Kilamsetty et al., Glister: Generalization based data subset selection for efficient and robust learning.\n\n\n==After rebuttal\nI changed my score to 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your positive and thoughtful comments! Please find our responses below.\n\n> 1. [Reply to W1] Our RCS obtains **substantial improvement** compared to random selection (Random). \n\nAccording to Figure 2, we highlight the performance gain of RCS in terms of robustness transferability from CIFAR-10 to STL10. Besides, in Figure 2-4, the solid line (results of RCS) is always far above the dotted line (results of Random), which validates that RCS consistently yields substantial improvement in standard and robust test accuracy compared to Random. \n\nIn terms of computation, RCS spends slightly more pre-training time than Random since RCS needs to spend time in coreset selection every $\\lambda$ epoch.\n|DynACL on CIFAR-10|||ALF on STL10|ALF on STL10|\n|---|---|---|---|---|\n|Subset fraction $k$|Method|Pre-training time (hours)|SA (\\%)|RA (\\%)|\n|0.05|Random|6.5|52.49|26.20|\n|0.05|RCS|7.6 (+1.1)|60.60 (**+8.11**)|32.35 (**+6.15**)|\n|0.1|Random|8.3|54.85|27.85|\n|0.1|RCS|9.4 (+1.1)|62.79 (**+8.21**)|33.89 (**+6.04**)|\n|0.2|Random|11.9|55.86|29.45|\n|0.2|RCS|13.1 (+1.2)|63.41 (**+7.55**)|34.76 (**+5.31**)|\n> 2. [Reply to W2] Simply applying our proposed RD loss to previous work [1,2] cannot obtain our proposed method. \n\nSimply applying RD loss to previous work [1,2] cannot obtain the objective function of RCS. To obtain a coreset, [1,2] formulates a bi-level optimization where they first obtain the parameters via minimizing the inner loss and then find the coreset via minimizing the outer loss. If we simply replace the outer loss of [1,2] with RD loss, the objective function still does not apply to ACL since the inner loss of [1,2] is a natural loss and requires labels. Therefore, to adapt the CS to speed up ACL, we also utilize the ACL loss as inner loss. \n\nBesides, we cannot simply apply the RD loss to the greedy search algorithm in [1,2] for solving our proposed RCS. Since the objective function of RCS is different from [1,2], we are not sure whether the greedy search is applicable to our proposed RCS with an optimality guarantee. Therefore, we provide Theorem 1 and 2, which are basically based on the mathematical knowledge of monotonicity and $\\gamma$-weakly submodularity, to theoretically prove that the algorithm of greedy search can be applied to our proposed RCS with an optimality guarantee. Finally, we proposed our algorithm of RCS via greedy search.\n> 3. [Reply to W3] Besides these four tricks, we propose an unique trick to enable efficient RCS on large-scale datasets with limited GPU memory in Appendix B.1. \n\nRCS on large-scale datasets such as ImageNet-1K needs a large amount of GPU memory to calculate and save the gradient for each minibatch of training data on GPU as the first step (Line 5–7 in Algorithm 1). When the GPU memory is limited, there exists an issue that we are unable to save all the gradients on GPU.\n\nTo solve this issue, we split the entire training set into several training subsets. Then, we conduct RCS on each training subset and combines the coresets from each training subset together as the final coresets for robust pre-training. In this way, we enable RCS to efficiently search for the coresets from large-scale datasets with limited GPU memory. We apply this trick to the experiments regarding ACL on ImageNet-1K in Section 4.2 and SAT on ImageNet-1K in Section 4.3. We believe this trick can help the implementation of RCS on large-scale datasets in a computational resource-restricted environment.\n> 4. [Reply to W4] We show that a lower RD loss (i.e., the KL divergence between natural data and its adversarial variant) on the test set corresponds to better adversarial robustness in Figure 5 (in Appendix).\n\nHere, we copy the results of Figure 5 to help explain.\n|ACL on CIFAR-10||SLF on CIFAR-10|SLF on CIFAR-10| \n|---|---|---|---|\n| | RD loss (lower is better)|SA (\\%)|RA (\\%)| \n|Entire|0.1243|78.87|39.19|\n|Random-0.05|0.3357|67.45|22.96|\n|RCS-0.05|0.1730|72.56|32.49|\n|Random-0.1|0.3094|70.68|27.19|\n|RCS-0.1|0.1695|74.67|34.30|\n|Random-0.2|0.2333|72.01|29.87|\n|RCS-0.2|0.1664|75.96|37.21|\n\nThe table shows that our proposed RCS, whose objective is to find coresets that help minimize the RD loss, indeed minimizes the RD loss. Meanwhile, it empirically shows that a lower RD loss corresponds to higher robust test accuracy. Note that TRADES [1] proposed to obtain adversarial robustness by penalizing the KL divergence between natural data and its adversarial variant which equals our proposed RD loss. Therefore, our proposed RCS can help maintain adversarial robustness by selecting coresets that aim to minimize the RD loss. \n> 5. [Reply to W5] Thanks for your suggestions! We provide a visualisation analysis as follows.\n\nWe count the frequency of each training sample in the CIFAR-10 dataset being selected into the coreset. Then, we visualize the top-5 most-frequently selected (MFS) data and the top-5 least-frequently selected (LFS) data in ***Figure G1*** in the \"global\" file. \n\n***Figure G1*** shows that, compared to LFS data, MFS data are images whose backgrounds are more complex and are more difficult to be distinguished from the subject. Recent work [3,4] has shown that exempting the representations from the nuisanse style factors such as the background factor can improve robustness against distribution shifts. RCS prefers to select the images of complex backgrounds helps the model learn representations that are independent of the background factors, thus helping maintain robustness against adversarial pertubations.\n\nReferences\\\n[1] Kilamsetty et al., RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning\\\n[2] Kilamsetty et al., Glister: Generalization based data subset selection for efficient and robust learning\\\n[3] Representation learning via invariant causal mechanisms. Mitrovic et al., ICLR 2021\\\n[4] Invariant risk minimization. Arjovsky et al., 2020\\\n[5] Theoretically principled trade-off between robustness and accuracy. Zhang et al., ICML 2019"}, {"review_id": "Cqe0WDAXE3", "reviewer": "Reviewer_LfDi", "summary": "This paper proposes a robustness-aware coreset selection (RCS) method, which is applied to accelerate adversarial contrast learning (ACL) in the absence of labeling information. Especially, the coreset searched by RCS minimizes the representation difference between the natural data and their adversarial examples, which is achieved by a greedy search method. And experimental results demonstrate that RCS can indeed speed up ACL without signiﬁcantly compromising the robustness transferability.", "questions": "Please refer to the weaknesses part.", "limitations": "Please refer to the weaknesses part.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and easy to follow.\n2. The coreset searched by RCS is not only small in number but also beneficial in improving the adversarial robustness of representations.\n3. Experimental results demonstrate that RCS can indeed speed up ACL without signiﬁcantly compromising the robustness transferability.\n", "weaknesses": "1. This paper points out that ACL with RCS trains the model on the previously selected coreset for $\\lambda$ epochs, and for every $\\lambda$ epochs a new coreset is selected. So, does the value of $\\lambda$ affect the effectiveness as well as the efficiency of ACLs? It may be worthwhile for the authors to make further discussion.\n2. In the experimental part, it can be found that the coreset searched by RCS without label information achieves better performance compared to the coreset searched by ACS, or even the whole dataset, as shown in Tables 12 and 13. How to explain this interesting phenomenon?\n3. The core of this paper is the coreset found by RCS, so the authors should have given a more focused discussion, such as whether it is class-balanced and what the distribution of the subset is.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your positive and constructive comments! Please find our responses below.\n\n> 1. [Reply to W1] A larger $\\lambda$ leads to more pre-training time and higher robust and standard test accuracy in downstream tasks.\n\nWe pre-trained ResNet-18 on CIFAR-10 via ACL with RCS using $\\lambda \\in \\\\{ 10,20,50\\\\}$, $k=0.2$, and $\\omega=100$. Then, we evaluate the performance on CIFAR-10 via SLF.\n\n| | | SLF on CIFAR-10 | SLF on CIFAR-10 |\n|---|---|---|---|\n| $\\lambda$| Pre-training time (hours) | SA (\\%) | RA (\\%) |\n| 10 | 15.6 | 76.31 | 38.17 |\n| 20 | 13.0 | 75.96 | 37.21 |\n| 50 | 12.2 | 75.87 | 35.54 |\n\nThe above table shows that a larger $\\lambda$ leads to more pre-training time since the frequency of conducting coreset selection becomes larger. \n\nMeanwhile, a larger $\\lambda$ can lead to higher robust and standard test accuracy in downstream tasks. This is because a larger $\\lambda$ enables the coreset to be updated more timely to adapt to the latest state of the model and select data points that can help the model to improve its robustness.\n\n> 2. [Reply to W2] Thanks for pointing out this interesting phenomenon! We provide an explanation below. \n\nThe phenomenon shown in Table 12-13 is that RCS can speed up Fast-AT and Free-AT while maintaining robust test accuracy. However, RCS sacrifices some natural test accuracy compared to the entire training set. For example, Fast-AT with RCS improves robust test accuracy by 0.02\\% while degrading natural test accuracy by 2.73\\% compared to Fast-AT on the entire set (Fast-AT-Entire).\n\nHere, we provide an explanation. We empirically find that the RD losses on the CIFAR-10 test set of ResNet-18 trained by Fast-AT-Entire, Fast-AT with ACS, and Fast-AT with RCS are 0.0394, 0.0423, and 0.0375, respectively. It indicates that our proposed RCS, whose objective is to find coresets that help minimize the RD loss, indeed minimizes the RD loss. Note that TRADES [1] proposed to obtain adversarial robustness by penalizing the KL divergence between natural data and its adversarial variant which equals our proposed RD loss.  Therefore, our proposed RCS can help maintain and even slightly improve adversarial robustness by selecting coresets that aim to minimize the RD loss.\n\nBesides, our proposed RCS actually utilizes the information of the entire training set during selecting a coreset. In particular, RCS dynamically updates the coreset according to the entire training set and the latest model parameters every $\\lambda$ epoch. In this regard, RCS does not use less information compared to the entire training set. Therefore, RCS can achieve the robustness comparable to the entire training set.\n\n> 3. [Reply to W3] Thanks for your great suggestions! We demonstrate that RCS can select a coreset that is closer to the full training set than Random.\n\nNote that the imbalance ratio [2] is the ratio of the sample size of the largest majority class and that of the smallest minority class. Maximum mean discrepancy (MMD) [3] based on the Guassian kernel is a classical measurement of the distance between two distributions.\n\nThe left panel of ***Figure G2*** in the \"global\" file shows that the corset selected by RCS is almost class-balanced since the imbalance ratio of RCS is slightly higher than 1.0. The right panel of ***Figure G2*** in the \"global\" file shows that RCS yields a lower MMD between the entire training set and the selected coreset compared to Random.\nTherefore, our quantitative analysis demonstrates that RCS generates a coreset that is closer to the entire training set than Random.\n\n*References*\n\n[1] Zhang, Hongyang, et al. \"Theoretically principled trade-off between robustness and accuracy.\" International conference on machine learning. PMLR, 2019.\\\n[2] Ortigosa-Hernández, J., Inza, I., & Lozano, J. A. (2017). Measuring the class-imbalance extent of multi-class problems. Pattern Recognition Letters, 98, 32-38.\\\n[3] Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., & Smola, A. (2012). A kernel two-sample test. The Journal of Machine Learning Research, 13(1), 723-773."}, {"review_id": "f9OrMW9nA3", "reviewer": "Reviewer_ZwZm", "summary": "This paper introduces a robustness-aware coreset selection (RCS) method without requiring label information to speed up adversarial contrastive learning. RCS selects an informative training subset that minimizes the representational divergence (RD) between adversarial and natural data. Theoretically, the authors prove a greedy search algorithm can solve a proxy problem and provide the lower bound of the solution. Empirically, comprehensive experimental results demonstrate RCS can significantly speed up ACL while maintaining robustness transferability. To my knowledge, it is the first effort to apply ACL on the large-scale datasets ImageNet-1K and obtain effective robust representations.", "questions": "See weakness.", "limitations": "The limitations and the possible negative societal impacts of this submission have been adequately discussed by the authors in the appendix.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "(1) The proposed method is reasonable and novel. The authors propose to select subsets guided by the RD between natural data and its adversarial variant. RCS does not need label annotations while the existing related work requires labels during coreset selection, which supports the originality of the proposed method. This paper is the first to obtain robust representations by ACL pre-training on ImageNet-1K efficiently via RCS. I think this paper has adequately cited the related work.\n\n(2) The submission is technically sound. The claims are well supported by theoretical analyses and experimental results. Theoretically, the authors prove solving a proxy problem efficiently via the greedy search can guarantee the optimality of the solution for the original problem. Empirically, the authors apply RCS to speed up ACL and its variant DynACL on various datasets and show that RCS can maintain natural and robust test accuracy on various downstream tasks. Besides, the authors provide extensive results that validate RCS can be applied to accelerate supervised adversarial training including Madry’s and TRADES on CIFAR-10 and ImageNet-1K.\n\n(3) Self-supervised robust pre-training can provide improved robustness transferability without requiring label annotations. However, due to the computational prohibition, ACL methods have not been applied to large-scale datasets previously. This paper solves this important issue and enabled ACL to be conducted on large-scale datasets, tackling a very practical meaningful chanllenge.\n\n(4) This paper is well-organized and well-written. The reviewer can easily follow most of the content.\n", "weaknesses": "Minor comment - although the authors propose to use the greedy search algorithm to efficiently search the coreset, it still needs to consume extra time for CS during robust pre-training. How to further improve the efficiency of ACL and maintain its effectiveness should be an interesting future direction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your positive and constructive comments! \n\n> [Reply to Weakness] We believe this is an interesting future direction!\n\nWe conjecture that applying a better submodular function optimization method to solve the objective function of our proposed RCS can further improve efficiency. For example, [1] has shown the greedy algorithm can be further accelerated through lazy evaluations. [2] proposes to efficiently approximate the set function with some relative error when the set function itself is difficult and computationally expensive to calculate. Meanwhile, [2] shows that most methods of maximizing submodular functions are robust against such errors.\n\n*References*\n\n[1] Krause, Andreas, and Daniel Golovin. \"Submodular function maximization.\" Tractability 3.71-104 (2014): 3.\\\n[2] Golovin, Daniel, and Andreas Krause. \"Adaptive submodularity: Theory and applications in active learning and stochastic optimization.\" Journal of Artificial Intelligence Research 42 (2011): 427-486."}, {"review_id": "YpMgrLBMW3", "reviewer": "Reviewer_HkKZ", "summary": "This paper focuses on the problem of efficient adversarial contrastive learning, the authors propose Robustness-Aware Coreset selection to speed up ACL, and according to the theoretical analysis and experimental results, the proposed framework is effective while does not hurt the performance.", "questions": "Please refer to the previous section", "limitations": "The authors discussed limitations in the paper appendix.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The topic is interesting and the proposed method is simple yet effective;\n2. The experiments are well-designed and solid. Also, several runs are performed on each task, making it more convincing;\n3. The writing is easy to follow;\n4. The proofs are well-written and sound.", "weaknesses": "1. According to the experimental results, it still takes much time to run the proposed algorithm. While the authors discussed it in the limitations, it might a challenge for it to be put into practical use;\n2. Figure 2,3,4 are a bit hard to read, might need some re-work;\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your supportive and constructive comments! Please find our replies below.\n\n> 1. [Reply to W1] Thanks for pointing out this challenge!\n\nWe conjecture that applying better submodular function optimization methods to solve the objective function of our proposed RCS can further improve efficiency. For example, [1] has shown the greedy algorithm can be further accelerated through lazy evaluations. [2] proposes to efficiently approximate the set function with some relative error when the set function itself is difficult and computationally expensive to calculate. Meanwhile, [2] shows that most methods of maximizing submodular functions are robust against such errors.\n\nBesides, our work is orthogonal to large batch optimization [3] in practice. We conjecture that incorporating our proposed method with the large batch optimization methods would be practical for efficiently learning robust representations using large-scale datasets and large models.\n\n> 2. [Reply to W2] Thanks for the constructive suggestion! We will provide more explanations in the captions of the figures in revisions.\n\nWe will add clarifications in the captions: ACL with RCS and DynACL with RCS correspond to the red and orange solid lines, respectively. ACL with Random and DynACL with Random correspond to the blue and green dotted lines, respectively. \n\n*References*\n\n\n[1] Krause, Andreas, and Daniel Golovin. \"Submodular function maximization.\" Tractability 3.71-104 (2014): 3.\\\n[2] Golovin, Daniel, and Andreas Krause. \"Adaptive submodularity: Theory and applications in active learning and stochastic optimization.\" Journal of Artificial Intelligence Research 42 (2011): 427-486.\\\n[3] Large Batch Optimization for Deep Learning: Training BERT in 76 minutes, You et al., ICLR 2020."}]

Author rebuttal:
[**Rebuttal Highlights**]

Many thanks for all reviewers' supportive and constructive comments! 

Following reviewers' suggestions, we uploaded extensive ***Figure G1*** and ***Figure G2*** in the "**global**" file to provide a focused discussion of the coreset.

> 1. [For Reviewer **fJND**] In ***Figure G1***, we provide a visualization analysis to interpret why the coreset selected by RCS can help maintain robustness.

We count the frequency of each training sample in the CIFAR-10 dataset being selected into the coreset. Then, we visualize the top-5 most-frequently selected (MFS) data and the top-5 least-frequently selected (LFS) data in ***Figure G1***.

***Figure G1*** shows that, compared to LFS data, MFS data are images whose backgrounds are more complex and are more difficult to be differentiated from the subject. Recent work [1,2] has shown that exempting the representations from the nuisanse style factors such as the background factor can improve the robustness against distribution shifts. RCS prefers to select the images of complex backgrounds helps the model learn representations that are independent of the background factors, thus helping maintain robustness against adversarial pertubations.

> 2. [For Reviewer **LfDi**] In ***Figure G2***, we demonstrate that RCS can select a coreset that is closer to the full training set than Random.

Note that the imbalance ratio [3] is the ratio of the sample size of the largest majority class and that of the smallest minority class. Maximum mean discrepancy (MMD) [4] based on the Guassian kernel is a classical measurement of the distance between two distributions.

The left panel of ***Figure G2*** shows that the corset selected by RCS is almost class-balanced since the imbalance ratio of RCS is slightly higher than 1.0. The right panel of ***Figure G2*** shows that RCS yields a lower MMD between the entire training set and the selected coreset compared to Random. Therefore, our quantitative analysis demonstrates that RCS selects a coreset that is closer to the entire training set than Random.

*References*

[1] Representation learning via invariant causal mechanisms. Mitrovic et al., ICLR 2021.\
[2] Invariant risk minimization. Arjovsky et al., 2020.\
[3] Ortigosa-Hernández, J., Inza, I., & Lozano, J. A. (2017). Measuring the class-imbalance extent of multi-class problems. Pattern Recognition Letters, 98, 32-38.\
[4] Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., & Smola, A. (2012). A kernel two-sample test. The Journal of Machine Learning Research, 13(1), 723-773.

Meta-review: There is consensus among all four expert referees that the paper should be accepted. During the initial reviewing phase, there were some questions raised, mainly about the experiments. The rebuttal provided further clarifications, which should be incorporated in the final version of the paper.
Decision: Accept (spotlight)