Title: On Adversarial Training without Perturbing all Examples
Venue: NeurIPS 2023
Accepted: no
Keywords: adversarial robustness

adversarial training

adversarial robust transfer
Abstract: Adversarial training is the de-facto standard for improving robustness against adversarial examples. This usually involves a multi-step adversarial attack applied on each example during training. In this paper, we explore only constructing adversarial examples (AE) on a subset of the training examples. That is, we split the training set in two subsets $A$ and $B$, train models on both ($A\cup B$) but construct AEs only for examples in $A$. Starting with $A$ containing only a single class, we systematically increase the size of $A$ and consider splitting by class and by examples. We observe that: (i) adv. robustness transfers by difficulty and to classes in $B$ that have never been adv. attacked during training, (ii) we observe a tendency for hard examples to provide better robustness transfer than easy examples, yet find this tendency to diminish with increasing complexity of datasets (iii) generating AEs on only $50$% of training data is sufficient to recover most of the baseline AT performance even on ImageNet. We observe similar transfer properties across tasks, where generating AEs on only $30$% of data can recover baseline robustness on the target task. We evaluate our subset analysis on a wide variety of image datasets like CIFAR-10, CIFAR-100, ImageNet-200 and show transfer to SVHN, Oxford-Flowers-102 and Caltech-256. In contrast to conventional practice, our experiments indicate that the utility of computing AEs varies by class and examples and that weighting examples from $A$ higher than $B$ provides high transfer performance.

Abstract

Adversarial Training (AT) is the de-facto standard for improving robustness against adversarial examples. This usually involves a multi-step adversarial attack applied on each example during training. In this paper, we explore only constructing Adversarial Examples (AEs) on a subset of the training examples. That is, we split the training set in two subsets A and B , train models on both ( A ∪ B ) but construct AEs only for examples in A . Starting with A containing only a single class, we systematically increase the size of A and consider splitting by class and by examples. We observe that: (i) adv. robustness transfers by difficulty and to classes in B that have never been adv. attacked during training, (ii) we observe a tendency for hard examples to provide better robustness transfer than easy examples, yet find this tendency to diminish with increasing complexity of datasets (iii) generating AEs on only 50% of training data is sufficient to recover most of the baseline AT performance even on ImageNet. We observe similar transfer properties across tasks, where generating AEs on only 30% of data can recover baseline robustness on the target task. We evaluate our subset analysis on a wide variety of image datasets like CIFAR-10, CIFAR-100, ImageNet-200 and show transfer to SVHN, OxfordFlowers-102 and Caltech-256. In contrast to conventional practice, our experiments indicate that the utility of computing AEs varies by class and examples and that weighting examples from A higher than B provides high transfer performance.

1 Introduction

Imperceptible changes in the input can change the output of a well performing model dramatically. These so-called Adversarial Examples (AEs) have been the focus of a large body on deep learning vulnerabilities of works since its discovery [1]. To date, Adversarial Training (AT) [2, 3] and its variants [4-6] is the de-facto state-of-the-art in improving the robustness against AEs. Essentially, AT generates adversarial perturbations for all examples seen during training. While adversarial training is known to transfer robustness to downstream tasks [7-9] and that robustness is distributed unevenly across classes [10, 11], common practice dictates that AT 'sees' adversarial examples corresponding to the whole training data, including all classes and concepts therein. This is independent of whether only adversarial robustness is optimized or a trade-off between robustness and clean performance is desired [12]. This also holds for variants that treat individual examples differently [13-15] or adaptively select subsets to attack during training to reduce computational overhead [16, 17]. It is largely unclear how adversarial robustness is affected when training is limited to seeing adversarial examples only on specific subsets of the training data.

To shed light on this issue, we consider the adversarial training setup depicted in figure 1, called 34

Subset Adversarial Training (SAT), where we split the training data into two subsets A and B , train 35

the model conventionally on the union ( A ∪ B ), but generate AEs only on examples from A (indicated 36

𝜖

by the emoji). For example, we can split training data by class, with A = { car } or A = { cat } and 37 B = A c , and investigate how adversarial robustness transfers. Surprisingly, we observe significant 38 adversarial robustness on B val at test time, the degree of which depends on the class(es) in A . Of 39 course, A and B can be arbitrary partitions of the training data. For example, we could put only 40 'difficult' examples in A during training. At test time, we evaluate overall adversarial robustness 41 (since there is no natural split into A val or B val). These experiments reveal a rather complex interaction 42 of adversarial robustness between classes and examples. 43

Our analysis provides a set of contributions revealing a surprising generalizability of robustness 44 towards non-adv. trained classes and examples even under scarce training data setups. First , selecting 45 subsets of whole classes, we find that SAT provides transfer of adversarial robustness to classes which 46 have never been attacked during training. E.g. only generating adversarial examples for class car on 47 CIFAR-10, achieves a non-trivial robust accuracy of 17 . 1% on all remaining CIFAR-10 classes (see 48 figure 1, right). Secondly , we observe classes and examples that are hard to classify do generally 49 provide better robustness transfer than easier ones. I.e. class cat achieves more than twice the robust 50 accuracy on the remaining classes ( 37 . 8% ) over class car ( 17 . 1% ). Thirdly , SAT with 50% of 51 training data is sufficient to recover the baseline performance with vanilla AT even on hard datasets 52 like ImageNet. Lastly, we observe similar transfer properties of SATed models to downstream tasks. 53 In this setting, exposing the model to only 30% of AEs during training, can recover baseline AT 54 performance on the target task. 55

2 Related Work 56

Since their discovery [1], robustness against adversarial examples has mainly been tackled using 57 adversarial training [18, 2, 4]. Among many others, prior work proposed adversarial training variants 58 working with example-dependent threat models [19, 13-15], acknowledging that examples can have 59 different difficulties. Some works also mine hard examples [16] or progressively prune a portion of 60 the training examples throughout training [17, 20]. However, all of these methods generally assume 61 access to adversarial examples on the whole training set. That is, while individual examples can 62 be dropped during training or are treated depending on difficulty, the model can see adversarial 63 perturbations for these examples if deemed necessary. Adversarial training is also known to transfer 64 robustness to downstream tasks [8, 9, 7] and adversarially robust representations can be learned 65 in a self-supervised fashion [21]. Here, a robust backbone is often adapted to the target task by 66 re-training a shallow classifier - sometimes in an adversarial fashion. It is generally not studied 67 whether seeing adversarial examples on the whole training set is required for good transfer. This is 68 despite evidence that achieving adversarial robustness is easier for some classes/concepts than for 69 others [22, 23, 11, 10], also for robustness transfer [24]. Complementing these works, we consider 70 only constructing adversarial examples on a pre-defined subset of the training set, not informed by 71 the model or training procedure, and study how robustness transfers across examples and tasks. 72

3 Background and Method 73

3.1 Adversarial Training (AT) 74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

It is a well known fact that conventional deep networks are vulnerable to small, often imperceptible, changes in the input. As mitigation, AT is a common approach to extend the empirical risk minimization framework [2]. Let ( x, y ) ∈ D train be a training set of example and label pairs and θ be trainable parameters, then AT is defined as:

where δ is a perturbation that maximizes the training loss L and thus training error. The idea being that, simultaneously to minimizing the training loss, the loss is also optimized to be stable within a small space ϵ around each training example || δ || 2 ≤ ϵ (we consider the L 2 norm). This additional inner maximization is solved by an iterative loop; conventionally consisting of 7 or more steps. In some settings [18, 12, 4], the robust loss is combined with the corresponding loss on clean examples in a weighted fashion to control the trade-off between adversarial robustness and clean performance.

3.2 AT without Perturbing all Training Examples

Most proposed AT methodologies generate AEs on the whole training set. This being also valid for methods which adaptively select subsets [16, 17] during training or more traditional AT in which only a subset per batch is adversarially attacked. These methods do not guarantee the exclusion of examples, that is, the model is likely to see an AE for every example in the training set. From a broader perspective, the necessity to generate AEs exhaustively for all classes appears unfortunate though. Ideally, we desire robust models to be scalable, i.e. transfer flexibly from few examples and across classes to unseen ones [25]. We propose SAT to investigate to what extent AT provides this utility. To formalize, let A be a training subset and B contain the complement: A ⊂ D train , B = D train \ A . Then SAT applies the inner maximization loop of AT on the subset A only; on B the conventional empirical risk is minimized:

where ✶ ( x,y ) ∈ A is 1 when the training example is in A and 0 otherwise. w A and w B define optional weights, which are by default both set to 1 . Note that this is different from balancing robust and clean loss as discussed in [18, 12, 4], where the model still encounters adversarial examples on the whole training set.

̸

Loss balancing. The formulation in equation 2 implies an imbalance between left and right loss as soon as the training split is not even ( | A | = | B | ). To counteract, we assign different values to w A and w B based on their subset size. E.g., to equalize the loss between both subsets, we assign w B = 1 and w A = | B | / | A | . We will utilize this loss balancing to improve robustness for transfer learning in section 4.3.

3.3 Training and evaluation recipes

Consider the depiction of SAT in figure 1. Prior to training, the training set is split into A and B (left). For evaluation (middle), we split the validation set into a corresponding split of A val and B val , if possible. For Class-subset Adversarial Training (CSAT) , this split aligns with the classes on the dataset: A and B are all training examples corresponding to two disjoint sets of classes while A val and B val are the corresponding test examples of these classes. As experimenting with all possible splits of classes is infeasible, we motivate splits by class difficulty where we measure difficulty by the average entropy of predictions per class - introduced as H C in the next paragraph. In contrast, we can also split based on individual example difficulty. We provide empirical support for this approach in the experimental section 4. Additionally, example difficulty has been frequently linked to proximity between decision boundary and example [26, 13, 15, 16, 27]. The closer the example is to the boundary, the harder it is likely to classify. The hypothesis: hard examples provide a larger

contribution to training robust models, since they optimize for large margins [13, 14]. We refer to 117 this experiment as Example-subset Adversarial Training (ESAT) . In contrast to CSAT, however, 118 there is no natural split of the test examples into A val and B val such that we evaluate robustness on 119 the whole test set (i.e., D val ). 120

121

122

123

124

125

As difficulty metric, we utilize entropy over softmax, which we empirically find to be as suitable as alternative metrics (discussed in the supplement). Consider a training set example x ∈ D train and a classifier f mapping from input space to logit space with N logits. Then the entropy of example x is determined by H ( f ( x )) and of a whole class C ⊂ D train is determined by H C ( f ) - the average over all examples in C :

where σ denotes the softmax function. For our SAT setting, we rank examples prior to adversarial 126 training. This requires a classifier pretrained on D train enabling the calculation of the entropy. To 127 strictly separate the effects between entropy and AT, we determine the entropy using a non-robust 128 classifier trained without AT. Similar to [27], we aggregate the classifier states at multiple epochs 129 during training and average the entropies. Let f 1 , f 2 , ...f M be snapshots of the classifier from multiple 130 epochs during training, where M denotes the number of training epochs. Then the average entropy 131 for an example is given by H ( x ) and for a class by H C ( f ) : 132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

4 Experiments

As aforementioned, common practice performs AT for the whole training set. In the following, we explore CSAT and ESAT, which splits the training set in two subsets A and B and only constructs AEs for A such that the model never sees AEs for B . We start with single-class CSAT A contains only examples of a single class - and increase the size of A (section 4.1) by utilizing the entropy ranking of classes H C (equation 3). ESAT, which splits into example subsets is discussed in section 4.2. Both SAT variants reveal complex interactions between classes and examples while indicating that few AEs can provide high transfer performance to downstream tasks when weighted appropriately (section 4.3).

Training and evaluation details. Since AT is prone to overfitting [28], it is common practice to stop training when robust accuracy on a hold-out set is at its peak. This typically happens after a learning rate decay. We adopt this 'early stopping' for all our experiments by following the methodology in [28] but utilize Auto Attack (AA) to evaluate robust accuracy. Throughout the course of the training, we evaluate AA on 10% of the validation data D val after each learning rate decay and perform final evaluation with the model providing the highest robust accuracy. This final evaluation is performed on the remaining 90% of validation data. This AA split is fixed throughout experiments to provide consistency. If not specified otherwise, we generate adversarial examples during training with PGD-7 within an L 2 epsilon ball of ϵ = 0 . 5 (all CIFAR variants) or ϵ = 3 . 0 (all ImageNet variants) - typical configurations found in related work. We train all models from scratch and use ResNet-18 [29] for all CIFAR-10 and CIFAR-100 [30] experiments and ResNet-50 for all ImageNet-200 experiments. Here, ImageNet-200 corresponds to the ImageNet-A subset [31] to render random baseline experiments tractable (to reduce training time). This ImageNet-200 dataset, contains 200 classes that retain the class variety and breadth of regular ImageNet, but remove classes that are similar to each other (e.g. fine-grained dog types). We use all training and validation examples from ImageNet [32] that correspond to this subset classes. All training details can be found in the supplement.

4.1 Class subset splits

We start by investigating the interactions between individual classes in A using CSAT on CIFAR-10, followed by an investigation on increasing the number of classes. Single-class subsets (CSAT). We train all possible, single class CSAT runs ( 10 ) and evaluate robust accuracies on the adv. trained class (A) and the non-adv. trained classes (B). The results are shown in figure 2, left. Each rows

20

40

60

80

100

Robust accuracy (

L

2

=0.5)

plane

car

bird

cat

deer

dog

frog

horse

ship

truck

ATed class

+25.7%

+17.1%

+34.3%

+37.8%

+34.2%

+34.9%

+28.4%

+27.3%

+24.5%

+21.9%

= 0%

no AT

full AT

ATed class

Non-ATed classes

A

B

plane

car

bird

cat

deer

dog

frog

horse

ship

truck

Robust accuracy

plane

car

bird

cat

deer

dog

frog

horse

ship

truck

ATed class

89

57

15

5

10

16

16

38

28

46

28

94

9

3

6

9

8

32

24

35

33

57

80

12

12

24

28

45

41

56

41

61

28

50

16

25

28

50

37

53

36

62

24

8

84

24

26

41

35

52

34

55

28

11

16

61

30

49

36

54

27

58

17

10

11

18

93

42

27

44

31

55

17

6

10

22

24

89

33

47

34

56

12

5

9

10

12

35

92

47

33

35

15

3

9

18

14

37

32

92

Full AT

77

85

53

41

60

53

77

79

81

81

represents a different training run. Note that the baseline robust accuracy, trained without AT achieves 163 practically 0% (indicated by red line). Most importantly, we observe non-trivial robustness gains for 164 all classes that have never been attack during training ( B -sets). That is, irrespective of the chosen 165 class, we gain at least 17 . 1% robust accuracy (A= car ) on the remaining classes and can gain up to 166 37 . 8% robust accuracy when A= cat . These robustness gains are unexpectedly good, given many 167 features of the non-adv. trained classes can be assumed to not be trained robustly. 168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

plane

car

bird

cat

deer

dog

frog

horse

ship

truck

0.0

0.04

0.08

0.12

0.00

0.13

0.27

0.40

Entropy

H

C

Many-class subsets (CSAT). To increase the number of classes in A while maintaining a minimal computational complexity, we utilize the average class entropy H C proposed in equation 3 to inform us which ranking to select from. To improve clarity, we begin with a reduced set of experiments on CIFAR-10 before transitioning to larger datasets. We utilize the observed correlation between class difficulty, average class entropy and robustness transfer H C to rank classes and construct 4 adv. trained subsets. Ranked by class entropy H C , we select 4 subsets showing in figure 4, left. As observed before, cat and dog are hardest and thus first chosen to be in subset A . Truck and car on the other hand are easiest and thus last. To gauge the utility of this ranking, we provide a robust and clean accuracy comparison with a random baseline in figure 4, center and right. I.e., for each subset A we select 10 random subsets and report mean and std. deviation (red line and shaded area). Similar to the single-class setup, we observe subsets of the hardest classes to consistently outperform the random baseline (upper middle plot), up until a subset size of | A | = 8 , when it draws even. Also note that the robust accuracy on B val is improved across all splits, thus providing support that harder classes - as initially observed on animate vs inanimate classes - offer greater robustness transfer.

Class error To investigate this phenomenon further, we analyze robust gains for each individual class and present robust accuracies in the matrix in figure 2, right, where training runs are listed in rows and robust accuracies per class are listed in columns. Blue cells denote the adv. trained class and green cells denote non-adv. trained classes. While we see some expected transfer properties, e.g. CSAT on car provides greater robust accuracy on the related class truck ( 46% ) than unrelated animal classes bird, cat, deer, dog (between 5% and 16% ), the reverse is not straight-forward. CSAT on bird provides 56% robust accuracy on the seemingly unrelated class truck , 10% -points more than CSAT on car . More generally, animal classes provide stronger robustness throughout all classes than inanimate classes. We observe, that these classes are also harder to classify and have a higher entropy H C as shown in figure 3.

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

Data splits

cat

dog

cat

dog

bird

deer

cat

dog

bird

deer

plane

frog

cat

dog

bird

deer

plane

frog

horse

ship

truck

car

horse

ship

truck

car

plane

frog

horse

ship

truck

car

bird

deer

plane

frog

horse

ship

truck

car

A

B

2

4

6

8

|A|

Performance on

CIFAR-10val

40

50

60

70

Rob. accuracy

2

4

6

8

10

Number of adv. trained classes (

A

)

85

.

0

87

.

5

90

.

0

92

.

5

95

.

0

Clean accuracy

full AT

split as left

random class split

Performance on

Non-adv. trained classes (

B

val

)

0

25

50

75

Rob. accuracy

2

4

6

8

10

Number of adv. trained classes (

A

)

80

85

90

95

100

Clean accuracy

no AT

split as left

random class split

|

|

|

|

For our experiments on larger datasets like CIFAR-100 and ImageNet-200, we additionally evaluate a third ranking strategy. Beside selecting at random and selecting the hardest first, we additionally compare with selecting the easiest (inverting the entropy ranking). We construct 9 subsets per type of ranking (instead of 4 ) and report robust accuracies for selecting the easiest classes as well. Results are presented in three columns in figure 5; one dataset per column. As before, we show robust accuracies on the tested dataset (upper row) and robust accuracies on B val (lower row). For CIFAR-10, we calculate mean and std. dev. over 10 runs, for CIFAR-100 over 5 runs and for ImageNet-200 over 3 runs. Selecting hardest first (highest entropy) is marked as a solid line and easiest first (lowest entropy) as a dashed line. First and foremost, we observe that irrespective of the dataset and the size of A , we see robustness transfer to B val. This transfer remains greatest with classes we consider hard, while easy classes provide the least. Nonetheless, we see diminishing returns of such an informed ranking when dataset complexity is increased. E.g. the gap between dashed and solid line on ImageNet-200 is small and random class selection is on-par with the best. The results are similar on CIFAR-100, as shown in figure 5, middle). Based on these results, entropy ranking and selecting classes provides only slight improvements in general. Importantly though, we continue to see the tendency of increased robustness transfer to B val, which we will come back to in section 4.3.

On

B

val

On whole dataset

CIFAR-10

40

60

Rob. accuracy

1

2

3

4

5

6

7

8

9

10

Number of adv. trained classes (

A

)

0

25

50

75

Rob. accuracy

CSAT-hardest first

CSAT-easiest first

CSAT-random

one std.

dev.

full AT

no AT

CIFAR-100

20

30

40

Rob. accuracy

10

20

30

40

50

60

70

80

90

100

Number of adv. trained classes (

A

)

0

20

40

60

Rob. accuracy

CSAT-hardest first

CSAT-easiest first

CSAT-random

one std.

dev.

full AT

no AT

ImageNet-200

20

40

Rob. accuracy

20

40

60

80

100

120

140

160

180

200

Number of adv. trained classes (

A

)

0

20

40

60

Rob. accuracy

CSAT-hardest first

CSAT-easiest first

CSAT-random

one std.

dev.

full AT

no AT

|

|

|

|

|

|

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

4.2 Example subset splits (ESAT)

Considering that splits along classes are inefficient in terms of reaching the full potential of adversarial robustness, we investigate ranking examples across the whole dataset (ESAT). We follow with the same setup as before but rank examples - and not classes - by entropy H . Since it is not feasible to construct corresponding rankings on the validation set, we cannot gauge robustness transfer to B val . Instead, we will test transfer performance to downstream tasks in section 4.3. We consequently report robust accuracy and clean accuracy on the whole validation set in figure 6.

Firstly, note that the increase in robust accuracy is more rapid than with CSAT w.r.t. the size of A . AT only on 50% of training data ( 25 k examples on CIFAR and 112 k on ImageNet-200) and the resulting average robust accuracy is very close to the baseline AT performance (gray line). Secondly, note that gap between hard (solid line) and easy example selection (dashed line) has substantially widened. In practice, it is therefore possible to accidentally select poor performing subsets, although the chance appears to be low given the narrow variance of random rankings (red). To some extent, this observation supports the hypothesis that examples far from the decision border (the easiest to classify) provide the least contribution to robustness gains. This is also supported by the reverse gap in clean accuracy (bottom row in figure 6). That is, easiest-first-selection results in higher clean accuracies than hardest-first, while robust accuracies are much lower. In contrast however, we observe random rankings (red) to achieve similar performances to hard rankings (solid lines) on all datasets and subset sizes. This is somewhat unexpected, especially on small sizes of A (e.g. 5 k ). Given the results, we conjecture that the proximity to the decision boundary plays a subordinate role to increasing robustness. Instead, it is plausible to assume that diversity in the training data has a large impact on learning robust features, also indicated by [33].

On whole dataset

CIFAR-10

40

60

Rob. accuracy

5k

10k

15k

20k

25k

30k

35k

40k

45k

50k

Number of adv. trained examples (

A

)

90

92

94

Clean accuracy

ESAT-hardest first

ESAT-easiest first

ESAT-random

one std.

dev.

full AT

CIFAR-100

10

20

30

40

Rob. accuracy

5k

10k

15k

20k

25k

30k

35k

40k

45k

50k

Number of adv. trained examples (

A

)

70

75

Clean accuracy

ESAT-hardest first

ESAT-easiest first

ESAT-random

one std.

dev.

full AT

ImageNet-200

20

40

Rob. accuracy

22k

44k

67k

89k

112k 134k 156k 179k 201k 224k

Number of adv. trained examples (

A

)

75

.

0

77

.

5

80

.

0

82

.

5

Clean accuracy

ESAT-hardest first

ESAT-easiest first

ESAT-random

one std.

dev.

full AT

|

|

|

|

|

|

4.3 Transfer to downstream tasks

Previous experiments on ESAT could not provide explicit robust accuracies on the non-adv. trained subset B val since training and testing splits do not align naturally - recall the evaluation recipe outlined in section 3.3. In order to test transfer performance regardless, we make use of the fixed-feature task transfer setting proposed in [7]. The recipe just slightly changes: split the data into A and B as usual and perform SAT. Fix all features, replace the last classification layer with a 1 -hidden layered classifier and finetune only the new classifier on the target task. Importantly, neither training nor validation set for the target task are split. We consider CIFAR-100 and ImageNet-200 and transfer to CIFAR-10, SVHN, Caltech-256 [35] and Flowers102 [36]. We call SAT trained for transfer Source-task Subset Adversarial Training (S-SAT), to emphasize that the subset training is performed on the source-task dataset.

In this section, we consider models that have 'seen' only a fraction of AEs on the source task and 245 investigate the robustness transfer capabilities to tasks on which they have not explicitly adversarially 246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

On adv. trained

CIFAR-100 classes (

A

val

)

On non-adv. trained

CIFAR-100 classes (

B

val

)

On non-adv. trained

downstream task

CIFAR-10val

10

20

30

40

Rob. accuracy

10

20

30

40

50

60

70

80

90

100

Number of adv. trained classes (

A

)

50

55

60

65

Clean accuracy

vanilla

w

A

=

|

B

|

/

|

A

|

w

A

= 2

→

10

|

B

|

/

|

A

|

full AT

20

40

60

Rob. accuracy

10

20

30

40

50

60

70

80

90

100

Number of adv. trained classes (

A

)

60

80

Clean accuracy

vanilla

w

A

=

|

B

|

/

|

A

|

w

A

= 2

→

10

|

B

|

/

|

A

|

20

30

40

Rob. accuracy

CIFAR10

10

20

30

40

50

60

70

80

90

# of adv. trained classes (

A

) on CIFAR100

75

80

Clean accuracy

CIFAR10

vanilla

w

A

=

|

B

|

/

|

A

|

w

A

= 2

→

10

|

B

|

/

|

A

|

full AT on CIFAR100

|

|

|

|

|

|

trained on. We find unexpectedly strong transfer performances for models that have both low clean and robust accuracy, only by putting more weight on the AEs.

̸

Loss balancing improves robustness transfer. In contrast to the previously explored setting, we observe the transfer setting to benefit from loss balancing. Recall equation 2 in section 3.2 in which w A and w B can be assigned different values to balance the loss when | A | = | B | . We show that the vanilla configuration w A = w B = 1 transfers robustness to downstream tasks poorly, that balancing the loss with w B = 1 , w A = | B | / | A | lacks transfer performance for small | B | and that weighting examples from A higher results in improved robustness transfer. We present results for all three weightings in figure 7. The figure is organized in three columns, all reporting robust accuracy. The first column reports the robust accuracy on subset A val, the second on subset B val and the third reports the robust accuracy on the downstream task. Here, we train on CIFAR-100 and transfer to CIFAR-10. The vanilla loss is indicated by circles and a solid line, the balanced loss w A = | B | / | A | by squares and a dotted line and the loss overemphasizing A by a plus and a dashed line.

First and foremost, note that the robustness transfer for the vanilla configuration is substantially worse than both alternatives (robust accuracy in top right). Transfer improves with use of loss balancing, e.g. for | A | = 10 , robust accuracy improves from 8% to 30% , but does not converge to the baseline AT performance (gray line). This is an unwanted side effect of equalizing the weight between A and B . When A is much smaller than B , less weight is assigned to the AEs constructed for A and robustness reduces. Note, this effect can also be seen on A val (top left in figure). Instead, we find it beneficial to overemphasize on the AEs (plus with dashed line). This configuration assigns w A = 2 | B | / | A | for | A | = 10 and increases the weight to w A = 10 | B | / | A | for | A | = 90 . This results in improved robust accuray on A val, but low robust and clean accuracy on B val. Interestingly, while the generalization to B val is low, robustness transfer to CIFAR-10 is very high. We use this loss weighting for all following task transfer experiments.

Robustness transfer from example subsets. Using the weighted loss, we focus in the following on S-ESAT on two source tasks: CIFAR-100 and ImageNet-200, and train on three downstream tasks. Similar results for S-CSAT and SVHN as additional downstream task can be found in the supplement. Figure 8 presents results for three settings: CIFAR-100 → CIFAR-10 and ImageNet-200 → Caltech-256 , Oxford-Flowers-102. The first and second row show robust and clean accuracy on the downstream task respectively. As before, we compare with a random (red) and a full AT baseline (gray line). Selecting A to contain the hardest examples first (highest entropy) is marked by a solid line; selecting easiest is marked by a dashed line.

In line with the improvements seen using the appropriate loss weighting, we see similarly fast recovery of baseline AT performance across all dataset. In fact, | A | containing only 30% of training data ( 15 k and 70 k) is sufficient to reach near baseline performance. On CIFAR-100 → CIFAR-10 and ImageNet-200 → Flowers-102 even slightly outperforming the same with a further increase in size. Similar to the non-transfer settings tested before, we also see similar interactions between

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

CIFAR-100

→

CIFAR-10

20

30

40

Rob. accuracy

CIFAR10

5k

10k

15k

20k

25k

30k

35k

40k

45k

# of adv. trained examples (

A

) on CIFAR100

65

70

75

80

Clean accuracy

CIFAR10

S-ESAT hardest first

S-ESAT easiest first

S-ESAT-random

full AT on CIFAR100

ImageNet-200

→

Caltech-256

30

40

Rob. accuracy

Caltech256

23k

47k

70k

94k 117k 140k 164k 187k 211k

# of adv. trained examples (

A

) on ImageNet200

60

65

70

Clean accuracy

Caltech256

S-ESAT hardest first

S-ESAT easiest first

S-ESAT-random

full AT on ImageNet200

ImageNet-200

→

Flowers-102

30

35

40

45

50

Rob. accuracy

Flowers102

23k

47k

70k

94k 117k 140k 164k 187k 211k

# of adv. trained examples (

A

) on ImageNet200

70

75

Clean accuracy

Flowers102

S-ESAT hardest first

S-ESAT easiest first

S-ESAT-random

full AT on ImageNet200

|

|

|

|

|

|

subset selection strategies. I.e. hardest examples (solid line) provide greater robustness transfer than easiest (dashed line) while a random baseline (red) achieves competitive performances. The latter consistently outperforming entropy selection on ImageNet-200 → Flowers-102, supporting our observation in section 4.2: with increasing dataset complexity, informed subset selection provides diminishing returns. Note that all robust accuracy increases proportionally correlate to an increase in clean accuracy as well. This is in stark contrast to the inverse relationship in previous settings. C.f. figure 5 and 6, for which clean accuracy decreases. This interaction during transfer is similar to what is reported in [8]: increased robustness of the source model results in increased clean accuracy on the target task (over a non-robust model). Intriguingly though, with appropriate weighting, the biggest robustness gains on the downstream task happen under fairly small A . This is a promising outlook for introducing robustness in the foundational setting [37], where models are generally trained on very large datasets, for which AT is multiple factors more expensive to train. Note that our results generalize to single-step attacks like fast gradient sign method (FGSM) [18, 38] as well. We provide evaluations in the supplement. While we consider the fixed-feature transfer only, recent work has shown this to be a reliable indicator for utility on full-network transfer [8, 39].

5 Conclusion

In this paper, we presented an analysis of how adversarial robustness transfers between classes, examples and tasks. To this end, we proposed the use of Subset Adversarial Training (SAT), which splits the training data into A and B and constructs AEs on A only. Trained on CIFAR-10, CIFAR100 and ImageNet-200, SAT revealed a surprising generalizability of robustness between subsets, which we found to be based on the following observations: (i) adv. robustness transfers among classes even if some or most classes have never been attacked during training and (ii) hard classes and examples provide better robustness transfer than easy ones. These observations remained largely valid in the transfer to downstream tasks like Flowers-102 and Caltech-256 for which we found that overemphasizing loss minimization of AEs in A provided fast convergence to baseline AT robust accuracies, even though transfer to B was severely reduced. Specifically, it appears that only few AEs ( A containing 30% of the training set) learn all of the robust features which generalize to downstream tasks. This finding could be particularly interesting for AT in the foundational setting, in which very large datasets render training computationally demanding.

More broadly, improving adversarial robustness remains one of the most important problems to 313 solve in deep learning, especially in high-stake decision making like autonomous driving or medical 314 diagnostics. Our findings shed new light onto the properties of adversarial training and may lead to 315 more efficient robustness transfer approaches which would allow easier deployment of robust models. 316 We provided an account on a broad variety of datasets and used models commonly evaluated in 317 related work. It needs to be seen whether our findings generalize to other threat models [40] as well. 318

319

320

321

References

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. ICLR , 2014.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 322 Towards deep learning models resistant to adversarial attacks. ICLR , 2018. 323

Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris 324 Tsipras, Ian J Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial 325 robustness. ICLR , 2019. 326

327

328

Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. ICML , 2019.

329

330

331

332

333

334

335

336

337

338

Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. NeurIPS , 2019.

Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. NeurIPS , 2020.

Ali Shafahi, Parsa Saadatpanah, Chen Zhu, Amin Ghiasi, Christoph Studer, David Jacobs, and Tom Goldstein. Adversarially robust transfer learning. ICLR , 2020.

Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? NeurIPS , 2020.

Yutaro Yamada and Mayu Otani. Does robustness on imagenet transfer to downstream tasks? CVPR , 2022.

Qi Tian, Kun Kuang, Kelu Jiang, Fei Wu, and Yisen Wang. Analysis and applications of 339 class-wise robustness in adversarial training. SIGKDD , 2021. 340

Zhikang Xia, Bin Chen, Tao Dai, and Shu-Tao Xia. Class aware robust training. ICASSP , 2021. 341

David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and 342 generalization. CVPR , 2019. 343

Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: 344 Direct input space margin maximization through adversarial training. ICLR , 2020. 345

Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving 346 adversarial robustness requires revisiting misclassified examples. ICLR , 2020. 347

Minseon Kim, Jihoon Tack, Jinwoo Shin, and Sung Ju Hwang. Entropy weighted adversarial 348 training. ICML Workshop on Adversarial Machine Learning , 2021. 349

Weizhe Hua, Yichi Zhang, Chuan Guo, Zhiru Zhang, and G Edward Suh. Bullettrain: Acceler350 ating robust neural network training via boundary example mining. NeurIPS , 2021. 351

Hadi M Dolatabadi, Sarah Erfani, and Christopher Leckie. l-inf-robustness and beyond: Un352 leashing efficient adversarial training. ECCV , 2022. 353

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver354 sarial examples. ICLR , 2015. 355

Yogesh Balaji, Tom Goldstein, and Judy Hoffman. Instance adaptive adversarial training: 356 Improved accuracy tradeoffs in neural nets. arXiv preprint , 2019. 357

Maximilian Kaufmann, Yiren Zhao, Ilia Shumailov, Robert Mullins, and Nicolas Papernot. 358 Efficient adversarial training with data pruning. arXiv preprint , 2022. 359

Sven Gowal, Po-Sen Huang, Aäron van den Oord, Timothy A. Mann, and Pushmeet Kohli. 360 Self-supervised adversarial robustness for the low-label, high-data regime. In ICLR , 2021. 361

Philipp Benz, Chaoning Zhang, Adil Karjauv, and In So Kweon. Robustness may be at odds 362 with fairness: An empirical study on class-wise accuracy. NeurIPS Workshop on Pre-registration 363 in Machine Learning , 2020. 364

Vedant Nanda, Samuel Dooley, Sahil Singla, Soheil Feizi, and John P Dickerson. Fairness 365 through robustness: Investigating robustness disparity in deep learning. ACM FAccT , 2021. 366

Saachi Jain, Hadi Salman, Alaa Khaddaj, Eric Wong, Sung Min Park, and Aleksander Madry. 367 A data-based perspective on transfer learning. CVPR , 2023. 368

Mohamed Omran and Bernt Schiele. Towards systematic robustness for scalable visual recogni369 tion. ICML Shift Happens Workshop , 2022. 370

Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of 371 example difficulty. NeurIPS , 2021. 372

Chirag Agarwal, Daniel D'souza, and Sara Hooker. Estimating example difficulty using variance 373 of gradients. CVPR , 2022. 374

Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. 375 ICML , 2020. 376

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image 377 recognition. CVPR , 2016. 378

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 379 2009. 380

Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural 381 adversarial examples. CVPR , 2021. 382

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale 383 hierarchical image database. CVPR , 2009. 384

Paul Gavrikov and Janis Keuper. Adversarial robustness through the lens of convolutional filters. 385 CVPR , 2022. 386

Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 387 Robustness may be at odds with accuracy. ICLR , 2019. 388

Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007. 389

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large 390 number of classes. ICVGIP , 2008. 391

Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von 392 Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the 393 opportunities and risks of foundation models. arXiv preprint , 2021. 394

Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial 395 training. ICLR , 2020. 396

Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? 397 CVPR , 2019. 398

Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. Perceptual adversarial robustness: Defense 399 against unseen threat models. ICLR , 2021. 400

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new approach called Subset Adversarial Training (SAT), which differs from traditional adversarial training methods that generate adversarial examples on the whole training set. Instead, SAT applies adversarial training on a subset of the training data. They studied two variants of subset adversarial training (CSAT and ESAT). They found that robust training in one class could generalize to other classes that were not adversarially trained, which is surprising. The also found that ESAT, where they adversarially trained on harder examples, gives surprising boost to downstream robust performance with much less data.\n\nThe paper also discusses the concept of loss balancing, which is used to counteract an imbalance between the adversarial subset and non-adversarial subset when the training split is not even. The authors found that loss balancing is important for the adversarial robustness transfer observed.\n\nIn conclusion, the paper presents a novel approach to adversarial training that could help us better understand the underlying mechanism of robust learning as well as having potential implication to more efficient adversarial training.\n\nSTRENGTHS:\n- The setting of experiment is interesting. It is surprising that adversarially training on a single class yields adversarial robustness to other classes. The originality of the experiment is strong.\n- The experiments has demonstrated possibility to decrease the cost of adversarial training.\n- The paper is also very clear with thorough experiments and analysis\n\nWEAKNESSES:\n- Even though the finding is interesting, I think the paper could do more in terms of understanding its implication to  robust generalization. What does robust generalization to other classes imply or show about the process of adversarial learning?\n- While the experiment demonstrate possibility of decreasing cost of adversarial training, it doesn’t demonstrate this in more challenging scenarios. I understand that the paper’s intention is to understand how adversarial generalization happen as opposed to achieving the best performance, but if it is the paper’s intention to gain further understanding of the adversarial training process, I am hoping for more analysis and comments about its implication to robust generalization.\n\nQUESTIONS:\nOne thing that I would like to see a little bit more of is analysis and discussion of the implication of the findings. For example, the authors have observed that the difficulty of a class is the main driver for its robust transferability to other classes. Could the author for example construct a synthetic 11th class on CIFAR-10 of varying difficulty that is completely unrelated to CIFAR-10, and see whether adversarial training on only this 11th class can give robust generalization to the original 10 classes. If the difficulty is increased, what is the extent of the robust generalization? I am a little bit unsure as to how the 11th class could be constructed, but this is an example of experiment where I would like to see the authors do by digging deeper into the possible implications.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitation of the potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper demonstrates an interesting observation: when we conduct adversarial training, we can only choose to generate adversarial examples on a subset of the training data, if this subset contains the hardest examples, then adversarial training on a subset can achieve competitive performance in robustness over the whole dataset. In addition, models trained in such a manner demonstrate good transferable feature.\n\nSTRENGTHS:\n1. The observation is surprising and interesting, which indicates the transferability of adversarial examples across different classes.\n\n2. The authors conduct comprehensive experiments on various datasets to validate the findings.\n\nWEAKNESSES:\n1. One major concern is the contribution, the proposed method neither improve the robust accuracy (or clean accuracy) nor improve the training efficiency (because SAT still uses PGD-7 to generate adversarial examples, which is inefficient).\n\n2. All the experiments are conducted on the $l_2$ bounded adversarial perturbations, more types of adversarial perturbations should be included, especially the $l_\\infty$ bounded ones which is popular for benchmarking. In addition, for CIFAR10, the adversarial budget $\\epsilon = 0.5$ is very small when considering the dimensionality of the input image. Experiments based on larger adversarial budgets should be included, e.g. $\\epsilon = 2$ for CIFAR10.\n\n3. Similar to the first point, the experiments does not demonstrate the advantages of the proposed method. In addition to adversarial training, is the method general and compatible to other popular robust learning method, such as TRADES? Is the observation the same in this context?\n\n4. It would be better if the authors can provide some intuition or explanations for the observations in this paper.\n\nQUESTIONS:\nIn addition to the concerns in the \"weakness\" part that I expect answers from the authors, I have minor questions as below:\n\n1. Why does the authors rank the difficulty of training instances based on the non-adv trained models, because the difficulty ranking can be quite different between adv and non-adv trained models.\n\n2. Figure 8 demonstrates the transferability can vary a bit given the number of instances in the set A. For practitioners, does the authors have some hint for how to choose |A|?\n\nBecause of the weakness part and the questions in this section, I cannot recommend acceptance based on the current manuscript. I will do a re-evaluation after the authors' feedback.\n\nLIMITATIONS:\nThe limitations and the broader societal impacts are not adequately discussed in the current manuscripts, although ethnicity should not be an issue for general research like this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
The authors proposed the use of Subset Adversarial Training (SAT), a technique that splits the training data into A and B and constructs AEs only for data in A. Using SAT, they demonstrate how adversarial robustness transfers between classes, examples, and tasks. The authors report several insights: 1) that they've observed robustness transfers by difficulty and to classes in B 2) hard examples to provide better robustness transfer, and 3) Generating AEs on part of the data (e.g, 50%) is enough to get the standard AT accuracy.

STRENGTHS:
1. The paper is relatively easy to follow
2. Existing empirical results seem sound

WEAKNESSES:
1. If I understand correctly, the experiments were done only on L2, even though the most common AT is done using L_inf. Can the authors present results using L_inf? 
2. I'm missing many details about the AT process, you need to be much more specific for reproduction purposes. which AT is the baseline? did you try other methods? which method did you use? Madry's/TRADES/Other? the paper needs to be much clearer. Many important implementation details are missing.
3. It's hard to validate the results without supplying code/models.
4. The novelty is marginal, due to the fact that much prior art exists on the transferability of AEs, revisiting hard examples, or pruning a part of the training examples throughout the training. The paper will benefit from a comparison of these methods to SAT, so we can see the differences in performance/resources requirements/etc.

QUESTIONS:
See Weaknesses section.

LIMITATIONS:
No discussion on limitations, I suggest the authors to add one.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the transferability of adversarial robustness for partially adversarially trained models. The authors examine 3 variants of subset adversarial training (SAT): Class SAT, where only samples from selected, difficult classes are adversarially perturbed in training; Example SAT, where only examples with the highest predictive entropy are perturbed; Source-task SAT, where SAT-trained, robust models are fine-tuned on downstream training sets and evaluated for downstream adversarial robustness. They further draw connections between SAT and loss balancing, thus proposing a method for sample-efficient, low-cost adversarial robustness transfer between datasets in foundational settings.  \n  \nThe authors report interesting insights from various experiments. From CSAT, it is noted that difficult classes transfer best; class-wise transfer gains are asymmetric; and robustness transfers between seemingly unrelated classes. From ESAT, the authors concur with previous findings that harder examples contribute more to training robust models; the gain in robust accuracy is more rapid than CSAT with respect to the size of subset A; hardness rankings suffer from a possible lack of sample diversity and its performance is matched by random rankings. From S-SAT, they find that SAT on the source dataset with only 30% of AEs can match the robustness transfer gains using normal adversarial training, on the downstream dataset; both clean and robust downstream accuracies are transferred and they are positively correlated under appropriate loss balancing.\n\nSTRENGTHS:\n1. **Data efficiency.** The proposed SAT greatly reduces the amount of data required for adversarial training, which is promising for resource limited or real-world settings. ESAT with only 50% of AEs matches normal AT performance; S-SAT with 30% of AEs matches AT (on the source dataset) as well.   \n2. **Loss balancing.** I appreciate the discussion on connections between the SAT formulation and loss balancing. I also recognise that both clean and robust accuracy transfer positively  from source to downstream tasks, under S-SAT with appropriate loss balancing, which is rare and difficult for adversarial training.  \n3. **Experimentation.** The experiments are relatively thorough (except that only ResNet-18 and ResNet-50 are SAT-trained) and many details (such as the inter-class robustness transfer statistics for CSAT, or the difficulty rankings of classes) are provided, which give rise to valuable insights.  \n4. **Presentation.** The presentation of this work is exemplary. It is well-organised, logically-coherent and persuasive.\n\nWEAKNESSES:\n### 1 Cost and efficiency  \n1.1 $\\hspace{5pt}$ SAT relies on meticulous pre-processing to discover hard classes and requires access to the per-epoch model weight snapshots of a normally trained classifier, to compute the difficulty metric of Equation 3. This shifts the partial cost of adversarial training to the pre-processing stage and can be costly for large models / datasets in foundational settings.   \n1.2 $\\hspace{5pt}$ More importantly, SAT relies on this non-robust classifier, presumably with identical architecture and training data as the target model for subset adversarial training. This means that all the pre-training and loss balancing procedures have to be repeated for every single new model-dataset combination, which might end up being more costly than normal adversarial training.    \n1.3 $\\hspace{5pt}$ One also notes that hardness ranking is important for the guaranteed performance of SAT, especially for CSAT and to a lesser extent for ESAT (where the authors acknowledge that it is \"possible to accidentally select poor performing subsets\", as per the easy rankings experiment).    \n### 2 Experimentation\n2.1 $\\hspace{5pt}$ SAT is only verified for ResNet-18 and ResNet-50, which is a non-negligible shortcoming, for the reasons described above.   \n2.2 $\\hspace{5pt}$ Does SAT hold for other convolutional and non-convolutional architectures of variable capacity?   \n2.3 $\\hspace{5pt}$ Is it not cost and time prohibitive to run SAT for more than 2 baseline models? Would this also be a barrier that impedes the practical adoption of SAT?  \n### 3 Scaling up\n3.1 $\\hspace{5pt}$ SAT experiments are notably performed on smaller datasets with fewer (or a subset of) classes. The computational complexity of SAT seems to scale non-negligibly with the number of classes (CSAT) and the size of the dataset (ESAT), which is not ideal on real-world datasets with fine-grained labels in foundational settings.   \n3.2 $\\hspace{5pt}$ As aforementioned, even considering the S-SAT setup (where one does not need to do SAT on every downstream dataset), it is very costly to add a new model to the SAT experiments because of the method's dependence on snapshots of a normally-trained, non-robust version of the same model, for hardness rankings and loss balancing.\n\nQUESTIONS:\n### Major concerns.  \n1. Could the authors address the concern about shifting the time/computational cost of adversarial training from the training phase to pre-training / pre-processing phases?  \n2. Furthermore, even if the non-robust, pre-trained weight snapshots of the target model are available off-the-shelf, using S-SAT still further requires one to adversarially pre-train from scratch a model on a large-scale dataset. Since the transferred robustness of S-SAT is comparable to that of normally AT'd models on the source dataset, could the authors elucidate what is the particular advantage of SAT (as opposed to simply fine-tuning with an off-the-shelf robust, AT'd model)?\n3. Could the authors suggest an efficient method of validating SAT for different dataset-model combinations (which does not require recomputation of task-specific hardness rankings and loss balancing settings); or alternatively, validate SAT on diverse convolutional and non-convolutional baselines?  \n### Minor comments.  \n4. What do the authors think about the connections between loss balancing and oversampling? How does CSAT perform for long-tailed or imbalanced datasets?    \n5. Have the authors considered the impact of homogeneous and heterogeneous data splitting methods (and resultant label or covariate shifts) on SAT models' clean and robust accuracy?\n\nLIMITATIONS:\nThe authors have adequately addressed the societal and ethical limitations of their work. This work strives to improve the foundational adversarial robustness of AI systems in practice; experimental and implementation details have been documented.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the transferability of adversarial robustness among different classes and different examples. Different from previous studies, authors split the training dataset into two groups and only apply adversarial training on one group while another one using clean training. Based on experiment results, authors obtain several interesting observations, including classes without adversarial training can still have some capacity to defense against adversarial attacks, hard classes and examples can provide better robustness transferability than easier ones, and only 50% of training data is sufficient to recover the performance with vanilla adversarial training method in terms of robustness, etc.\n\nSTRENGTHS:\n1. This paper explores the transferability of adversarial robustness from a new perspective and hence proposes a novel training mechanism to study.\n2. This paper conducts a series of experiments to investigate the robustness transferability. Based on experiments, some interesting observations are obtained, which may give some new insights for future works.\n\nWEAKNESSES:\n1. The motivation of this work is not quite clear. Although authors find that some classes still can obtain capacity to defense against adversarial attacks without adversarial training, data samples of these classes are available in the training dataset. Hence, applying adversarial training on all data samples of all classes directly can achieve much better robustness, comparing with the transferred robustness obtained in this paper. Hence, it's not clear why authors study this kind of transferability when all training data are available and which scenarios are suitable for the problem studied in this paper.\n2. Based on experimental results, authors claim that utilizing only half training data can achieve comparable robustness performance with vanilla adversarial training methods. However, related experiments only report robustness of different methods. Considering the trade-off between clean accuracy and robustness in adversarially trained models, it would be better if corresponding clean accuracy of each method can also be provided.\n\nQUESTIONS:\n1. Can authors discuss more about possible application scenarios or benefits of the study conducted in this paper? Although the transferability of adversarial robustness between different classes and examples under different training mechanisms is observed in this paper, it seems this observation cannot bring benefits to real applications. In practice, simply applying adversarial training on all data samples of all classes can achieve much better robustness. Hence, it‘s not clear what's the benefits of the study conduced in this paper in real applications, considering all data samples of all classes are used in training stage.\n2. Can authors provide some understanding about why the rank class split cannot consistently outperform the random class split on CIFAR10 validation set in terms of both clean accuracy and robustness as shown in Figure 4? Does this result indicate the hardness based split on class level cannot boost model overall performance?\n\nLIMITATIONS:\nQuestions about how to apply the proposed training mechanism and observations obtained from experiments in real applications to boost the robustness of models need to be discussed in detail.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}