On Adversarial Training without Perturbing all Examples
Authors: Max Losch
Mohamed Omran
David Stutz
Mario Fritz
Bernt Schiele
Venue: ICLR 2024 poster (2024)
Keywords: adversarial robustness
adversarial training
adversarial robust transfer
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose a subset adv. training experiment to investigate robustness interactions between classes, examples and tasks and find that few attacked examples generalise surprisingly well

Abstract: Adversarial training is the de-facto standard for improving robustness against adversarial examples. This usually involves a multi-step adversarial attack applied on each example during training. In this paper, we explore only constructing adversarial examples (AE) on a subset of the training examples. That is, we split the training set in two subsets $A$ and $B$, train models on both ($A\cup B$) but construct AEs only for examples in $A$. Starting with $A$ containing only a single class, we systematically increase the size of $A$ and consider splitting by class and by examples. We observe that: (i) adv. robustness transfers by difficulty and to classes in $B$ that have never been adv. attacked during training, (ii) we observe a tendency for hard examples to provide better robustness transfer than easy examples, yet find this tendency to diminish with increasing complexity of datasets (iii) generating AEs on only $50$% of training data is sufficient to recover most of the baseline AT performance even on ImageNet. We observe similar transfer properties across tasks, where generating AEs on only $30$% of data can recover baseline robustness on the target task. We evaluate our subset analysis on a wide variety of image datasets like CIFAR-10, CIFAR-100, ImageNet-200 and show transfer to SVHN, Oxford-Flowers-102 and Caltech-256. In contrast to conventional practice, our experiments indicate that the utility of computing AEs varies by class and examples and that weighting examples from $A$ higher than $B$ provides high transfer performance. Code is available at [http://github.com/mlosch/SAT](http://github.com/mlosch/SAT).

Published as a conference paper at ICLR 2024
ON ADVERSARIAL TRAINING WITHOUT PERTURBING ALL EXAMPLES
Max Losch', Mohamed Omran', David Stutz', Mario Fritz², Bernt Schiele'
1 Max Planck Institute for Informatics, Saarland Informatics Campus,
2 CISPA Helmholtz Center for Information Security, Saarbrücken (mlosch, mohomran, dstutz, schiele @mp -nf iniiiiimm mpg. de, fritzecispa.co de
ABSTRACT
Adversarial Training (AT) is the de-facto standard for improving robustness against adversarial examples. This usually involves a multi-step adversarial attack applied on each example during training. In this paper, we explore only constructing Adversarial Examples (AEs) on a subset of the training examples. That is, we split the training set in two subsets and B, train models on both (A U B) but construct AEs only for examples in A. Starting with A containing only single class, we systematically increase the size of and consider splitting by class and by examples. We observe that: (i) adv. robustness transfers by difficulty and to classes in B that have never been adv. attacked during training. (ii) we observe a tendency for hard examples to provide better robustness transfer than easy examples, yet find this tendency to diminish with increasing complexity of datasets (iii) generating AEs on only 50% of training data is sufficient recover most of the baseline AT performance even on ImageNet. We observe similar transfer properties across tasks, where generating ABs on only 30% of data can recover baseline robustness on the target task. We evaluate our subset analysis on a wide variety of image datasets like CIFAR-10, CIFAR- 100, ImageNet- Flowers- 102 and Caltech-256. In contrast to conventional practice, our experiments indicate that the utility of computing AEs varies by class and examples and that weighting examples from A higher than B provides high transfer performance. Code is available at http: / / gighhu......... com m1 oscoosschsssse SST.
and show transfer SVHN, Oxford-
INTRODUCTION
Imperceptible changes an image can change the output well-performing classification model dramatically. These so-called Adversarial Examples (AEs) have been the focus of a large body on deep learning vulnerabilities of works since its discovery (Szegedy et al., 2014). date, Adversarial Training (AT) (Madry et al., 2018; Carlini et al., 2019) and its variants (Zhang et al., 2019; Carmon et al., 2019; Wu et al., 2020) is the de-facto state-of-the-an in improving the robustness against AEs. Essentially, AT generates adversarial perturbations for all examples seen during training. While adversarial training is known to transfer robustness to downstream tasks (Shafahi et al., 2020; Salman al., 2020; Yamada & Otani, 2022) and that robustness is distributed unevenly across classes (Tian et al., 2021; Xia et al., 2021). common practice dictates that AT "sees" adversarial examples corresponding to the whole training data, including all classes and concepts therein. This is independent of whether only adversarial robustness is optimized or a trade-off between robustness and clean performance is desired (Stutz al., 2019). This also holds for variants that treat individual examples differently (Ding et al., 2020; Wang et al., 2020; Kim et al., 2021) or adaptively select subsets to attack during training to reduce computational overhead (Hua et al., 2021: Dolatabadi et al., 2022). In this common setup, it is thus difficult to ascertain if generic, transferable robustness is learned by the model, or adversarial robustness is learned separately for each class in the training set. In the worst case, model that can only learn to be robust on a class-specific basis would have to be presented with adversarially perturbed images for every class of interest. A more preferable outcome, on the other hand, would be model with systematic adversarial robustmess (Bahdanau et al., 2019; Omran & Schiele, 2022): i.c. a model that systematically generalizes to novel combinations of perturbations and classes that were separately encountered during training.

---

Published as a conference paper at ICLR 2024
Subeet Adv. Training (SAT)
Evelastion
Robust accuracy
Medil
stlack
90.0
Robosttres wes
OIAR
CIFAR 13
fransiers
D
trann
CEE
Dval
Bls
Hobust
A
Only
8
Iranafit
Spiil traiiing dkls
Carmepoor tang sulchtton
Robust socuracy (22
Figure 1: Adversarial robustness transfers among classes. Using Subset Adversarial Training (SAT), during which only a subset of all training examples (A) are attacked, we show that robust training even on single class provides robustness transfer all other, non adv. trained, classes (B). E.g., SAT for A=cat, we observe an robust accuracy of 37.8% on B. Noteworthy is the difference of transfer utility between classes. Le. A=car provides very little transfer to B (17.1%). We investigate this transfer among classes and provide new insights for robustness transfer to downstream tasks.
To shed light on this issue, we consider an analysis setup depicted in figure 1, we coin Subset Adversarial Training (SAT), where split the training data into two subsets A and B, train the model conventionally on the union (A U B), but generate AEs only on examples from A (indicated by the emoji). For example, we can split training data by class, with A = {car] or A {cart and B = A and investigate how adversarial robustness transfers. Surprisingly, we observe significant adversarial robustness on Beat at test time, the degree of which depends on the class(es) in A. Of course, A and B can be arbitrary partitions of the training data. For example, we could put only "difficult" examples in A during training. At test time, we evaluate overall adversarial robustness (since there is no natural split into A or B.at). These experiments reveal a rather complex interaction of adversarial robustness between classes and examples.
Our analysis provides a set of contributions revealing a surprising generalizability of robustness towards non-adv. trained classes and examples even under scarce training data setups. First, selecting subsets of whole classes, we find that SAT provides transfer of adversarial robustness to classes which have never been attacked during training. E.g. only generating adversarial examples for class car on 10, achieves a non-trivial robust accuracy of 17.1% on all remaining CIFAR- 10 classes (see figure 1, right). Secondly, we observe classes and examples that are hard to classify do generally provide better robustness transfer than easier ones. Lee. class cat achieves more than twice the robust accuracy on the remaining classes (37.8%) over class car (17.1%). Thirdly, SAT with 50% of training data sufficient recover the baseline performance with vanilla AT even on hard datasets like ImageNet. Fourthly, we observe similar transfer properties of SATed models to downstream tasks. In this setting, exposing the model only 30% of AEs during training, can recover baseline AT performance on the target task. Lastly, while we observe promising amounts of transfer, there is room for improvement. Our SAT approach can be viewed as an analytical tool that can foster the development of models that exhibit stronger systematic generalization to adversarial perturbations.
2 RELATED WORK
Since their discovery (Szegedy et al., 2014), robustness against adversarial examples has mainly been tackled using adversarial training (Goodfellow et al., 2015; Madry et al., 2018; Zhang et al., 2019). Among many others, prior work proposed adversarial training variants working with example- dependent threat models (Balaji et al., 2019; Ding et al., 2020; Wang et al., 2020; Kim et al., 2021), acknowledging that examples can have different difficulties. Some works also mine hard examples (Hua et al., 2021) or progressively prune a portion of the training examples throughout training (Dolatabadi et al., 2022; Kaufmann et al., 2022). However, all of these methods generally assume access to adversarial examples on the whole training set. That is, while individual examples can be dropped during training or are treated depending on difficulty, the model can see adversarial perturbations for these examples if deemed necessary. Adversarial training is also known to transfer robustness to downstream tasks (Salman et al., 2020: Yamada & Otani, 2022; Shafahi et al., 2020) and adversarially robust representations can be learned in a self-supervised fashion (Gowal et al., 2021). Here, a robust backbone often adapted to the target task by re-training a shallow classifier sometimes in an adversarial fashion. It is generally not studied whether seeing adversarial examples on the whole training set is required for good transfer. This is despite evidence that achieving adversarial robustness is easier for some classes/concepts than for others (Benz et al., 2020; Nanda et al., 2021; Xia et al., 2021; Tian et al., 2021), also for robustness transfer (Jain et al., 2023). Complementing these works, we consider only constructing adversarial examples on a pre-defined subset of the training set, not informed by the model or training procedure, and study how robustness transfers across examples and tasks. Our work is thus related to the broader problem of measuring
2

---

Published as a conference paper at ICLR 2024
systematic generalization and considering the effects of spurious correlations (Bahdanau et al., 2019; Ruis et al., 2020; Geirhos et al., 2020; Montero et al., 2021; Schott et al., 2022; Omran & Schicle, 2022).
BACKGROUND AND METHOD
3.1 ADVERSARIAL TRAINING (AT)
is a well known fact that conventional deep networks are vulnerable to small, often imperceptible, changes in the imput. As mitigation, AT is common approach extend the empirical risk minimiza- tion framework (Madry et al., 20.8). Let (I, E Drain be training set of example and label pairs and be trainable parameters, then AT is defined as:
min E (r.)))
max L(x + 8, y: [101b
(I)
where 5 is perturbation that maximizes the training loss L and thus training error. The idea being that, simultaneously minimizing the training loss, the loss also optimized be stable within small space around each training example | p I e, 2 1. We consider the L2 norm in our main paper and provide results for Lo in the appendix A.8. This additional inner maximization is solved by an iterative loop: conventionally consisting of more steps. In some settings (Goodfellow et al., 2015; Stutz et al., 2019; Zhang et al., 2019), the robust loss is combined with the corresponding loss on clean examples in a weighted fashion control the trade-off between adversarial robustness and clean performance.
3.2 AT WITHOUT PERTURBING ALL TRAINING EXAMPLES
Most proposed AT methodologies generate AEs on the whole training set. This being also valid for methods which adaptively select subsets (Hua et al., 2021; Dolatabadi et al., 2022) during training, for adaptive attack iterations (Zhang al., 2020), for adaptive example weightings (Huang et al., 2020; Zhang et al., 2021; Ge et al.. 2023) or more traditional AT in which only subset per batch adversarially attacked. These methods do not guarantee the exclusion of examples, that is, the model is likely to see AE for every example in the training set. From broader perspective, the necessity to generate AEs exhaustively for all classes appears unfortunate though. Idcally, we desire robust models to be scalable. i.c. transfer flexibly from few examples and across classes to unseen ones (Omran& & Schiele, 2022). We propose SAT to investigate to what extent AT provides this utility. To formalize, let A be training subset and B contain the complement: A Durins B Dirain Then SAT applies the inner maximization loop of AT on the subset A only; on B the conventional empirical risk is minimized:
W. WAA(s.)E max C(x + 0, w Wwg v:0) ||8||aS
min E (x.g)) 0
where 1 (x.v)EA is 1 when the training example is in A and 0 otherwise. wA and w define optional weights, which are by default both set to 1. Note that this is different from balancing robust and clean loss discussed in (Goodfellow et al., 2015; Stutz et al.. 2019: Zhang et al., 2019), where the model still encounters adversarial examples on the whole training set.
Loss balancing. The formulation in equation implies an imbalance between left and right loss when the training split is uneven ((A) # BB). To counteract, we assign different values to w and wB based on their subset size. E.g., equalize, we assign WB and WA = [Bl/AA), We will utilize this loss balancing to improve robustness for transfer learning in section 4.3.
3.3 TRAINING AND EVALUATION RECIPES
Consider the depiction of SAT in figure 1. Prior to training, the training set is split into A and B (left). For evaluation (middle), split the validation set into corresponding split of Aval and B possible. For Class-subset Adversarial Training (CSAT), this split aligns with the classes on the dataset: A and B are all training examples corresponding two disjoint sets of classes while A
3

---

Published as a conference paper at ICLR 2024
and Baal are the corresponding test examples of these classes. As experimenting with all possible splits of classes is infeasible, we motivate splits by class difficulty where we measure difficulty by the average entropy of predictions per class introduced as Hc in the next paragraph. In contrast, also split based on individual example difficulty. We provide empirical support for this approach in the experimental section 4. Additionally, example difficulty has been frequently linked to proximity between decision boundary and example (Baldock et al.. 2021; Ding et al., 2020; Kim et al., 2021; Hua et al., 2021; Agarwal al., 2022). The closer the example is the boundary, the harder is likely to classify. The hypothesis: hard examples provide larger contribution to training robust models, since they optimize for large margins (Ding et al., 2020; Wang et al.. 2020). We refer to this experiment as Example-subset Adversarial Training (ESAT). In contrast to CSAT, however, there is no natural split of the test examples into Aval and Bual such that we evaluate robustness on the whole test set (i.e., vaa)
can
As difficulty metric, we utilize entropy over softmax, which we empirically find to be as suitable as alternative metrics (discussed in the supplement, section A.2). Consider training set example x E Dmuin and a classifier f mapping from input space to logit space with N logits. Then the entropy of example is determined by (xx() and of whole class C' C Dirain is determined by Hc(f), the average over all examples in C:
N
H(f())) - Σ (o.((())) logo))))) Hc(l)
1 Σ H(f(x)), |C| #EC
where a denotes the softmax function. For our SAT setting, we rank examples prior to adversarial training. This requires classifier pretrained on Duain enabling the calculation of the entropy. To strictly separate the effects between entropy and AT, we determine the entropy using a non-robust classifier trained without AT. Similar to (Agarwal al., 2022), we aggregate the classifier states at multiple epochs during training and average the entropies. Let J2. classifier from multiple epochs during training, where M denotes the number of training epochs. Then the average entropy for an example is given by H(x) and for a class by Hc(f):
be snapshots of the
M
H(xx)
Σ H(L(())) Ho M
1
1 M
1
+HHole Ho(s)).
EXPERIMENTS
As aforementioned, common practice performs AT for the whole training set. In the following, we explore CSAT and ESAT, which splits the training set in two subsets A and B and only constructs AEs for A such that the model never sees AEs for B. We start with single-class CSAT examples of single class and increase the size of A (section 4.1) by utilizing the entropy ranking of classes Ho (equation 3). ESAT, which splits into example subsets is discussed in section 4.2. Both SAT variants reveal complex interactions between classes and examples while indicating that few AEs provide high transfer performance to downstream tasks when weighted appropriately (section 4.3).
contains only
Training and evaluation details. Since AT is prone overfitting (Rice al., 2020), it is common practice to stop training when robust accuracy on a hold-out set is at its peak. This typically happens after learning rate decay. We adopt this "early stopping" for all our experiments by following the methodology in (Rice et al., 2020) but utilize Auto Attack (AA) to evaluate robust accuracy. Throughout the course of the training, we evaluate AA after each learning rate decay on 10% of validation data Dval and perform a final evaluation with the model providing the highest robust accuracy. This evaluation is performed on the remaining 90% of validation data. This AA split is fixed throughout experiments to provide consistency. If not specified otherwise, we generate adversarial examples during training with PGD-7 within an I-2 epsilon ball of < 0.5 (all CIFAR variants) or 3.0 (all ImageNet variants) typical configurations found in related work. We train all models from scratch and use ResNet- 18 (He et al., 2016) for all CIFAR- 10 and CIFAR- 100 (Krizhevsky et al., 2009) experiments and ResNet-50 for all ImageNet-200 experiments. Here, ImageNet-20X corresponds to the ImageNet-A subset (Hendrycks et al., 2021) to render random baseline experiments tractable (to reduce training time). This ImageNet-200 200 dataset, contains 200 classes that retain the class variety and breadth of regular ImageNet, but remove classes that are similar each other (e.g. fine-grained dog types). We use all training and validation examples from
4

---

Published as a conference paper at ICLR 2024
Robust accuracy
plane
blrd
cat
deer
60g
frog
horse
ship
truck
plane
plane
car
car
Aled class
134.%
ATed class
bird
187.8%
deer
dog
34
55
28
53
36
10
08
36
freg 723.4%
frog
27
56
27
20
19
02
27
127.3%
horse
124.5%
ship
truck
truck
20
40
60
80
100
Full AT
Robust accurasy
Figure 2: CSAT on single CIFAR-IO class A (blue), we observe non-trivial transfer to the non-adv. trained classes B (green). Classes considered hard in CIFAR-10 (cat) offer best generalization (+37.8% gain on non-adv. trained), while easy classes offer the worst (car, +17.1% gained). Note that without AT, robust accuracy is close 0% for all classes (orange). Right: same as left, but robust accuracy is evaluated per class (along columns). Here, we observe an unexpected transfer property: hard classes provide better transfer to seemingly unrelated classes (cat truck: 53%) than related classes (car - truck: 35%). Additional results for e = 0.25 and 1.0 in the appendix, section A.3.
ImageNet (Deng et al., 2009) that correspond this subset classes. All training details can be found in the supplement, section A.1.
4.1 CLASS SUBSET SPLITS
We start by investigating the interactions between individual classes in A using CSAT on CIFAR- 10, followed by an investigation on increasing the number of classes. Single-class subsets (CSAT). We train all possible, single class CSAT runs (10) and evaluate robust accuracies the adv. trained class (A) and the non-adv. trained classes (B). The results are shown in figure 2, left. Each row represents a different training run. Note that the baseline robust accuracy, trained without AT achieves practically 0% (indicated by orange line). Most importantly, we observe non-trivial robustness gains for all classes that have never been attacked during training (B-sets). That is, irrespective of the chosen class, we gain at least 17.1 1% robust accuracy (A=car) on the remaining classes and can gain up to 37.8% robust accuracy when A=cat. These robustness gains are unexpectedly good, given many features of the non-adv. trained classes can be assumed to not be trained robustly. This is consistent across different values for e, as we show in the appendix section A.3.
To investigate this phenomenon further, we analyze robust gains for each individual class and present robust accuracies in the matrix in figure 2, right, where training runs listed in rows and robust ac- curacies per class are listed in columns. Blue cells denote the adv. trained class and green cells denote non-adv. trained classes. While we see some expected transfer properties, e.g. CSAT on car provides greater robust accuracy on the related class truck (35%) than unrelated animal classes bird, cat, deer, dog (between 5% and 16%). the reverse not straight-forward CSAT on bird provides 56% robust accuracy on the seemingly unrelated class truck, 20%-points more than CSAT on car. More generally, animal classes provide stronger robustness throughout all classes than inanimate classes. We observe, that these classes are also harder to classify and have a higher entropy Hc as shown in figure 3. This influence of class-entropy might also be utilized to augment the dataset, by adding high entropy samples. We provide a simple proof-of-concept in the appendix, section A.4, adding 1 1thh class to CIFAR-10.
0.40
W
Class errer
0.00
Entrayy Rc
2.04
0.13
0.0
0.00
sied
stip
Figure 3:
The hardest classes (blue) have the high- est entropy (green).
Many-class subsets (CSAT). To increase the number of classes in A while maintaining a minimal computational complexity, we utilize the average class entropy He proposed in equation 3 to inform us which ranking to select from. To improve clarity, we begin with a reduced set of experiments on CIFAR-10 10 before transitioning larger datasets. We utilize the observed correlation between
5

---

Published as a conference paper at ICLR 2024
Performance on CIFAR-10,al
Performance on Non-adv. trained classes (Bva)
Data splits
Rob. acouragy
accuracy
Foo 0ogg bvr plew
car 60g lind
dog
0og
bird
Doar
plano
Imng
Fmg
Fmg
Clesn sccuracy
Terro
70/000
8 8
2hpp
Tuck
Buck
stuck
Trock
Dar
A
4
6
8
6
10
Number of adv. trained classes (1:41)
Number of adv. trained classes (IAI)
Figure 4: Ranking CIFARIO classes by difficulty (using entropy as proxy), we perform CSAT with an increasing size of adv. trained classes in A. Class splits used for training (A and B) are stated on the left. The resulting robust and clean accuracies on the validation set is shown the right, separated into performance on Beal and all. Compared with a random baseline of random class ranking (red), we find the ranking by difficulty to have consistently better transfer non-adv. trained classes (B). Overall, this results in an improved robust accuracy on average over all classes.
class difficulty, average class entropy and robustness transfer He to rank classes and construct 4 adv. trained subsets. Ranked by class entropy Hc. we select 4 subsets showing in figure 4, left. As observed before, cat and dog are hardest and thus first chosen to be in subset A. Truck and car on the other hand are easiest and thus last. To gauge the utility of this ranking, we provide robust and clean accuracy comparison with a random baseline in figure 4, center and right. Iee., for each subset A we select 10 random subsets and report mean and std. deviation (red line and shaded area). Similar to the single-class setup, we observe subsets of the hardest classes to consistently outperform the random baseline (upper middle plot), up until subset size of |A| = 8, when it draws even. Also note that the robust accuracy on Bval is improved across all splits, thus providing support that harder classes as initially observed on animate vs inanimate classes- offer greater robustness transfer.
For our experiments on larger datasets like CIFAR-100 and ImageNet-200, we additionally evaluate third ranking strategy. Beside selecting at random and selecting the hardest first, we additionally compare with selecting the easiest (inverting the entropy ranking). We construct 9 subsets per type of ranking (instead of 4) and report robust accuracies for selecting the easiest classes as well. Results are presented in three columns in figure 5; one dataset per column. As before, we show robust accuracies on the tested dataset (upper row) and robust accuracies on Beal (lower row). For CIFAR-IO, we calculate mean and std. dev. over 10 runs, for CIFAR-100 over 5 runs and for ImageNet-2200 over 3 runs. Selecting hardest first (highest entropy) is marked as solid line and easiest first (lowest entropy) as a dashed line. First and foremost, we observe that irrespective of the dataset and the size of A, we see robustness transfer to B This transfer remains greatest with classes we consider hard, while easy classes provide the least. Nonetheless, we see diminishing returns of such an informed ranking when dataset complexity is increased. E.g. the gap between dashed and solid line on ImageNet-200 is small and random class selection is on-par with the best. The results are similar on CIFAR- 100, as shown in figure 5, middle). Based on these results, entropy ranking and selecting classes provides only slight improvements in general. Importantly though, we continue to the tendency of increased robustness transfer to Bea, which we will come back to in section 4.3.
4.2 EXAMPLE SUBSET SPLITS (ESAT)
Considering that splits along classes are inefficient in terms of reaching the full potential of adversarial robustness, we investigate ranking examples across the whole dataset (ESAT). We follow with the same setup as before but rank examples and not classes - by entropy H. Since it is not feasible to construct corresponding rankings on the validation set, we cannot gauge robustness transfer to B Instead, we will test transfer performance to downstream tasks in section 4.3. We consequently report robust accuracy and clean accuracy on the whole validation set in figure 6. Similar results for Loo are provided in the appendix, section A.8.
Firstly, note that the increase in robust accuracy is more rapid than with CSAT w.r.t. the size of A. AT only 50% of training data (25k examples CIFAR and 112k on ImageNet-2000 and the resulting average robust accuracy is very close to the baseline AT performance (gray line). Secondly, note that the gap between hard (solid line) and easy example selection (dashed line) has substantially

---

Published as a conference paper at ICLR 2024
CIFAR-10
CIFAR-100
ImageNet-200
Rob. accuracy
Rob. axcuracy
Reb sccure
On whode datcet
CSTT
CSAT
CSAT
fel
EST
CSAT
CSAT
Rob. accuraoy
Rob. accuracy
Rob. Sccura
On Bual
86
6
10
10
20
30
50
90
70
80
90 (1:4)
500
20
42
60
100
120
140
100
150
200
Number of aclv. trained classes (1A)
Number of adv. trained classes
Number of adv. trained classes (I:1)
Figure 5: Class-subset Adversarial Training (CSAT) produces non-trivial robustness classes that have never been attacked during training (Bva). Along the x-axes we increase the class subset size of A on which AEs are constructed and compare three different class-selection strategies: select hardest first (solid lines), select easiest first (dashed line) and select at random (red). On average, random selection performs well informed ranking (upper row), while the robustness transfer to B.al is best for the hardest classes (lower row). AT on a single class provides already much greater robust accuracies than without AT (orange).
widened. In practice, it is therefore possible to accidentally select poor performing subsets, although the chance appears to be low given the narrow variance of random rankings (red). To some extent, this observation supports the hypothesis that examples far from the decision border (the easiest to classify) provide the least contribution to robustness gains. This is also supported by the reverse gap in clean accuracy (bottom row in figure 6). That is, castess-fistt-sllction results in higher clean accuracies than hardest-first, while robust accuracies are much lower. In contrast, however, we observe random rankings (red) to achieve similar performances to hard rankings (solid lines) on all datasets and subset sizes. This is somewhat unexpected, especially on small sizes of A (e.g. 5k). Given the results, we conjecture that the proximity to the decision boundary plays a subordinate role to increasing robustness. Instead, it is plausible to assume that diversity in the training data has large impact on learning robust features, also indicated by (Gavrikov & Keuper, 2022). Note, when |A] is less than half the dataset, clean accuracy is improved over full AT (gray dot). In the appendix, section A.7, we show it is possible to trade-off these improvements for additional robust accuracy gains via TRADES Zhang et al. (2019).
CIFAR-10
CIFAR-100
ImageNet-200
Rob accuracy
sccuros
Pob Scca
On whoke dinaee
ESA l-anddrr dmy.
Clean accuracy
Clean accuracy
Clean accuracy
825
22A 44k 67%, B0G 11281344, 1.5x 275h 2006 224k 75.0
10t 5c
15k
206
25t
30k
354
424
43k
12k 5k
15h
22th
28c
30k
35k
40c
45h.
52k
adv.
examples (1,4)
Number
adv. trained examples (1:41)
Number adv. trained examples (lAl)
Figure 6: Example-subset Adversarial Training (ESAT) on CIFAR and ImageNet-200, provide quick convergence to a full AT baseline (gray line and dot) with increasing size of A. We report robust accuracy (upper row) and clean accuracy (lower row) and observe similar characteristies as with CSAT (figure 5). I.e., selecting the hardest examples first (solid line) provide higher rob. accuracy than easy ones (dashed line), although the gap substantially widens. Random example selection (red) provides competitive performance on average. Across all datasets, common clean accuracy decreases while robust accuracy increases (Tsipras et al., 2019). Loo results in appendix, figure 18.
4.3 TRANSFER TO DOWNSTREAM TASKS
Previous experiments on ESAT could not provide explicit robust accuracies on the non-ady. trained subset B since training and testing splits do not align naturally recall the evaluation recipe outlined in section 3.3. In order to test transfer performance regardless, we make use of the fixed-feature task transfer setting proposed in (Shafahi et al., 2020). The recipe just slightly changes: split the data
7

---

Published as a conference paper at ICLR 2024
On adv. trained CIFAR-100 classes (Aval)
On non-adv. trained
On non-adv. trained
CTFAR-100 classes (Bua))
downstream task CI
Rob. accurscy
Rob. accuracy
Rob, socuracy CIFARED
8
30
2-4 1M/
vorills
2- to (8/144
STAAT
Clean accurscy
0s
Clean accuracy
80
Claan socuracy
CIFARIO
80
-35
60
10
20
40
50
70
50
100
10
20
30
50
50
70
60
90
100
10
20 30
60
72
80
Numbor of adv. trained classes
(144)
Number of adv. traincd classes
(1:11
# of adv. trained classes
(1-41)
CIFAR100
Figure 7: Impact of cross-entropy weighting on robustness transfer. For subset AT, we test different weighting strategies for sets A and B given they are of unequal size. We observe that vanilla cross- entropy (circle) offers the worst robustness transfer to CIFAR-10 (right). The best transfer (plus) provided when loss weights are chosen such that training is overemphasized on A, indicated by dropping robust accuracies on B (compare left and center).
into A and as usual and perform SAT. Fix all features, replace the last classification layer with a 1-hidden layered classifier and finetune only the new classifier the target task. Importantly, neither training nor validation set for the target task are split. We consider CIFAR-100 and ImageNet-200 200 and transfer to CIFAR- 10, SVHN Caltech-256 (Griffin et al., 2007) and Flowers 102 (Nilsback & Zisserman, 2008). We call SAT trained for transfer Source-task Subset Adversarial Training (S-SAT), to emphasize that the subset training is performed on the source-task dataset.
In this section, we consider models that have "seen" " only a fraction of AEs on the source task and investigate the robustness transfer capabilities to tasks on which they have not explicitly adversarially trained on. We find unexpectedly strong transfer performances, boosted by putting more weight on the AEs.
Loss balancing improves robustness transfer. In contrast to the previously explored setting, we observe the transfer setting to benefit from loss balancing. Recall equation in section 3.2 in which WA and Wg can be assigned different values to balance the loss when |A| * [B]. We show that the vanilla configuration WA wB 1 transfers robustness to downstream tasks poorly, that balancing the loss with wB = 1, W. = |B1/|A] lacks transfer performance for small [B] and that weighting examples from higher results in improved robustness transfer. We present results for all three weightings in figure 7. The figure is organized in three columns, all reporting robust accuracy. The first column reports the robust accuracy on subset Aval, the second on subset Bval and the third reports the robust accuracy on the downstream task. Here, we train on CIFAR-100 and transfer to CIFAR-10 10. The vanilla loss is indicated by circles and a solid line, the balanced loss WA = (B1///|| by squares and dotted line and the loss overemphasizing A by plus and a dashed line.
First and foremost, note that the robustness transfer for the vanilla configuration is substantially worse than both alternatives (robust accuracy in top right). Transfer improves with use of loss balancing, e.g. for |A| = 10, robust accuracy improves from 8% to 30%, but does not converge to the baseline AT performance (gray line). This is an unwanted side effect of equalizing the weight between A and B. When A is much smaller than B, less weight is assigned to the AEs constructed for A and robustness reduces. Note, this effect can also be seen on Aval (top left in figure). Instead, we find it beneficial to overemphasize on the AEs (plus with dashed line). This configuration assigns WA 2/ (A] for | = 10 and increases the weight to W B0/B1/|A| for |A| = 90. This results in improved robust accuray on Aval, but low robust and clean accuracy on B Interestingly, while the generalization to B is low, robustness transfer to CIFAR-10 is very high. We use this loss weighting for all following task transfer experiments.
Robustness transfer from example subsets. Using the weighted loss, we focus in the following on S-ESAT on two source tasks: CIFAR- 100 and ImageNet-200, and train on three downstream tasks. Similar results for S- S-CSAT and SVHN as additional downstream task can be found in the supplement, sections A.5 and A.9. Figure 8 presents results for three settings: CIFAR- 100 CIFAR-10 and MifoordFFlowrr-1000 The first and second row show robust and clean
ImageNet-205 > Caltech-256, accuracy on the downstream task respectively. As before, we compare with random (red) and full
8

---

Published as a conference paper at ICLR 2024
CIFAR- 100 + CIFAR-I0 10
ImageNet-200 -> Caltech-256
ImageNet-2006 - Flowers- 102
Caltech2555
Rdk: asomee
fTowes102 B
PSST
Clen pccuuery CIFARLO
Clan accurary Caleen 6
Clean ancusasy F'owers222
22
236 12t 304
2541157
DR: 11B
#9 adlr. trainad socmmpes [141] on CIFAR10D
* of adc. trained ccample (I4l) on InageNas2200
trained ecamples (I:4) ImzgeNNe2200
Figure 8: Transfer from S-ESAT to three different downstream tasks. S-ESAT is trained dataset CIFAR-100 (left) and ImageNet-2000 (middle and right). We report robust (top row) and clean (bottom) accuracies for increasing size of A. Similar to our investigation on transfer from A B, we find that hard examples provide better robustness transfer than easy ones, but random selections (red) achieve competitive performances. Most importantly "seeing" only few AEs (here 30% of source data) recovers baseline AT performance (gray line). Loc results in appendix, figure 19.
source
AT baseline (gray line). Selecting A contain the hardest examples first (highest entropy) is marked by solid line; selecting easiest is marked by a dashed line. Similar results for L are provided in the appendix, section A.8.
In line with the improvements seen using appropriate loss weighting, we see similarly fast recovery of baseline AT performance across all dataset. In fact, |-A] containing only 30% of training data (15k and 70k) is sufficient to reach near baseline performance. On CIFAR-100 - CIFAR-10 10 and ImageNet- 200 - Flowers- 102 even slightly outperforming the same with a further increase in size. Similar to the non-transfer settings tested before, we see similar interactions between subset selection strategies. Lee. hardest examples (solid line) provide greater robustness transfer than easiest (dashed line) while random baseline (red) achieves competitive performances. The latter consistently outperforming entropy selection on ImageNet-200 + Flowers- 102, supporting our observation in section 4.2: with increasing dataset complexity, informed subset selection provides diminishing returns. Note that all robust accuracy gains correlate proportionally to an increase in clean accuracy as well. This is in stark contrast the inverse relationship in previous settings. C.f. figure 5 and 6, for which clean accuracy decreases. This interaction during transfer is similar to what is reported in (Salman et al., 2020): increased robustness of the source model results in increased clean accuracy on the target task (over a non-robust model). Intriguingly though, with appropriate weighting, the biggest robustness gains on the downstream task happen under fairly small A. This is a promising outlook for introducing robustness in the foundational setting (Bommasani et al., 2021), where models are generally trained on very large datasets, for which AT is multiple factors more expensive to train. Note that our results generalize to single-step attacks like fast gradient sign method (FGSM) (Goodfellow et al., 2015; Wong et al., 2020) as well. We provide evaluations in the supplement, section A. While we consider the fixed-fcature transfer only, recent work has shown this to be a reliable indicator for utility on full-network transfer (Salman et al., 2020; Kornblith et al., 2019).
CONCLUSION
We presented an analysis of how adversarial robustness transfers between classes, examples and tasks. To this end, we proposed the use of Subset Adversarial Training (SAT), which splits the training data into A and B and constructs AES on A only. Trained on CIFAR-10, CIFAR-100 and ImageNet-200, SAT revealed a surprising generalizability of robustness between subsets, which we found to be based on the following observations: (i) adv. robustness transfers among classes even if some or most classes have never been attacked during training and (ii) hard classes and examples provide better robustness transfer than easy ones. These observations remained largely valid in the transfer to downstream tasks like Flowers- 102 and Caltech-256 for which we found that overemphasizing loss minimization of AES in A provided fast convergence to baseline AT robust accuracies, even though transfer to B was severely reduced. Specifically, it appears that only few AEs (A containing 30% of the training set) learn all of the robust features which generalize downstream tasks. This finding could be particularly interesting for AT in the foundational setting, in which very large datasets make full AT challenging and 90% of the full robustness using 30% of data is preferred over robustness.

---

Published as a conference paper at ICLR 2024
Acknowledgements This work partially funded by the ELSA European Lighthouse Secure and Safe Al funded by the European Union under grant agreement No. 101070617. Views and opinions expressed are however those of the authors only and do not necessarily reflect those of the Huropean Union or European Commission. Neither the European Union nor the European Commission be held responsible for them.
REFERENCES
Chirag Agarwal, Daniel D' D'souza, and Sara Hooker. Estimating example difficulty using variance of gradients. CVPR, 2022.
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron C. Courville. Systematic generalization: What is required and can it learned? ICLR, 2019.
Yogesh Balaji, Tom Goldstein, and Judy Hoffman. Instance adaptive adversarial training: Improved accuracy tradcoffs in neural nets. arXiv preprint, 2019.
Robert Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. NeurIPS, 2021.
Philipp Benz, Chaoning Zhang, Adil Karjauv, and In So Kweon. Robustness may be at odds with fairness: An empirical study on class-win accuracy. NeurlPS Workshop on rreitegisttation in Machine Learning 2020.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint, 2021.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian J Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. ICLR, 2019.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy Liang. Unlabeled data improves adversarial robustness. NeurlPS, 2019.
Haw-Shiuan Chang, Erik Learned-Miller. and Andrew McCallum. Active bias: Training more accurate neural networks by emphasizing high variance samples. NeurIPS, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, 2009.
Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Direct input space margin maximization through adversarial training. ICLR, 2020.
Hadi M Dolatabadi, Sarah Erfani, and Christopher Leckie. l-inf-robustness and beyond: Unleashing efficient adversarial training. ECCV, 2022.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019. URL /eattps://gitnie com wwwdddyyl.a/robusstsess...
Paul Gavrikov and Janis Keuper. Adversarial robustness through the lens of convolutional filters. CVPR, 2022.
Yao Ge, Yun Li, Keji Han, Junyi Zhu, and Xianzhong Long. Advancing example exploitation can alleviate critical challenges in adversarial training. ICCV, 2023.
Robert Geirhos, Henn-eerikk Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2020.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015.
10

---

Published as a conference paper at ICLR 2024
Sven Gowal, Po-Sen Huang, Aäron van den Oord, Timothy A. Mann, and Pushmeet Kohli. Self- supervised adversarial robustness for the low-label, high-data regime. In ICLR, 2021.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021.
Weizhe Hua, Yichi Zhang, Chuan Guo, Zhiru Zhang, and G Edward Suh. Bullettrain: Accelerating robust neural network training via boundary example mining. NeurIPS, 2021.
Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk minimization. NeurIPS, 2020.
Saachi Jain, Hadi Salman, Alaa Khaddaj, Eric Wong. Sung Min Park, and Aleksander Madry. data-based perspective on transfer learning. CVPR, 2023.
Maximilian Kaufmann, Yiren Zhao, Ilia Shumailov, Robert Mullins, and Nicolas Papernot. Efficient adversarial training with data pruning. arXiv preprint, 2022.
Minseon Kim, Jihoon Tack, Jinwoo Shin, and Sung Ju Hwang. Entropy weighted adversarial training. ICML Workshop on Adversarial Machine Learning, 2021.
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? CVPR, 2019.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ICLR, 2018.
Milton Llera Montero, Casimir JH Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey Bowers. The role of disentanglement in generalisation. ICLR, 2021.
Vedant Nanda, Samuel Dooley, Sahil Singla, Soheil Feizi, and John P Dickerson. Fairness through robustness: Investigating robustness disparity in deep learning. ACM FAceT, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification large number of classes. ICVGIP, 2008.
Mohamed Omran and Bernt Schiele. Towards systematic robustness for scalable visual recognition. ICMI. Shift Happens Workshop. 2022.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting adversarially robust deep learning. ICML, 2020.
Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M. Lake. A benchmark for systematic generalization in grounded language understanding. NeurlPS, 2020.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversari- ally robust imagenet models transfer better? NeurIPS, 2020.
Lukas Schott, Julius von Küigelgen, Frederik Träuble, Peter V. Gehler, Chris Russell, Mathhia Bethge, Bernhard Schölkopf, Francesco Locatello, and Wieland Brendel. Visual representation learning does not generalize strongly within the same domain. ICLR, 2022.
Ali Shafahi, Parsa Saadatpanah, Chen Zhu, Amin Ghiasi, Christoph Studer, David Jacobs, and Tom Goldstein. Adversarially robust transfer learning. ICLR, 2020.
11

---

Published as a conference paper at ICLR 2024
David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and generaliza- tion. CVPR, 2019.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. JCLR, 2014.
Qi Tian, Kun Kuang, Kelu Jiang, Fei Wu, and Yisen Wang. Analysis and applications of class wise robustness in adversarial training. SIGKDD, 2021.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. ICLR, 2019.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. ICLR, 2020.
Eric Wong, Leslie Rice, and Zico Kolter. Fast is better than free: Revisiting adversarial training. ICLR, 2020.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust general- ization. NeurlPS, 2020.
Zhikang Xia, Bin Chen, Tao Dai, and Shu-Tao Xia. Class aware robust training. ICASSP, 2021.
Yutaro Yamada and Mayu Otani. Does robustness on imagenet transfer to downstream tasks? CVPR, 2022.
Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. ICML, 2019.
Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankan- halli. Attacks which do not kill training make adversarial learning stronger. ICML, 2020.
Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-aware instance-reweighted adversarial training. ICLR, 2021.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 FULL TRAINING DETAILS
For all training setups listed in table 2, we train our models from scratch using SGD with momentum of 0.9. Dataset sizes are listed in table 1. All are data augmented based on the definitions in (Engstrom al., 2019). The sequence of transformations are listed in figure 9. Left, for CIFAR-10, 10, CIFAR-100 and SVHN. Right, for ImageNet-200, Caltech-256 and Flowers- 102.
Adversarial training for the L2 norm is performed with 7 steps of projected gradient descent (PGD-7) within an 62 = 0.5 for CIFAR and SVHN and e = 3.0 for ImageNet-200. Caltech-256 and Plowers-102. For each step, we use step size of 0.1 and 0.5 respectively. For the L norm, we use 10 steps of PGD) during ESAT and a step size of 1/255 to avoid catastrophic forgetting when |A| is small (see section A. For S-ESAT, we find it to be sufficient to perform PGD-7 with step-size of 2/255. This is likely mitigated by the weighted loss. For all datasets, we constrain the maximum perturbation norm to Ex. = $/255. For all experiments, including L2 and Loo. we maximize the default cross-entropp loss. For vanilla AT, all examples in a training batch are attacked. For all SAT variants, we randomly sample training examples construct a training batch and attack only examples that are contained in A.
Class order. In the following, we list the order of classes ranked by entropy Hc (equation 3). CIFAR-10 can be derived from figure 4 in the main paper. In figures 10 and 11, we provide the list for CIFAR-100 and ImageNet-200. On CIFAR-100, the first and thus hardest classes consist mostly of animate categories like otter, rabbit and crocodile. The easiest the other hand are inanimate categories, specifically vehicle related classes, e.g. road, motorcycle or pickup-truck. Overall, the pnimate-inanimate order is similar to CIFAR-10. 10. On ImageNet-200 200, we observe very different order. Inanimate categories like spatula, drumstick or umbrella are among the hardest, while animate classes like monarch (butterfly), flamingo fammnno or lorikeet are among the easiest. Named hard classes may be difficult to distinguish due to a frequent presence of people in the images.
A.2 ALTERNATIVE RANKINGS
For simplicity, we focused our experiments on using entropy proxy to measure example and class difficulty (c.f. equation 3). Multiple such difficulty metrics have been proposed literature (Chang et al., 2017; Hua et al., 2021; Baldock et al., 2021; Agarwal et al., 2022), of which we select few from recent literature to compare to: signed variance (Srr)) (Hua et al., 2021) 1) and variance of gradients (VoG) (Agarwal et al., 2022). We want to highlight, that they perform very similar to entropy metric when utilized in our SAT framework. Figure 12 compares these two metrics with our used entropy metric using ESAT CIFAR-100. Overall, VoG has a slight edge over Var and Entropy, yet the differences remain small. On 5k attacked examples, Entropy (yellow line) achieves 21.0%, VoG (red line) 21.9% and Sar (purple line) 22.3% robust accuracy. On 25k attacked examples, Entropy achieves 38.00 38.0%. VoG 38.8% and Sar 38.1% 38.1%. While some improvements over our simple Entropy metric are possible, no proposed metric has a clear edge over the other.
pad pixels
random crop to 32x32
random crop to 224x224
random horizontal flip
random horizontal flip
color jitter [0.25, 0.25, 0. 0.25]
color jitter [0.1, 0.1, 0.1]
random rotation within +/- deg. random rotation within +/- 2 deg.
Figure 9: Input transformation for CIFAR and SVHN datasets (left) and ImageNet-200, Caltech-256 and Flowers- 102 (right) during training. During testing, no transformations are applied to CIFAR and SVHN. The remaining datasets are resized such that the shortest side equals 256, after which they are center cropped 224.
13

---

Published as a conference paper at ICLR 2024
otter, lizard, seal, rabbit, mouse, crocodile, lobster, shrew, shark, woman, beaver, bowl, turtle, squirrel, possum, snail, girl, kangaroo, ray, forest, caterpillar, man, baby, dinosaur, lamp, elephant, couch, poy, porcupine, snake, butterfly, leopard, crab, table, mushroem, dolphin, willcw_tree, beetle, spider, clock, fox, sweet_pepper, bee, house, raccoon, tulip, bridge, bus, rose, tank, whale, train, worm, lion, poppY, trout, bed, plate, can, telephone, tiger, hamster, aquarium_fish maple_ tree, orchid, pear, mountain, tractor, oak_tree, rocket, skunk, cockroach, television, cup, sea, cloud, lawn_mower, mower, castle, bottle, Salm_tres keyboard, apple, plain, accup_ttuck, bicycle, orange, chair wardrobe, motorcycle, road
Figure 10: CIFAR-100 100 classes ranked by decreasing entropy Hc. Animal classes are hardest, inanimate classes easiest.
spatula, shovel, syringe, drumstick, hand blower, lighter, nail, maraca barrow, unbrella, bow, quill, iron, stethoscope, soap dispenser, dumbbe mask, reel, toaster, ant, walking stick, envelope, candle, sleeping pag sandai, tricycle, cowboy boot, cradle, breastplate, bubble, banjo, chest, cliff, wine bottle, fountain, crayfish, doormat, Chihuahua, chain, apros, kimono, cockroach, accordion, sewing machine, ocarina, revelver, torch, piggy bank, goblet, studio couch, wreck, hermit crab, grand piano, beaker, snail, marimba, sundial, mantis, vulture, sea lion, flagpcle, washer, acoustic guitar, mongoose, grasshopper, Christmas stocking, bikini, cort, balance bean, fox squirrel, American alligator, academic gown, feather boa, suspension bridge, stingray, acorn, common iguana, forklift, parachute, mushroom, hotdog, American black bear, beacon, garbage truck, collo, pug, bec, banana, volcano, baboon, centipede, golfcart, marmot, limousine, African chameleon, leafhopper, canoc, wood rabbit, agama, starfish, lynx, German shepherd, capuchin, balloon, goose, submarire, golden retriever, mitten, jeep, hummingbird, armadillo, weevil, porcupine, puck, snowplow, barn, fly, tarantula, Rottwei ler, pool table, red fox, harvestman, pretzel, ballplayer, American egret, puffer, ladybug, pelican, obelisk, bald eagle, go-kart, bell pepper, castle, snommobile, junco, lemon, spider web, lion, water tower, basketball, guacamole, toucan, tank, jellyfish, viaduct,
robin, ambulance, broccoli, flatworm, pomegranate, bison, sea anemone, jay, rugby ball, organ, drake, cheeseburger, mosque, kcala, garter snake, African elephant, lycaenid, cystercatcher, box turtle, cabbage butterfly, locomotive, goldfinch, manhole cover, rapeseed, flamingo, yellow lady's slipper, menarch
school bus, lorikest,
Figure 11: ImageNet-200 classes ranked by decreasing entropy He. In contrast to the order CIFAR-10 and CIFAR-100, animate classes are generally not the most frequent among the hardest. Instead its mostly inanimate objects.
Rob. accuracy
10
Entropy
30
VoG
full AT
10k
15k
30k 35k
50k
Number of adv. trained instances (IAl)
Figure 12: Various hardness metrics result in similar rob. aces. for ESAT on CIFAR-100.
14

---

Published as a conference paper at ICLR 2024
A.3 SINGLE CLASS CSAT WITH DIFFERENT
Supplementary to our CIFAR-10 single class results in figure 2. we provide additional results for smaller and larger L2 e figure 13. Overall, we continue to observe non-trivial robustness transfer to B, irrespective of e. Interesting though, the robustness transfer notably increases for smaller e = 0.25 (left), but also notably decreases for larger 1.0 (right).
Polk
Poburt
(a) c 0.25
(b) 0.5
(c) 1.0
Figure 13: For all e, we observe non trivial robustness transfer to B. yet transfer diminishes with increasing e, For smaller though, the transfer is notably stronger. For e 1.0 we use a step-size of 0.2 and 0.1 otherwise.
A.4 DATASET AUGMENTATION CIFAR-10 +1
As briefly mentioned in the main paper, we can utilize the class- to robustness transfer relationship to augment a dataset by samples with high entropy. Then, SAT is performed only on these new samples to establish strong robustness on the original dataset. As proof-of-concey we synthesized set of 1 1thh classes from CIFAR- super-classes (see figure 15) - of which there 20 and perform SAT this 11th class to evaluate the robust accuracy gains the original CIFAR-10 10 classes. Results are reported in figure 14. We continue to observe a correspondence between average entropy He and the robustness transfer of class. As in the main paper, evaluate He on non-adv. trained models. Note that, while the best performing setup (A = {rodenr)) with rob. acc of 33.3% does not improve upon the best in the main paper (A {car), with a rob. acc > 37.8%), the number of examples in A is only A|| 2500, thus less than 5% of training data. This provides an indication that such a dataset augmentation is possible.
A.5 FULL RESULTS FOR CSAT
Results for CSAT can be plotted for three different validation subsets: Aval, Beal and on the whole dataset D For clarity, we only showed robust accuracies on Dyal and Baa in the main paper in figure 5. Here, we provide all results. That is, in figure 16, we show robust accuracies in the upper split and clean accuracies in the lower split for all 3 subsets.
Dataset
Classes
Size (Train/Test)
CIFAR- (Krizhevsky et al., 2009)
10
50 000 / 100
CIFAR- 100 (Krizhevsky et al., 2009)
100
50 000
ImageNet-200 200 (Deng al., 2009; Hendrycks et al., 2021)
200
259 906 / 10 000
Caltech- 256 (Griffin et al., 2007)
257
245 6122
Flowers-102 (Nilshack & Zisserman, 2008)
102
1020 1020
SVHN (Netzer et al., 2011)
10
257 26 032
Table I: Number of training and validation examples per dataset used. ImageNet-200 uses examples from (Deng al., 2009) only for classes defined in (Hendrycks et al., 2021)
15

---

Published as a conference paper at ICLR 2024
0.20
$1.5
301
Robust accuracc on
(CIFAR-1D 10 classes
Entropy H
0.125
0.100
20.0
0.075
H (CIFAR-10 Classes)
( (Mmus--ne-rlass
13.0
0.005
aquatic mammal
fish
flower
container
fruit
device
furniture
insect
carnivoree
building
scene
large mummal
smal small I-mammal
crustacean
human
reptile
rodent
tree
vehicle
utliity vehidle
Figure 14: CSAT CIFARIO plus additional class, synthesized from CIFAR-100. AA consists only this additional 11th class, to test how much robust accuracy can be gained on the original CIFAR-10 10 classes (orange). We observe consistent link with the entropy of this 11th class (green) with respect to the entropy of the CIFAR-10 classes (blue). The higher He, the higher the robust accuracy.
A.6 CLEAN ACCURACY INDEPENDENT CSAT RESULTS
So far, we presented robust accuracies as well clean accuracies. To highlight, that the hardest examples provide strongest robustness transfer which is independent of clean accuracy gains, we show figure 17 an additional robustness metric: Attack defense rate. We define Attack defense rate as the robust accuracy select examples that are accurately predicted and perform AA. The fraction robust examples are reported. We observe, that the relationship between hardest, easiest and random examples retained w.r.t. figure 16 on dataset. We thus conclude that the reported gains are independent of clean accuracy.
purely accurately classified examples. That is, given a dataset, we
A.7 ROBUSTNESS ACCURACY TRADE-OFF WITH TRADES
In the following, we investigate the impact of applying TRADES (Zhang et al., 2019) adversarial training with tuneable trade-off between clean and robust accuracy. This is of particular interest given
Conventional setting
Dataset
Architecture
Epochs
Batchsize
Ir-decays
1.2 decay
CIFAR- R-00
PreActResNet-18
200
128
0.1
100, 150
5 10
CIFAR-108 100
PreActResNet-18
200
128
0.1
100, 150
5 10
ImageNet-200
ResNet-50
150
256
0.1
50, 100
1. 10-4
Transfer setting
Dataset
Architecture
Epochs
Batchsize
Ir
Ir-decays
Le decay
CIFAR-I0
PreActResNet-188 + [512,10]
40
128
0.1
20,30
10-1
SVHN
PreActtResNet-18 + [512,10]
40
128
0.1
20, 30
5 10-1
Caltech-258
ResNet-50 + [2048.257]
100
128
0.1
50, 75
10-1
Flowers- 102
ResNet-50 [2048,102]
100
102
0.1
50, 75
1.10
Table Training settings for all used datasets for the conventional (upper rows) and the transfer
setting (lower rows). In the transfer setting, the last classifier layer is replaced with two linear layers of size K K and K x N, abbreviated as [K, NJ. K defines the number feature channels and N the number of classes.
16

---

Published as a conference paper at ICLR 2024
aguetic-mammal: l'beaver', 'dolphin', 'otter', 'seal', 'whale'] figh: M'aparruu_ffaa 'flatfish', 'roy', 'shark', "srout"]
flower: (Morchid', 'poppy', 'rose', 'su:fflwer', "tulip"]
container: l'bettla', 'bowl', 'can', 'cup', 'plase'] l'apple', 'mushroom', "orange', "pear", "aweet_pepper (fclock', 'keysoard', 'lamp', 'Lelephone', 'television") furniture: ['bed', 'chair', 'couzh', 'table', 'wardrobe') insect: ('bee', 'bectle', 'outterfly', "catospilaar', 'cockreach") varge-oarntvore: ("bea"', " 'leopard', (Hon', 'tiger', (wolf(] building: ('bridge', 'castle', 'house', "road", 'skyscraper') scene: ('cloud', fooret'' 'mountain', 'plain', 'sea')
fruit:
large manmal- [' l'camel',, 'cattle', 'chimpanzee', 'elephant', "kangaroo") small nemmal; ['Eox', 'percupine', 'possum', 'raccoen', 'skunk') crustacesn: [' l'crab', 'labster', 'szzil', 'spider', 'worn!] humen: ('baby', 'any", 'girl',
'woman'l
reptile: l'orocodile", 'cinosaur", 'lizard', 'anake", "turtle"] rodent: ('hamstar', "mouse', 'rabbit', 'shrev', 'squirsal') l'maple_tree 'oak_ ppalm_tree" 'pine_tree', 'wil 'ufllow_tree'" vehicle: l'bicycle', "bus", 'motozcycle', pickup_tzuek', "tzain"] weility-vehicles ['lemm_mower', 'rocket', 'stresscar', 'taak', 'tractor']
Figure 15: 20 CIFAR-100 Superclasses for reference.
that clean accuracy for small |A] is higher than with full AT (e.g. see figure enabling improved robust accuracy.
clean acc. % on CIFAR-10
robust acc.
= {cat] w/o TRADES TRADES
on
on
on CIEAR-10
49.6
37.8
39.0
92.4
41.5
32.0
33.0
85.9
70.2
42.1
44.9
12.0
81.0
78.2
42.9
46.4
24.0
81.6
80.1
41.8
45.6
Table 3: TRADES (Zhang et al., 2019), applied to CSAT improves robustness transfer from single class cat to B over baseline CSAT training. That is, we gain 5.1%-points to achieve robust accuracy on B of 42.9%.
TRADES on ESAT
|A|
5k
10k
15k
20k
25k
30k
35k
40k
45k
full AT
12.0
11.25
10.5
9.75
9.0
8.25
7.5
6.75
6.0
6.0
yes
Roh, acc
57.2
62.1
65.7
66.1
67.7
67.6
68.2
66.4
69.0
69.0
Clean acc
87.0
87.4
84.2
85.6
84.3
87.0
84,0
88.0
88.4
88.6
no
Rob, acc
52.1
59.1
63.3
64.9
66.3
66.8
67.7
68.7
68.5
69.0
Clean acc
91.9
91.2
90.5
89.7
89.4
89.4
89.2
89.4
89.5
89.2
Table ESAT on CIFAR-10 combined with TRADES (Zhang et al., 2019). Trading off clean for robust accuracy, we gain > 66.0% robust accuracy earlier at 20k samples in contrast to 25k for vanilla ESAT. However, our choice decreases clean accuracy below full AT.
Zhang et al. (2019) proposed loss with principled trade-off capability between clean and robust accuracy for AT. We observe that clean accuracy is higher than with full AT when |A] is small, i.e. less than 50% of examples. We investigate what extent robust accuracy can be improved. First, we conduct hyperparameter sweep for TRADES 3 parameter which controls the trade-off. As baseline we choose the best performing CSAT configuration on CIFAR-10 with L2e 0.5, that is A = [cat). Results are presented in table 3. Note that expected clean accuracy decreases with increasing while robust accuracy increases. Although Zhang al. (2019) recommend 3 to be less than 10.0, we find 12.0 to provide best robustness transfer from A to B. This configuration achieves 42.9% (+5.1) robust accuracy on B while also increasing rob. acc. on A to 78.2% +28.6), with the expected decrease in overall clean accuracy (81. (3100((-100)))
In a second experiment, we apply TRADES to ESAT on CIFAR-10. Given the 3 = 6.0 recommen- dation in (Zhang al., 2019) and our B 12.0 observation for single class CSAT, use linear interpolation between these two values for general ESAT. That is, when |A| is small, we trade off some of the gained clean accuracy for increased robustness with large 8 12, but decrease 3 6
17

---

Published as a conference paper at ICLR 2024
CIFAR-10 10
CIFAR-100
ImageNett220
On Whik diaset
Rob. 30c/ocy
Rob. 300002 D
Rob. socu Rannee
Rob acouracy
Rdb. scouacy
Oa Avai
Fandet frrt
CSA mandor
1: AT
CDwll aurdest trat
Grandor
ClAl Harter
CSei rander
fel
CSAT wadien Fine
s1d des
CSAT uudian
AT
CSAT sadee Bust
uod dev
ro
Rob. accuracy
acarssy
Rob. accurscy
On Hul
8
30
50
109
60
120
180
Number of adv. trained classes ([,4)
Number of adv. trained classes ([AA))
Number of adv. trained classes ([4[)
Robust accuracies
Clean accuracies
Ob whole danteet
Clain accuracy
Clann accuracy
Claan accurary
15
62.5
80.0
77.5
Clean accurady
100
sccuracy
Cican sccuracy
Ou Aval
Clean sccuracy
sccurocy
Clean sccuracy
On Blaal
8
500
100 120
160 190
classes ([,1)
classes (1A41)
adv. classes (1:4))
Figure 16: Full robust (upper split) and clean accuracies (lower split) from CSAT experiments, plotted for the whole dataset, Aval and Bynt. Selecting the hardest classes first (solid lines), clean accuracies and robust accuracies on Ayal steadily increase, while selecting the easiest in contrast (dotted lines) results in steady decline. This provides additional support that entropy as metric provides a useful account difficulty, since easy classes can achieve higher accuracy. Furthermore, we note that clean accuracy the whole dataset is increasing mostly stable, while on other datasets it is steadily decreasing. This should investigated further.
with increasing size of [A]. We compare ESAT-hardest first with and without TRADES in table We observe quicker robust accuracy convergence to the baseline. 66% robust accuracy is achieved 20k samples, instead of 25k. However, our choice of 3 also induces drop in clean accuracy below full AT clean accuracy (85.6% clean accuracy at 20k vs 89.7% at 25k vs 89.2% for full AT).
A.8 Loo RESULTS
While the main paper focuses the L2 norm, also provide corresponding evaluation for Lec-
ESAT. We evaluate ESAT on CIFAR-10, CIFAR-100 and ImageNet- in figure 18 and S-ESAT CIFAR-100 100 - CIFAR-10 10 and ImageNet-200 figure 18, we find characteristics similar to the Lg results in figure 6. That is, few examples contribute large amount of overall robustness to the model and robustness increases quickly with size of A.
[Caltech-256. 1l0w For ESAT in
18

---

Published as a conference paper at ICLR 2024
CIFAR-10 10
CIFAR-100
ImageNet-200
Attack deferee rate Attack deferse rate Attack deferse rote
Attock deforse rabe Attack ceferce rote:
On whole diasect
Attack delane rate Attack defienaa rato Attack deffece
0.6
03.0
044
0.4
0.3
02
Oa Avai
114
Tell Al
CD4ll Jundest teat
h A1
LS4) nurbor fre.
CSAI warcem
1x1
CSAT
usd
no AT
CSAT uudea
a.70
CSAT -aslet Sec
dee uts cou.
05. AT
E.2
On Hual
6.50
0.50
0.50
0:25
0.35
0.28
0.00
20
30
50
70
80
20
60
100 120
140 100 180 200
Number of adv. trained classes (1-4)
Number of adv. trained classes (IAA))
Number of acv. trained classes ([AA))
t Attack defense rate success rate) 1
Figure 17: As complement to the CSAT results in figure 16, we additionally show the defense rates (for what fraction of accurately classified validation examples does Auto-Attack fail to find adv. examples). As the defense rate shows robustness independent of clean accuracy, we can test whether the robustness gains are in fact based on the improved robustness transfer from hard classes. We see this being the case across all datasets.
Noteworthy though, is the low robust accuracy when A contains 10% of data, dropping to 0% on CIFAR-I0 and CIFAR-100. We have found this to be similar to catastrophic overfitting described for single-step AT (Wong et al., 2020). The effect can be mitigated with more PGD iterations (10) and a smaller step size (1/255), but is not resolved entirely. That is, note that the drop only occurs for the hardest examples, but not for the easiest or a random selection.
S-ESAT. For S-ESAT, show the same transfer configurations in the main paper. In figure 19, we observe similar characteristics as for T.2. That is, fast convergence to baseline AT performance and hard examples providing better robust transfer than easy examples. Also similar to L2, is the observation that robustness transfer from ImageNet-2000 to Flowers- 102 is best when using a random selection of examples.
CIFAR-10 10
CIFAR-100
ImageNet-200
Rob. accurac Claan scuraoy
securacy
Reb. Sccnce
88
20
On whole dineed dinnet
-ardio crandoo
fil
ESAT-crroo
-ardar
ESAT-ardsm
-aveett frtt
nec.
Clean scursey
Clean sccuracy
a
206
30
45k
19
25h
35k
50k
99% 13% 196k 201 Number adv. examples ([4))
adv. examples (I,4))
Number adv. trained examples (IAI))
Figure 18: Example-subset Adversarial Training (ESAT) CIFAR datasets and ImageNet-200 using Los with norm 6 8/235, provide quick convergence to full AT baseline (gray line and dot) with increasing size of A. We report robust accuracy (upper row) and clean accuracy (lower row) and observe similar characteristics as with CSAT (figure 5). L.e., selecting the hardest examples first (solid line) provide higher rob. accuracy than easy ones (dashed line), although the gap substantially widens. Random example selection (red) provides competitive performance on average. Across all datasets, we see the common clean accuracy decrease while robust accuracy increases (Tsipras al., 2019).
19

---

Published as a conference paper at ICLR 2024
CIFAR-100 CIFAR-10
ImageNet-200 - Caltech-256
ImageNet-200 - Flowers- 102
Rob accumery
Rob. accursty Caheeh28 1
Foo aoreinn
5-GSHT badeet
5.54T-urdee
5-ESAT lndec S-csat-aican S-EAAT mase
5-GGAT hadled Firt ewen
S- 5-54T-ordi
esaen
Clee acceen CIFAR30
Clea acorso CalweleS5
Caven acceaary Flowess
1 5
2
655
54 10
19
734
754
314
304
63c
7% en Tta 96 117-1466 1514 19t6 2016
75 <h 73 6eL 118 H4k 164. 181-
adv. trainel soomppes
[141] CIFARIOD
ath.. trained socamples [[A[] 1m0goNae2200
+ abv. traiee stampios ([Al) ImagsNaz200
Figure 19: Transfer from S-ESAT to three different downstream tasks using Loo with norm Coc $/255. S-ESAT is trained on source dataset CIFAR-100 (left) and ImageNet-200 (middle and right). We report robust (top row) and clean (bottom) accuracies for increasing size of A. Similar to our observations for L2 in figure 8, we find that hard examples provide better robustness transfer than easy ones, but random selections (red) achieve competitive performances. Most importantly, "seeing" only few AEs (here 30% of source data) recovers baseline AT performance (gray line).
CIFAR-100 + CIFAR-10
ImageNet-200 + Caltech-
ImageNet-22 Flowers- 102
Fob. scouti Someeeii
201522011
S-STT (1561103
S.AA
Chen souuser
Clus accussty CHhteh256
Geen accurccy Fowers22 2
90
adv trained classes (1.4) CIFARLDO
adv. trained classss (1.4) ImageNac200
adv.
trained dasses [1A]) ImageNer220
Figure 21: Transfer from S-CSAT to the same downstream tasks as in figure 8. S-CSAT is trained on source dataset CIFAR- 100 (left) and ImageNet-200 (middle and right). We report robust (top row) and clean (bottom) accuracies for increasing size of A. We observe similar properties to S S-ESAT, yet find convergence to the baseline AT performance to be substantially slower; in line with our discussion on SAT in section 3.2.
A.9 FULL RESULTS FOR TRANSFER SETTINGS
In the main paper, omitted transfer results to SVHN as well as using S S-CSAT. Firstly, we provide the transfer result from CIFAR-100 to SVHN in figure 20. Robust accuracies on the upper plot, clean accuracies below. Note that 5k examples in A are sufficient reach baseline AT performance (gray line), while 15k provides a substantial improvement in robust accuracy 22% 20%). Secondly, transfer results on S-CSAT aligned with the experiments in section 4.3 are shown in figure 21. We observe similar characteristics the CSAT results section 3.2, i.e. selecting the hardest classes first (solid line) is only advanta- geous on small A, while generally it draws even with the random baseline (red). Overall, convergence to the full AT baseline is slower than with S-ESAT.
plotted
Rob accuracy SVHN
S-ESAT frx.
Clean eccuraty
SVHN 8
3
15k 70k 75%
30k 35k 40k 45
of adv. trained examples (IAl) on CIFARI
A. SINGLE-STEP AT
Figure 20: Robustness transfer from CIFAR-100 to SVHN using S-ESAT
While main Lg experiments use AT with PGD-steps, we here show that non-trivial robustness transfer can be achieved with single-step AT as well. We focus transfer downstream
20

---

Published as a conference paper at ICLR 2024
tasks and compare with the results shown in figure section 4.3. Le., train ESAT model on CIFAR-I00 and ImageNet- 200 respectively, and finetune an additional classifier on either CIFAR-10 10, Caltech256 Flowers-102. We use FGSM-RS (Wong et al., 2020), with step-size of 0.625 for < <= 0.5 and 75 for e = 3.0. All other training settings are consistent with previous experiments (c.f.
section A.1).
Results are shown in figure 22, comparing PGD-7 training (circles on solid line) and single-step FGSM-RS (squares on dotted line). Generally, we observe very similar clean and robust accuracies (lower and upper row) across all architectures. Specifically, FGSM-RS achieves slightly higher clean accuracies and slightly lower robust accuracies especially for small AA|. Nonetheless, single-step AT converges to the full AT baseline (gray line) in similar fast rate, i.c. generating AEs for around 30% of the training set sufficient.
CIFAR 100 + CIFAR-10
ImageNet-200 Caltech-256
ImageNet-200 Flowers- 102
not Atateey
Rob. zccupey Caheeh055
35.0
PGD-1 Ladett Enc.
AT ea OFFRLLO
PCD- wisd Srt POOULAS badest
Fal Waaphiaa
PCD7 kadee Eac:
Inageleec00
25 ween
Claa SccUPACe Caiieh2255 5
Lisen accureet 2o0e-00 2000
2
281 436
236 49 94 137-1404 2944 351 2014
23. 40. 72s p4t 21 348.144
adv. trained eiamples (144) CIFAR100
trained eimmples (14)) ImageNet220
trained
(IA1) ImageNet200
Figure 22: Comparison between PGD-7 and single step S-ESAT on the transfer setting three different downstream tasks. Training and evaluation using L2 norm. Our observations on robustness transfer remain valid even for single step attacks. Except for CIFAR 100 - CIFAR- 10, we find robust accuracies and clean accuracies to be consistent with PGD-7.
21


Reviews:
[{"review_id": "oUvCbrL4Sr", "reviewer": "Reviewer_QzqJ", "summary": "The paper studies the impact of adversarial training with a limited subset of data on the robust accuracy of computer vision models. The empirical study shows that a limited amount (30%) of carefully selected data is sufficient to achieve 90% of the robustness of the models. Additionally, the paper explores the transferability to other models of the robustness acquired with their method. Empirically, the paper shows that model robustness is best preserved with a custom-balanced loss and with hard-to-classify examples.", "questions": "- What is the exact process of adversarial training used in the experiments? Are all examples adversarially perturbed? Are some examples perturbed and others clean? In what proportion? In total is the number of perturbation executions the same for adversarial training as subset adversarial training? \n\n- What is the objective of transferring the robustness across classes and tasks? We need to retrain models to adapt to these new classes and tasks, isn’t it simpler to adversarial train them at retraining, therefore empirically obtaining a more robust model ?\n\nOther comments:\n\n- On page 6,  lines 6 and 11, of the paragraph left to fig 3, describe the results for the class “car” while the numerical value corresponds to the line “plane” on fig 2.\n- Figures are not readable when printed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper addresses the relevant problem of the generalization of the acquired robustness to unseen classes, examples, and tasks.\n- The paper is well-written and easy to grasp.\n- The experimental protocol is well described and allows to reproduce the study.\n- The paper provides an in-depth empirical study to support its claim.\n- The paper proposes a method to estimate the transferability of the acquired robustness which can be useful for testing ML models, as new classes and examples can appear after adversarial training in real-world systems.", "weaknesses": "I have a single but potentially critical concern regarding the significance of this work. In particular, I am unsure what are the benefits of considering a subset of the adversarial examples during adversarial training to improve the efficiency of the process. \n\nAs I understand, the key objective of this paper is to limit the size of the set of examples used for adversarial training, to achieve a similar robust accuracy than with full adversarial training. The reason is that full adversarial training is computationally expensive. Considering the settings described in the experimental protocol, I fail to understand why the proposed process is more efficient than full adversarial training. In both cases, the full datasets need to be labeled and the same amount of computational resources is needed, as only the set of available examples for adversarial training differs, not the total number of examples used during adversarial training (since the subset of adversarial examples is completed with the clean examples not used for generating adversarial examples. One could argue that we save 70% of the time to generate the adversarial examples, but the largest cost still come from model training. Considering the empirical results that demonstrate that the proposed method does not lead to better robust accuracy than full adversarial training, I do not see the added value for model robustness efficiency/effectiveness. It would be good if the authors could clarify the benefits of using only a subset of examples in the adversarial training process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JE3vz5YBWE", "reviewer": "Reviewer_gZFF", "summary": "This paper conducts extensive experiments to see how only perturbing a subset of adversarial examples in training impacts adversarial robustness.  The authors find that using adversarial examples from certain classes can lead to nontrivial gains in robustness in other classes (despite not training with those classes).  The authors find that the most useful examples/classes to train with are correlated with their difficulty which they measure using entropy.", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- paper is very clear\n- great scope in experiments which encompass multiple datasets and model architectures\n- experiments clearly demonstrate correlation between entropy and robust accuracy on the subset\n- experiments suggest that in certain settings, we can train with a smaller subset of adversarial examples instead of all adversarial examples which can reduce the runtime of adversarial training, making it more feasible in practice.  Robustness also transfers to other datasets as well suggesting that this approach can be used with pretraining models.", "weaknesses": "- while it's clear that training with smaller subsets of adversarial examples can be beneficial, are there guidelines for how to determine the size of this subset to use SAT in practice?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fiD7KTfX4Q", "reviewer": "Reviewer_3h75", "summary": "To investigate the transferability of adversarial robustness across different classes and tasks, the authors proposed Subset Adversarial Training (SAT), which splits the training data into A and B and constructs adversarial examples (AEs) on A only.  Using SAT, this paper shows that training on AEs of just one class (e.g., cars in CIFAR-10) can transfer a certain level of robustness to other classes. Hard-to-classify classes (like cats) tend to provide greater robustness transfer compared to easier ones. Moreover, using AEs generated from half of the training data can match the performance of full AT. These findings also apply to downstream tasks. This paper distinguishes itself from others by only creating AEs on a pre-defined subset of the training set, independent of the model's architecture or the specifics of the training process.", "questions": "1. Can you provide more theoretical justification or intuitive explanations for the observed efficacy of constructing AEs on only a subset of the training data? Specifically, what underpins the phenomenon where harder examples offer better robustness transfer?\n2. You mentioned that as dataset complexity increases, the trend of harder examples providing better robustness transfer diminishes. Can you explain the reasons behind this observation? Are there specific characteristics or properties of complex datasets that might be influencing this behavior?\n3. Could you explain more on why the robustness transfer notably increases for smaller $\\epsilon$ and decreases for larger $\\epsilon$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides valuable insights into the transfer properties of adversarial robustness. The observation that adversarial robustness can transfer to classes that have never been adversarially attacked during training is intriguing.\n2. The finding that generating AEs on merely 50% of the data can recover most of the baseline AT performance, especially on large datasets like ImageNet is insightful. This could potentially lead to significant computational savings without compromising the robustness of the model.\n3. The paper's findings are not limited to a single task or dataset. The authors have undertaken a thorough experimental evaluation across multiple datasets.", "weaknesses": "1. While this paper presents intriguing empirical results on the SAT approach, it falls short of providing a clear explanation for the observed transferability of adversarial robustness from subset A to subset B.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sgv4JKyw7p", "reviewer": "Reviewer_uHfk", "summary": "In this paper, the authors study the generalization of adversarial robustness from class-wise and sample-wise aspects.", "questions": "1. I notice that the authors use stronger data augmentation than usual. For example, when training on CIFAR-10, we usually only use random crop and flip. I hope the authors can provide ablation studies on these data augmentation methods.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. There are two different settings, i.e., CSAT and ESAT. For each of them, there are comprehensive experiments based on the proposed entropy metrics. On the other hand, the authors consider both $L_2$ and $L_\\infty$ attacks. All settings are studied on multiple datasets, which makes the results more convincing.\n\n2. Downstream task transferability is an interesting topic. The results indicate that the learned features can be transferred to other classes, which is aligned with the observation from CSAT.\n\n3. This paper is easy to follow. The writing is clear.", "weaknesses": "1. This paper mainly contains various experiments and their results, but lack the important analysis. Specifically, there are no analysis of the transferability observed from CSAT and ESAT. I cannot get any insightful information after reading this paper, although the results are informative.\n\n2. There is no theoretical analysis to rethink the observation as a special property of adversarial training. Additionally, from Figure 4, we can find that the clean accuracy has a similar tendency as the robust accuracy, therefore, it is possible that CSAT and ESAT are just because of the generalizability of the deep learning models.\n\n3. Only PGD-AT is considered. More advanced methods, like TRADES and AWP, should be evaluated under the same settings.\n\n4. For downstream task transferability, it is similar, but not exactly the same, as contrastive adversarial training. The authors should discuss the similarity and difference between these two methods in this case.\n\n\nMinor:\n\na. some figures' labels are blocked.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)