PubDef: Defending Against Transfer Attacks From Public Models
Authors: Chawin Sitawarin
Jaewon Chang
David Huang
Wesson Altoyan
David Wagner
Venue: ICLR 2024 poster (2024)
Keywords: adversarial robustness
adversarial examples
transfer attack
security
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We propose a new practical threat model, transfer attacks from public models (TAPM), and build a defense that provides higher robustness than adversarial training with almost no drop in the clean accuracy compared to undefended models.

Abstract: Adversarial attacks have been a looming and unaddressed threat in the industry. However, through a decade-long history of the robustness evaluation literature, we have learned that mounting a strong or optimal attack is challenging. It requires both machine learning and domain expertise. In other words, the white-box threat model, religiously assumed by a large majority of the past literature, is unrealistic. In this paper, we propose a new practical threat model where the adversary relies on transfer attacks through publicly available surrogate models. We argue that this setting will become the most prevalent for security-sensitive applications in the future. We evaluate the transfer attacks in this setting and propose a specialized defense method based on a game-theoretic perspective. The defenses are evaluated under 24 public models and 11 attack algorithms across three datasets (CIFAR-10, CIFAR-100, and ImageNet). Under this threat model, our defense, PubDef, outperforms the state-of-the-art white-box adversarial training by a large margin with almost no loss in the normal accuracy. For instance, on ImageNet, our defense achieves 62% accuracy under the strongest transfer attack vs only 36% of the best adversarially trained model. Its accuracy when not under attack is only 2% lower than that of an undefended model (78% vs 80%). We release our code at https://github.com/wagner-group/pubdef.

Published as a conference paper at ICLR 2024
PUBDEF: DEFENDING AGAINST TRANSFER ATTACKS FROM PUBLIC MODELS
Chawin Sitawarin UC Berkeley
Jaewon Chang*
David Huang
UC Berkeley
UC Berkeley
Wesson Altoyan
David Wagner UC Berkeley
King Abdulaziz City for Science and Technology
ABSTRACT
Adversarial attacks have been a looming and unaddressed threat in the industry. However, through decade-long history of the robustness evaluation literature, we have learned that mounting strong or optimal attack is challenging. requires both machine learning and domain expertise. In other words, the white-box threat model, religiously assumed by a large majority of the past literature, is unrealistic. In this paper, we propose a new practical threat model where the adversary relies on transfer attacks through publicly available surrogate models. We argue that this setting will become the most prevalent for security -senstivvee applications in the future. We evaluate the transfer attacks in this setting and propose a specialized defense method based game-theoretic perspective. The defenses are evaluated under 24 public models and 11 attack algorithms across three datasets (CIFAR- 10. CIFAR-I00, and ImageNet). Under this threat model, our defense, PUBDEF, outperforms the state-of-the of-tth--rttt art white-box adversarial training by a large margin with almost no loss in the normal accuracy. For instance, on ImageNet, our defense achieves 62% accuracy under the strongest transfer attack vs only 36% of the best adversarially trained model. accuracy when not under attack only 2% lower than that of an undefended model (78% vs 80%). Code is available here.
INTRODUCTION
Current ML models are fragile: they are susceptible adversarial examples (Biggio et al., 2013; Szegedy et al., 2014; Goodfellow et al., 2015), where small imperceptible change to an image radically changes its classification. This has stimulated a profusion of research on ML--based methods to improve the robustness to adversarial attacks. Unfortunately, progress has slowed and fallen far short of what is needed to protect systems in practice (Hendrycks et al., 2022). In this paper, we articulate a new approach that we hope will lead to pragmatic improvements resistance to attacks.
In the literature, we can find two defense strategies: systems-level defenses and ML-level defenses. Systems-level defenses include controls such as keeping the model weights secret, returning only the predicted class and not confidence scores, monitoring use of the model to detect anomalous patterns, etc. Unfortunately, systems- level defenses have proven inadequate attacks can successfully attack a target model even without knowing its weights. Therefore, most research focuses MI-elevel defenses, where we try to build models that are more robust against such attacks, for example through novel architectures and/or training methods, Researchers made early progress on ML-level defenses, with the introduction of adversarial training (Madry al., 2018), but since then progress has slowed dramatically, and there is no clear path to achieving strong adversarial robustness in any deployable model.
their own: for instance, transfer
Current ML-level defenses suffer from two major problems: first, they have an unacceptable negative impact on clean accuracy (accuracy when not under attack), and second. they focus on a threat model that is increasingly recognized to be unrealistic (Gilmer et al., 2018; Goodfellow, 2018; Hendrycks et al., 2022; Apruzzese et al., 2023). These problems are closely related: prevailing academic threat models are unrealistic as they grant the attacker excessive powers that are hard to realize in real life,
"Equal contribution. Corresponding email: chawi ns@berkeley eed..e.

---

Published as a conference paper at ICLR 2024
(a) Transfer Attack From
(b) Defense in Depth
Public Models (TAPM)
Public
available
quaries"
deta
Source
Black- Box Target Model
Model
White-
Query
Transfer Attack
Transfer
Algorithm
TAPM
Input Sample
Attack Sample
Source Model, Attack Algorithm
(c) PubDef
Defender's Model
Train
Clean Training Sample
Figure 1: (a) Proposed threat model: transfer attack with public source models (TAPM). We consider a low-cost black box adversary who generates adversarial examples from publicly available models with known attack algorithm. (b) Our approach is based on stopping each major category of attack with combination of multiple mechanisms. (c) Our defense, PUBDEF, trains the defended model to resist transfer attacks from several publicly available source models, Our model is robust to a wide range of transfer attacks, including both those from source models that were trained against and others that were not trained against, while also maintaining high clean accuracy.
making it too difficult to achieve strong security against such unrealistic attacks. Because of the existing trade-off between adversarial robustness and clean accuracy, this in turn means achieving any non-trivial adversarial robustness currently requires unacceptable degradation clean accuracy.
We advocate for a different approach, captured in slogan form as follows:
Secure ML
Realistic threat model +
Systems-level defenses MI--level defenses against those threats
+
We propose using all available systems-level defenses. We articulate a concrete threat model, influenced by what attacks cannot be stopped by systems-level defenses. Specifically, we propose security against transfer attacks from public models (TAPM; Fig. ((a) as the threat model we focus The TAPM threat model focuses ön transfer attacks where the adversary starts with publicly available model, attacks the public model, and then hopes that this attack will "transfer", i.e., will also be effective against the target model. Because public models are often widely available, e.g., in model zoos, this kind of transfer attack is particularly easy to mount and thus particularly important defend against. Under the TAPM threat model, we assume neither the model weights nor training set are known to the attacker, and the attacker cannot train their own model or mount query- query-based based attacks that involve querying the target model many times. These assumptions are driven partly by what kinds of attacks can be prevented or mitigated by existing systems-level defenses.
Finally, we introduce PUBDEF (Fig. I(c)), new method for training models that will be secure against transfer attacks from public models. PUBDEF models are attractive for practical deployment. For instance, they achieve clean accuracy close to that of an undefended model, so there is little loss in performance when not under attack. When under attack (via transfer from public models), adversarial accuracy remains fairly high: 88.6% for CIFAR-10 10 (almost 20 points higher than any previous defense), 50.8% for CIFAR-100 (18 points higher), and 62.3% for ImageNet (26 points higher than any previous defense). While our defense is not perfect and is not appropriate in all scenarios, believe it is pragmatic defense that be deployed without major loss of clean accuracy, while making life as difficult for attackers as possible within that constraint.
RELATED WORK
We provide an introduction to several types of attacks and threat models seen in the literature, for comparison new threat model.
2

---

Published as a conference paper at ICLR 2024
CIFAR-10
C1FAR-100
ImageNet
Defenses
Clean
Adv.
Clean
Adv.
Clean
Adv.
No defense
96.3
0.0
81.5
0.0
80.4
0.0
Best white-box adv. train
85.3
68.8
68.8
32.8
63.0
36.2
DVERGE adv. train
87.6
59.6
6.3
2.1
TRS adv. train
86.9
66.7
63.9
39.1
PUBDEF (ours)
96.1 (4-18.)
88.6 (4:19.8)
76.2 (-7.4)
50.8 (418.0)
78.6
14-15
63.0 (+26.8)
Table 1:
Clean and adversarial accuracy of PUBDEF vs the best previously published defenses against transfer attacks, Adversarial accuracy is measured in the TAPM threat model. "White-box adv. train" are the most robust models from ROBUSTBENCH which share the same architecture as PUBDEF. DVERGE (Yang et al., 2020) and TRS (Yang et al., 2021) are two ifate-of-the arr defenses against transfer attacks. "DVERGE is designed for CIFAR- 10 and is difficult to train on the other datasets. TRS/DVERGE with adversarial training is not included for ImageNet due to its computation cost.
White-box attacks. In this threat model, the attacker is assumed to know everything about the target model, including all model weights. This is the most studied threat model in the literature. Adversarial training (Madry et al., 2018) has been the primary defense against white-box adversarial examples. However, adversarial training sacrifices a considerable amount of clean accuracy (Tsipras et al., 2019), rendering it unattractive to deploy in practice.
Transfer attacks. Papernot et al. (2016) first demonstrated the threat of transfer attacks: adversarial examples generated on one ML model (the surrogate) can successfully fool another model if both models are trained on the same task. Liu et al. (2017); Tramèr et (2017); Demontis et al. (2019) propose various methods for quantifying a degree of the attack transferabilityy including the distance to the decision boundary as well as the angle between the gradients of two models. A great number of transfer attack algorithms have been proposed over the years (Zhao et al., 2022), e.g., using momentum during optimization (Dong et al., 2018; Lin et al., 2020; Wang et al., 2021b), applying data augmentation (Xie et al., 2019; Wang et al., 2021a; Lin et al., 2020), and alternative loss functions (Zhang et al., 2022; Huang et al., 2019). In this threat model, researchers often assume that the training set for the defended model is available the attacker, and the attacker can either train their own surrogate models or use publicly available models as a surrogate.
Query-based attacks. Consider the setting where the attacker can query the target model (submit an input and obtain the classifier's ' output for this input), but does not know the model's weights. This threat model is particularly relevant for classifiers made available via an API cloud service. Attacks can iteratively make a series of queries to learn the decision boundary of the model and construct an adversarial example (Brendel et al., 2018; llyas et al., 2018; Andriushchenko 2020). Systems-leve defenses include returning only hard-label predictions (the top-1 predicted class but not the confidence level), rate-limiting, and monitoring queries to detect query-based attacks (Biggio et al., 2013; Goodfellow, 2019; Chen et al., 2020b; Li et al., 2020).
al.,
3 THREAT MODEL.
We first define our threat model for this paper: transfer attack with public models (TAPM). It is designed to capture a class of attacks that are especially easy and low-cost to mount, do not require great sophistication, and are not easily prevented by existing defenses. It fills in a part of the attack landscape that has not been captured by other well-known threat models (e.g.. white-box, query-based, and transfer attack). Under TAPM, the adversary has the following capabilities:
1. They have white-box access to all publicly available models trained for the same task. They can mount a transfer attack, using any public model as the surrogate.
2. They cannot train or fine-tune neural network. This might be because a reasonable training set not publicly available, or because the training process requires substantial expertise and resources that outweighs the economic gain of the attack.
3. They can submit one or more adversarial inputs to the target model but cannot run query- based attacks. This assumption is particularly well-suited to security-sensitive tasks, e.g., authentication and malware detection, where the adversary is caught immediately if the attack fails, or to systems where other effective defenses against query- basey attacks can be deployed.
By default, we assume that the defender is also aware of the same set of public models. However, we will later show that our defense generalizes exceptionally well unseen public models.
3

---

Published as a conference paper at ICLR 2024
Notation. Let S {S (S...... S denote finite set of all public models the same task and A = {Ai...., A a set of known transfer attack algorithms. An attack generates an 4p ---norm bounded adversarial example from model A and an input sample 2::
Favv = A(S, (z, v)) such that ||Fady - x=l,, S
where S E S is a source or surrogate model. A transfer attack Padv is uniquely defined by pair (S, A). The attack is then evaluated on target model T é and considered successful (xaadv) o y-
THEORETI PERSPECTIVE
We begin by motivating our defense through a game-theoretic lens. Prior work has formulated adversarial robustness a two-player zero-sum game (Araujo al., 2020; Meunier et al., 2021; Rathbun et al., 2022) but under different threat models and contexts. Under our TAPM setup. the attacker's ' strategy is naturally discrete and finite. The attacker chooses a source model S E S and an attack algorithm A E A and obtains an adversarial sample Jadv (as defined in Eq. (1)). Essentially, each pair (S, A) corresponds to one of [S] |A| a attack strategies. We will describe two versions of the game with different defender strategies.
4.1 SIMPLE GAME
As warm-up, will first consider discrete defense strategy where the defender trains models, one against each the attack strategies. Denote a defender's 's model by T E 'T where [T] = 8 a. The defender's strategy is choose to classify given Fadv where T is trained to minimize the expected risk of both the normal samples and the transfer adversarial samples Tady from Eq. (1).
arg min En.y [L(Ta(x)) 8) LLLTe(Tadd)))) y)]
(2)
Note that this formulation is similar to the well-known adversarial training (Goodfellow et al., 2015; Madry et al., 2018) 8) except that Tndv is independent to O or the model being trained. The payoff of the defender is defined as the expected accuracy on Tadv chosen by the attacker:
(D(TA,, ET= En ( w))) = y
where AA. TD mixed (i.e., potentially randomized) strategies for the attacker and the defender, respectively. In other words, AA, TD each represents a multinomial distribution over the s a pure (i.e., non-randomized) strategies. The attacker's payoff' is matrix R. E Reaxee is defined by Ra,j = (D(
-LD π A, DD. The payoff
S
S2
$,
S,
As example, we empirically compute R (Fig. 2) choosing S = [S,,..., S to be four public models and A as only PGD attack (Madry et al., 2018). We will later describe how these mod- els are chosen in Section 5.2. The defender also has four models T {T, -....... T.}, where T; is udversarially trained be robust against transfer attacks from S,. Notice that the diagonal entries are large because T, is trained on the attack from S;. Von Neumann's 's minimax theorem guarantees the existence of a Nash equilibrium, ie., an optimal strategy for each player (v. Neumann, 1928) (see Appendix A.3). The optimal strategy can be efficiently computed using linear programming (van den Brand, 2020). For the payoff matrix in Fig. 2, the expected payoff for the optimal strategy is 73.0, meaning that when both the attacker and the defender choose their strategies optimally, the target model can achieve 73.0% accuracy on average. This is reasonable, but show next,
Ti
Defender's Options
Ta
65.0
74.0
42.0
14.0
Tar
63.0
93.0
400
T
80.3
63.2
61.6
94.8
Attacke's Options
Figure 2: The payoff matrix of the simple game.
do better.
4.2 COMPLEX GAME
Now we make the defender's action space more flexible: the defender can choose their model's weights arbitrarily, instead of being limited to one of s models. We extend the loss function in Eq. (2) to represent the loss against transfer attack chosen according to mixed strategy AA:
arg min Exy
L(fo(z).)) +
TgL (So (Iadv.i),
(4)
4

---

Published as a conference paper at ICLR 2024
where Tadv,i is attack generated by the i-th attack strategy, and the attacker's (mixed) strategy is given by π = (Ti,..... Toa) representing a probability distribution over the s a (pure) attack strategies. Note that T; =
recover the simple game if the adversary is restricted to choosing TT 's s.t. for a single i and 0 otherwise. However, when л represents any probability distribution, the reward function is longer linear in von Neumann's 's minimax theorem longer applies. A Nash equilibrium may exist, but there is no known efficient algorithm to compute it.'
One naive strategy for the defender is to assume the attacker will choose uniformly random from all 8 attacks and find the best response, i.e., find model weights 0 that minimize Eq. (4) when T, 1/sa for all i's. This amounts to adversarially training model against this particular (mixed) attacker strategy. In this case, the defender's 's payoff' (adversarial accuracy) against each of the four attacks turns out to be [96.3, 90.4, 94.6, 96.0), O). for the payoff matrix in Fig. 2. This means the defender achieves over 90% accuracy against all four transfer attacks, which is a significant improvement over the equilibrium of the simple game (73%). This suggests that while we may not be able to solve the complex game optimally, it already enables much better strategy for the defender. In Section 5, we will explore several heuristics to approximately find a "local" equilibrium.
5
OUR PRACTICAL. DEFENSE
5.1 LOss FUNCTION AND WEIGHTING CONSTANTS
We propose several heuristics for solving the complex game (Section 4.2) that do not require explicitly computing the full payoff matrix. Instead, the defender trains only one model with adjustable weights T.'s 's as defined in Eq. (4). We experiment with the three training losses (detailed in Appendix A.4) and report the best one. Overall, we find that simply sampling one attack randomly (a pair of A)) at each iteration performs well or best in most cases.
5.2 DEFENDER'S ' SOURCE MODEL SELECTION
Given that the set of publicly available models is public and known to both the attacker and the defender, the most natural choice for the defender is to train against all publicly available models. However, the computation cost can be prohibitive. We show that we can achieve nearly as good performance by choosing only a small subset of the publicly available models. However, finding an optimal set of source models is non-trivial without trying out all possible combinations.
Intuitively, to be robust against a wide range of transfer attacks, the defender should train the target model against diverse set of source models and algorithms. The "diversity" of set of models is challenging to define. Natural approaches include using diverse set of architectures (e.g., ConvNet vs methodologies (e.g., supervised vs unsupervised), and
Transformer), diverse set of diverse set of data augmentation strategies. In our experiments, we found that the training procedure- (4) corruption robusst training has the largest effect on the defense. For the rest of the paper, we categorize the source models into one of these four groups. 2 We will discuss the implications of this grouping in Sections 6.2 and 7.1.
namely (1) normal, (2) lox aaversarial,, (3) fz-adversarial,
This motivates a simple yet surprisingly effective heuristic that we use for selecting the set of source models: when training PUBDEF, we use one source model from each group (four source models in total for CIFAR-I0 10 and three for CIFAR- 100 and ImageNet). In more detail, we first choose four source models: the public model that is most robust against foc white-box attacks, the public model that is most robust against & white-box attacks, the public model that is most corruption robust, and one arbitrary public model that is normally trained. Then, we compute the adversarial accuracy against transfer attacks from every publicly available model. If the adversarial accuracy against transfer attacks from some other public model 5' is significantly lower than the adversarial accuracy against transfer attacks from S (the chosen model in the same group as S'), then we swap in S' and remove S. We made one swap for CIFAR- 100 and ImageNet and no swap for CIFAR-10. We find that this simple heuristic works well in practice and performs better than random subset (Section 7.1).
If we discretize each n. the equilibrium is still guaranteed exist by Nash's theorem (Nash, 1951), but we need to deal with an exponential (in sa) number of defense models. For CIFAR-100 100 and ImageNet, it is hard to find fz-adversarially trained models that are publicly available, so exclude this group and only consider the remaining three (normal, foc, and corruption).
5

---

Published as a conference paper at ICLR 2024
6 EXPERIMENTS
6.1 SETUP
Metrics. The two most common metrics used to compare models in the past literature are clean and adversarial accuracy. The clean accuracy is simply the accuracy on the test set, with no attack. There are multiple ways to measure the adversarial accuracy under our threat model depending on the attacker's strategy (c.g,, average or worst-case). We conservatively assume that the attacker knows the defender's 's strategy and chooses the best pair of (S, A) to run the attack. 3 In other words, we report the adversarial accuracy against the worst-case TAPM attack.
Baseline defenses. Adversarial training (Madry et al., 2018) can defend against transfer attacks but at the cost of excessive degradation to clean aceuracy (as we will show next). We compare PUBDEF to the best white-box adversarially trained (AT) model from ROBUSTBENCH that has the same architecture. For CIFAR-10, CIFAR-100, and ImageNet, the best are Addepalli et al. (2022b), Addepalli et al. (2022a), and Salman al. (2020), respectively. Additionally, we compare our method to DVERGE (Yang et al., 2020) and TRS (Yang et al., 2021), the two statc-of-the-arr defenses against transfer attacks. These use an ensemble of models for greater diversity, so that an attack that succeeds on one might fail on another, hopefully making transfer attacks harder.
Public source models. For each dataset, we select 24 public pre-trame mr-ttrineed models: 12 normally trained and 12 robustly trained models. The normal group comes from multiple public model zoos including Hugging Face (Face, 2023), t timm (Wightman, 2019), and (for CIFAR- 10/10-001000 10)) two Github repositories (Chen, 2023; Phan, 2023). The robust models are all hosted on ROBUST BENCH (Croce et al., 2020). We select wide variety of models based on their architecture (c.g., assorted ConyNet, VIT (Dosovitskiy et al., 2021), Swin (Liu et al., 2021), BeiT (Bao et al., 2022), ConvMixer (Trockman & Kolter, 2023), zero-shot CLIP (Radford et al., 2021), etc.) and training methods to ensure high diversity. We try our best not select two models with the same architecture or from the same paper.
Atta Attack algorithms. We evaluate the robustness of the defenses with 11 different attack algorithms. All of the attacks are gradient- based, but they utilize different techniques for improving the transferability of the attacks, e.g., momentum, data augmentation, intermediate representation, etc. Please refer to Appendix A.1 for the complete list of both the source models and the attack algorithms.
PUBDEF training. We adversarially train the defended model by selecting a subset of source models according to the heuristic in Section 5.2 and then using a PGD transfer attack on that subset. We use a WideRes architecture for CIFAR-10/100 and ResNet-50 for ImageNet. We do not use any extra data or generated data for training. Appendix A.1.3 has the complete training details.
6.2 RESULTS
PUBDEF is more robust to all 264 transfer attacks than the baselines by large margin. We generate 264 transfer attacks, one for each of 11 attack algorithms and 24 source models, and evaluate PUBDEF according to the most successful attack. For CIFAR- 10, PUBDEF achieves 20 percentage points higher adversarial accuracy than the best previously published defense, with comparable clean accuracy to undefended model (Table 1). For CIFAR- 100, PUBDEF achieves 10-18 p.p. higher robustness and 7-12 p.p. higher clean accuracy than the best prior defense. For ImageNet, PUBDEF achieves 26 p.p. better adversarial accuracy and 15 p.p. better clean accuracy than adversarial training; its clean accuracy is only 2 p.p. lower than an undefended model. is beyond our resources to train TRS and DVERGE for ImageNet, due to the combination of ensembling and adversarial training.
Fig. 3 shows all adversarial accuracies of PUBDEF by pairs of (S, A) on ImageNet. Here, the overall best attack is NI-Admix-TI-DD from a ResNet-50 on Hugging Face. M-PGD, Admix, and NI-Admix- TI-DIM are generally the strongest attack algorithms across source models and datasets. Surprisingly, PGD usually performs well above average compared to the other more sophisticated attacks. Appendix A.6.2 shows more detail.
PUBDEF maintains high clean accuracy. Compared to the Mfate-oftthe-ee top-1 clean accuracy for undefended models, our defense experiences only 0-5% drop in the clean accuracy (Table I). Compare white-box adversarial training, which suffers 11 11-18% drop. We emphasize that the
Thhis assumption is realistic because, with limited queries, the attacker can intelligently pick a good source model using the method from Maho et al. (2023), for example.

---

Published as a conference paper at ICLR 2024
Singi Singt22116 W0 rrriinngg..... ConvieXi T-C-T-Can Stom
Wlswin win-wwt-yyyy patch wii--iaddw 7-224
1.,224 grog. bitk it fninkk
(Normal)
(Pa)
(Corrupt))
Un Unmonoobil bilneieii Jarge. 100 inik
Singh Mighat22Revvi Reviatingg VIT Constea
L022123C ComperC Comprrleaaatt www.eeavvv...................... Swin-B
Light, S12
mikmme google bass pattttatttee patttt6--22
(aochokk basdhoo//cmmnnn www.eeoeeeeetttt.coo cil--tn---- 224
Unimi Unmieetic mmmieeiicicnnen miliiiieeet...... hi mz Jolk
Gram presseet 1881 mapp. .inik
Him Limmn momiieane m08 iilk
unansfined se mn. aug in
Thac wocco221ees Derppp Du/T.s
DeiT-88
Nobboy NobbyMis .ees
microviti Saaroaottieesst--- -30
timml L.v Jb
50
Hondryaks 2022 202thann
Hendyook wot.k:000AAggg
Geitho 2018 SIN IV
Unito www.iicciiiooo..
Salntn Salnon2tttl 3020
20 300000
Dehenerlatt
Salman
Teat 20ar20222
Exichemm
PGD
76
76
76
76
70
70
74
82.5
M-RGI
72
73
72
69
D2
13
72
74
72
7L
72
21
71
65
70
69
70
7
67
66
67
68
Transfer Attack Algorithms
Pregradient
71
73
72
72
21
69
66
67
68
50.0
DI
77.5
TI
74
Admix
68
24
74
20
22
75
63
6R
68
80
79
30
27
22
2O
28
79
mía
81
30
79
SD
S1
29
78
NI-I-TE-DIM
64
65
AM
72
TO
6O
75
74
AN
6S
64
67
NI-Admix-TI-DD TI-DIM
63
70
69
66
6F
61
66
70.0
AntoPGD-DLR
67.5
AutoPCiD-CC:
76
76
Arg
75
73
75
74
75
74
75
74
70
25
74
65.0
Dest
63
71
65
63
68
21
70
67
66
69
21
71
68
70
69
70
68
68
64
63
64
66
Transfer Attack Source Models
Figure 3: Adversarial accuracy of PUBDEF against 264 transfer attacks (24 source models x attack algorithms) on ImageNet. denotes the source models this defense is trained against. We cannot produce NA attack on t timm's VGG model (shown as "n/a") because of its in-place operation.
96
95.6
05.9
Src.
Algo.
CIFAR-10
CIFAR-100
ImageNet
Soen
Scen
90.3
52.6
70.6
94-
Accuracy
Unseen
Seen
90.3 (-0.0)
52.6 (-0.0)
68.6 (-2.0)
Clean Acb.
Seen
Unseen
90.3
50.8
63.0
Adv, Acc.
Unseen
Unseen
88.6
50.8 (-8)
63.0 (-7.6)
90
9022
88.6
390
88:0
Table 2: Adversarial accuracy of PUB DEF under seen/unseen transfer attacks. Seen attacks (seen src. and seen algo.) are the 3-4 attacks that were used train our defense, unseen attacks are all others from the set of 264 possible attacks. They are categorized by whether the source models (src.) and the attack algorithms (algo.) are seen. All non-PGD attacks unseen attack algorithms.
88
4x
4x2
4x3
All
Number of Source Models
Figure 4: Clean and adversarial accuracy four PUBDEF models trained with (4 x 1), (4 x 2), 12 (4 x 3), and 24 (All) source models. "4 xm means m source models are chosen from each of the four groups
minimal drop in the clean accuracy is one of the most attractive properties of PUBDEF making it far more practical than white-box adversarial training.
PUBDEF generalizes well to unseen source models and attack algorithms. Our defense is trained against only the PGD attack and either four (CIFAR-I0) or three (CIFAR-100, ImageNet) source models. This amounts to four potential transfer attacks out of 264. Table 2 shows that PUBDEF generalizes incredibly well the 260 unseen attacks losing only 1.7%, 1.8%, and 7.6% in robustness across the three datasets. This result implies that these 264 transfer attacks may be much more "similar" than the community expected; see Section 7.3.
PUBDE: DEF should be trained with one source model from each of the four groups. Table 9 shows an ablation study, where we omit one of the four source models during training. We see that including at least one source model from each of the four groups is important for strong robustness. Including at least one Ey-robust and at least one corruption-robut model in the source set seems particularly important. Without them, adversarial accuracy drops by 28.5% 31.7%, respectively. We provide further evidence to support this finding in with a more sophisticated ablation study (Appendix A.5) that controls for the number of source models (Fig.

---

Published as a conference paper at ICLR 2024
Training PUBDEF against more source models is not necessarily better. Fig. 4 shows that adding more source models (8, 12, or 24) to PUBDEF increases the adversarial accuracy by only ~ %, and it also decreases clean accuracy by ~9% This suggests that our simple heuristic of selecting one source model per group is not only necessary but also sufficient for training PUBDEF.
A simple loss function suffices. For CIFAR-10 and CIFAR- 100, the RANDOM training loss achieves the best results (Table 10). For ImageNet, ALL is slightly better and only slightly worse than the best result (DyNAMICLOSs). We use these training methods in all of our evaluations.
7 DISCUSSION
7.1 ABLATION STUDIES
Random source model selection. We experiment with two random methods for choosing the source models for training PUBDEF. In three out of four cases, PUBDEF with the random selection method still outperforms the white-box adversarial training by roughly 10 p.p.. but in all cases, it is worse than our default selection method. This result lets us conclude that (i) our simple selection scheme in Section 5.2 is much more effective than random and (ii) all of the source model groups should be represented which is in line with Section 6.2. We refer to Appendix A. for more details.
Attacks from ensembles of the public source models. A more sophisticated adversary could use an ensemble of public models to generate a transfer attack, which has been shown improve the attack transferability (Liu et al., 2017; Gubri et al., 2022). We experiment with this approach by creating three ensembles* of four source models (one model from each group) and generate adversarial samples with all 11 attack algorithms. This results in 33 attack candidates, and then, we report accuracy under the best attack among these 33. We found that PUBDEF is robust against such an ensemble attack, and this ensemble attack is no better than an attack constructed from the best single-source model (91.7% vs 88.6% adversarial accuracy). We leave more sophisticated ensemble- based attacks (e.g., Chen et al. (2023b;a)) as future work.
7.2 ROBUSTNESS TO WHITE-BOX AND QUERY- QUERY-BASED BASED ATTACKS
As we design PUBDEF to specifically defend against transfer attacks, we rely system-level defenses for other types of attacks. Here, we measure the risk against such attacks. First, we find that PUBDEF is not robust against white-box attacks; PGD attack with sufficient steps reduces its accuracy to 0%. This is expected as white-box AT has been the only reliable de- fense against white-box attacks. Against both soft and hard-label query-based attacks, PUBDEF is also less robust compared the best boxte-box AT (Tables 12 and 13). However, once we apply an existing defense against query-based attacks (Qin et al., 2021), PUB- DEF becomes the most robust (Table 3). Specifically, we chose the standard deviation of the additive noise {0.01, 0.02] and followed the same evaluation pro- cedure as Qin et al. (2021). We evaluated the defense against the soft-label Square attack (Andriushchenko
Models
None
Sq- 100
Sq-
No Defense (a=0)
96.3
36.6
0.2
No Defense (a=0.01)
92.0
71.0
53.2
No Defense (a=0.02)
89.2
75.4
66.2
White-Bor AT (a=0)
85.3
77.9
67.3
White-Bor Box AT (o=0.01)
85.2
80.7
76.5
White-Bor Box AT (o=0.02)
85.0
81.7
78.9
PUBDEF (a=0) PUBDEF (a=0.01)
96.1
55.2
8.8
92.6
81.4
75.6
PUBDEF (a=0.02)
88.9
82.1
79.8
Table 3: Accuracy under Square attack with 100 and 1000 queries. Each model is com- bined with Qin et al. (2021) with of 0 (no noise), 0.01, and 0.02.
et al., 2020) with 100 and 1000 queries. The PUBDEF models have both higher clean accuracy (4-7 pp.) and adversarial accuracy (~1 pp.) than the white-box AT. Adding PUBDEF to this query-based defense boosts the robustness up to 24 pp. (from 66 to 80) against the 1000-query Square attack.
7.3 GENERALIZATION AND ADVERSARIAL SUBSPACE
Surprising generalization. In Section 6.2, PUBDEF demonstrates excellent robustness against a broad range of TAPM attacks, even the transfer attacks from a source model and/or an attack algorithm that were not trained against. We suspect that the surprising generalization of PUBDEF indicates a dow-dimenlional structure underlying transfer adversarial examples. We visualize our intuition in Appendix A.6.4. In this section, we investigate this phenomenon more closely.
"The ensemble averages the logits rather than averaging the losses or the softmax scores; in past. experience, we have found that all three choices yield similar performance.
8

---

Published as a conference paper at ICLR 2024
Generalization with one source model. We train four PUBDEF models each against only one source model (not one per group). The adversarial accuracy by groups in Fig. 6 shows an unexpected result: training against either an l2 or corruption source model alone provides above 55 % accuracy against the best attack. Furthermore, training against the & (resp. corruption) source model provides over 80% robustness against the & (resp. normal) group. This generalization effect does not necessarily hold in reverse (e.g., training against foo source model yields little robustness to the l2 group). Some source models are better than others to train PUBDEF with.
To verify the manifold hypothesis, we attempt to quantify it using two metrics: cosine similarity and principal component analysis (PCA). Fig. 5 shows the pairwise cosine similarity values among all 264 attacks aggregated by the four source model groups and averaged over all CIFAR-10 test samples. The cosine similarity is notably higher when comparing adversarial examples generated within the same group in contrast to those across two groups, especially for the lo and the & adversarial training groups (0.23 and 0.24 respectively). The cosine similarity is albeit lower in the normal and the corruption groups. PCA analysis also supports this observation, showing evi- dence for low-dimensional linear subspace for the & and the l2 groups. We defer the detailed discussion to Appendix A.6.4.
Narmal
Ey
4
Comupt.
Noral
0.04
0.02
0.03
0.04
0.02
0.23
0.17
0.95
0.03
0.17
0.24
0.08
Corni
0.05
Cosine Sim. of Adv. Peturbation
Figure Cosine sim. among pairs of adversarial perturba- tions by source model group.
7.4 PRACTICAL CONSIDERATIONS
PUBDEF is intended to stop a specific class of attacks that are particularly easy to mount, and that are not stopped by any reasonable systems-level defense. However, it has many limitations:
1. PUB is not robust to white-box attacks and is only suitable if model weights can be kept secret. If the model deployed users, then attackers can reverse-engineer it to find the weights and mount a white-box attack (Liang et al., 2016; Tencent Keen Security Lab, 2019). sufficiently dedicated and well-financed attacker can likely train their own surrogate model, e.g.. by somehow assembling a suitable training set and paying annotators or using the target model to label it, then mount transfer attack from this private surrogate, potentially bypassing our defense. 3. We only defend against lox -oom- bounded attacks. Many researchers have argued persuasively that we also need to consider broader attack strategies (Gilmer et al., 2018; Kaufmann et al., 2023), which we have not examined in this work.
Despite these limitations, PUBDEF has several practical benefits: (1) Minimal drop in clean accuracy: The robustness gain of PUBDEF is (almost) free! This makes it almost ready to deploy in the real world. (2) Low training cost: Adversarial training takes ~2 x longer due to the adversarial example generation inside of the training loop. In contrast, PUBDEF training is much faster, as transfer attacks can be pre-generated prior to the training, only need to be done once, and can be done in parallel.
More precisely, the training time of PUBDEF is approximately 167 0.2NT where T is the time for one epoch of white-box adversarial training and is the number of epochs of training. In comparison, adversarial training takes NT' time. For N 50 epochs, PUBDEF' training time is 267' 507' for adversarial training. The first term in the formula (167) is a one-time cost for generating all the transfer attacks used in training: 4 source models x 4 instances per training sample x 10 PGD steps 167.. Regardless of how many PubDef models are trained or how many hyperparameters are swept, this step has to be done once. The second term is the cost of the training loop which depends on the exact loss function used (Section 5.1). For the "Random" loss function, the cost is 0.27 per epoch.
CONCLUSION
In this paper, we propose pragmatic method for achieving much robustness as possible, in situations where any more than minimal decrease in clean accuracy is unacceptable. We identify transfer attacks from public source models (TAPM) as particularly important class of adversarial attacks, and we devise a new method, PUBDEF, for defending against it. Putting everything together yields plausible defense against adversarial examples by aligning ML defenses with the most feasible attacks in practice that existing systems-level defenses cannot prevent. We hope other researchers will build on these ideas achieve even stronger protections against adversarial examples.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported in part by funds provided by the National Science Foundation (under grant 2229876), the KACST-UCB Center for Secure Computing, the Department of Homeland Security, IBM, the Noyce Foundation, Google, Open Philanthropy. and the Center for Al Safety Compute Cluster. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors.
REFERENCES
Sravanti Addepalli, Samyak Jain, and Venkatesh Babu Radhakrishnan. Efficient and effective augmentation strategy for adversarial training. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022a. URL https: wpw.neeieeeeeeeeeeeeeeeeeeeeeeee net / forum? woeee//oooddddokSIIIIII 1ididdd1130kkpphmm
Sravanti Addepalli, Samyak Jain, Gaurang Sriramanan, and R. Venkatesh Babu. Scaling adversarial training to large perturbation bounds. In European Conference on Computer Vision, 2022b. doi:
10.48550arxii.2210009555 6
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: A query-efficient black-box adversarial attack via random search. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision - ECCV 2020, volume 12368, pp. 484-501, Cham, 2020. Springer International Publishing. ISBN 978-3-030-5555 978-3-030-58592-11 doi: com/ http/10.1100/9/7
1_29. URL htttht:sp//nnnii http:///iinnn.iiinneeeeeeeee springer. 29. 8,8
Giovanni Apruzzese, Hyrum S Anderson, Savino Dambra, David Freeman, Fabio Pierazzi, and Kevin Roundy. "Real attackers don't Io compute gradients": Bridging the gap between adversarial ML research and practice. 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML.), pp. 339-364. IEEE, 2023.
Alexandre Araujo, Laurent Meunier, Rafael Pinot, and Benjamin Negrevergne. Advocating for multiple defense strategies against adversarial examples. In Irena Koprinska, Michael Kamp, Annalisa Appice, Corrado Loglisci, Luiza Antonie, Albrecht Zimmermann, Riccardo Guidotti, Özlem Özgöbek, Rita P. Ribeiro, Ricard Gavaldà, João Gama, Linara Adilova, Yamuna Krishna- murthy, Pedro M. Ferreira, Donato Malerba, Ibéria Medeiros, Michelangelo Ceci, Giuseppe Manco, Elio Masciari, Zbigniew W. Ras, Peter Christen, Eirini Ntoutsi, Erich Schubert, Arthur Zimek, Anna Monreale, Przemyslaw Biecek, Salvatore Rinzivillo, Benjamin Kille, Andreas Lommatzsch, and Jon Atle Gulla (eds.), ECML PKDD 2020 Workshops, pp. 165-177, Cham, 2020. Springer International Publishing. ISBN 978-3-030-66565-3.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEÎT: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022. URL https:://peereeieeeeee net / ftre//ooummmm -it.mmddd-pphhhhaeee
6
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šmdiić Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Hendrik Blockeel, Kristian Kersting, Siegfried Nijssen, and Filip Zelezný (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 387-402, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. ISBN 378-3-642-40999-3. 1,
Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models. International Conference on Learning Representations, 2018 URL https:
3
Bin Chen, Jia-Li Yin, Shukai Chen, Bo-Hao Chen, and Ximeng Liu. An adaptive model ensemble adversarial attack for boosting adversarial transferability. August 2023a. URL http://arxiv org/abs. 2308 02897.
Huanran Chen, Yichi Zhang, Yinpeng Dong, and Jum Zhu. Rethinking model ensemble in transfer- based adversarial attacks, March 2023b. URL http:://arxi wogg/brssss..................... /b333303. 091 05. 8
10

---

Published as a conference paper at ICLR 2024
Jianbo Chen, Michael I. Jordan, and Martin J. Wainwright. HopSkipJumpAttack query-efficieet decision-baace attack. vrXiv:/994.022 [cs, math, stat], April 2020a. URL http:: org/abs/ 4904. 02144. 24
Steven Chen, Nicholas Carlini, and David Wagner. Stateful detection of black-box adversarial attacks. In Proceedings of the 1st ACM Workshop on Security and Privacy on Artificial Intelligence, SPAI '20, pp. 30-39, New York. NY, USA, 2020b. Association for Computing Machinery. ISBN
978-1-4503-7611-2, doi: 10. 00.1145533880033341009999 URL 3385003. 000000334199200 3410925.
Yaofo Chen. Chenyaofoffyoorchcciaa----m-ddde sape cheeyaacce/pyttorch--laaaaa
September 2023. URL httth::/ggthhbb.. com/
15
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 2206-2216. PMLR, July 2020. URL http: http::/ppprooeeiiii w119/croce22 html. 15
mlr mlr.press!
Francesco Croce and Matthias Hein. Adversarial robustness against multiple $I_pS-threat mod- els at the price of one and how to quickly fine-tune robust models another threat model. arXiv:2105.12500 [cs], May 2021. URL httpt://rrxiiii org/ abs/2105 12508. 24
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. RobustBench: standardized adversarial robustness benchmark. Technical report, 2020. URL http:
http:/axiiv.iiiiaaaa/a /
/2000. 09670. O.
Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. Why do adversarial attacks transfer? Explaining transferabil- ity of evasion and poisoning attacks. In Proceedings of the 28th USENIX Conference Security Symposium, SEC' 19, pp. 321-338, USA, 2019. USENIX Association. ISBN 979-1-933933.--6-9 3
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, March 2018. URL http: http:::///aaxi..rraaaaaaa///////
06081. 1. 3, 15
Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses transferable adversarial ex- amples by translation-invariant attacks. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2019. 15
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. URL
net forum? i Mel-yicbbl criccbFNNC CNTTy 6
Hugging Face. Models Hugging Face, July 2023. URLhhtipss//hcuiiiggffee http:///nnn.accccccoooomeeeeeeee 6. 15
Justin Gilmer, Ryan P. Adams, Ian Goodfellow, David Andersen, and George E. Dahl. Motivating the rules of the game for adversarial example research, 2018. arXiv:1807, 887.06732. 1,9
Ian Goodfellow. Defense against the dark arts: An overview of adversarial example security research and future research directions, 2018. arXiv:1806.04166 1
Ian Goodfellow. research agenda: Dynamic models to defend against correlated attacks, 2019. arXiv:1903.0622 3
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial
examples. In International Conference on Learning Representations, 2015. URL http://
arxiv. wwxii.org/abb//4441 6572. 1,4
11

---

Published as a conference paper at ICLR 2024
Martin Gubri, Maxime Cordy, Mike Papadakis, and Yves Le Traon. Efficient and transferable adversarial examples from bayesian neural networks. In The 38th Conference on Uncertainty in Artificial Intelligence, 2022. URL
6B8
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ML safety, 2022. arXiv:2109.13916.
Qian Huang, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhancing adversarial example transferability with an intermediate level attack. Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019. 3
Andrew llyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2137-2146. PMLR, July 2018. URL https::tp///cc htttt:::/pppocceediiiee mlr . v80/ilyas 8a httm. 3
Max Kaufmann, Daniel Kang, Yi Sun, Steven Basart, Xuwang Yin, Mantas Mazeika, Akul Arora, Adam Dziedzic, Franziska Boenisch, Tom Brown, Jacob Steinhardt, and Dan Hendrycks. Testing robustness against unforeseen adversaries, 2023. arXiv:1908, 00...8..... 9
Huiying Li, Shawn Shan, Emily Wenger, Jiayun Zhang, Haitao Zheng, and Ben Y. Zhao. Blacklight: Defending black-box adversarial attacks deep neural networks. arXiv:2006. 14042 [cs], June 2020. URL pattp://areee
org/ abs 2006 6. 14042. 3
Bin Liang, Miaoqiang Su, Wei You, Wenchang Shi, and Gang Yang. Cracking classifiers for evasion: case study on the google's 's phishing pages filter. In Proceedings of the 25th International Conference World Wide Web, WWW 16, pp. 345-356, Republic and Canton of Geneva, CHE, April 2016. International World Wide Web Conferences Steering Committee, ISBN 978-1-4503- 4143-1. doi: 10.1 10.1145/2872427 2883060. 9
URL
111g//10..145/287227
Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E. Hopcroft. Nesterov accelerated gradient and scale invariance for adversarial attacks. International Conference on Learning Representations, 2020. URL https:: http::ss///peeevvee.... net. /ttp:///o-o1-/////YYO 3,
15
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In Proceedings of 5th International Conference on Learning Representations, 2017. 3,8
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant adversarial attacks. In International Conference on Learn- ing Representations, 2018. URL https:// /ttp::/:////////e//ee/ee/eee net/ forum? dd h 1, 3, 4, 6, 15
Thibault Maho, Seyed-Mohsen Moosavi-Dezfooli,, and Teddy Furon. How to choose your best allies for a transferable attack?, April 2023. URL http: //taxi///vvvv/////a wwr.ii.ooor/aaaeo httg/abs//// 2234 02312, 6
Laurent Meunier, Meyer Scetbon, Rafael Pinot, Jamal Atif, and Yann Chevaleyre. Mixed nash equilibria in the adversarial examples game, February 2021. URL http:///aaax 2102.06 06005. 4
John Nash. Non-cooperative games, Annals of Mathematics, 54(2):286-29 1951, ISSN 0003486X.
doi: 10.2307/1969529, URL http:/// h/tp:///wwwwww. jstor. org/
Tencent Keen Security Lab. Experimental security research of Tesla autopilot, 2019. 9
12

---

Published as a conference paper at ICLR 2024
Nicolas Papernot. Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: From phenomena to black-box attacks using adversarial samples. arXiv:1605.0727) [cs], May 2016.
URL http::/aaxiv..rr//////// abs/ /1055/666... 07277.
Huy Phan. Huyymphaw/PYToochcCraaa mwyoppaan//yTDrrcccccPPPP. 6, 15
September 2023. URL https://github.. com/
Zeyu Qin, Yanbo Fan, Hongyuan Zha, and Baoyuan Wu. Random noise defense against query-based black-box attacks. In Thirty- Fifth Conference on Neural Information Processing Systems, 2021.
URL httptttpp////ttttte httpa:s///ppeeevveeeee net/ /ete/forrmtt forum? 1d=2PSD4x2c663 8, 22
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8748-8763. PMLR, July 2021. URL https: http://poooeeeeeeee/e/e// www. www............mmmmmmmmmmmmm v1 v9 radford21e html. 6
Ali Rahmati, Seyed-Mohsen Mloosavi-ezzfou Pascal Frossard, and Huaiyu Dai. GeoDA: A geometric framework for black-box adversarial attacks, arXiv:2003.0644 064660 [cs], March 2020. URL
http:///rxtt: org/ abs / 22003 064 68. 24
Ethan Rathbun, Kaleel Mahmood, Sohaib Ahmad, Caiwen Ding, and Marten van Dijk. Game theoretic mixed experts for combinational adversarial machine learning. November 2022. URL http::/arxe org/ abs///b//// 2211 14669. 4
Tim Roughgarden. CS261: A second course in algorithms lecture #10: The minimax theorem and algorithms for linear programming. Technical report, Stanford University, February 2016. URL https://theor:: stanford. edu/~tim/ tttttiim/ww wl 1/1/100..11////////////11//11111 pdf. 18
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust ImageNet models transfer better?
In Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.). Advances in Neural In- formation Processing Systems, volume 33, pp. 3533-3545, Curran Associates, Inc., 2020. URL FLL1:2255Tc000222228888872622222
neurips wwwrips.cc/paper.... setaaperffileepppeeeeeeeee
pdf. 6
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014. URL http: pettt:://areee //arxiavvvvvv/////////a//// org/ abs /121.1111222 61.9.
Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Bonch, and Patrick McDaniel. The space of transferable adversarial examples. 1004.v/704.0335 [cs, stat], May 2017. URL http:
//arxivv aaa.ii..rgg/abs......oa.a 704. 03453. 3,24 24
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Bonch, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learn- ing Representations, 2018. URL https: httttt:t/opeereeeeeeeeee /wwpervveeeeeeeceooo/epe. net/ forum? www.miiieeeeeeeeeeeemmmme RZ.
16
Asher Trockman and J Zico Kolter. Patches are all you need? Transactions on Machine Learn- ing Research, 2023. ISSN 2835-8856. URL https: https/////ppeeeeie..eo/mm net forum?id=
rAnB7JSMXL. 6
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Ro- bustness may be at odds with accuracy. In International Conference on Leurning Representations, 2019. URL https::/
net worrmmmid-SSxxx3000e3
J. Neumann. Zur theorie der gesellsehaftsspiele. Mathematische Annalen, 100(1):295-33) December 1928. ISSN 1432-1807. doi: 10.1007/BFF11488847,, URL https: 1007 448447. 18
13

---

Published as a conference paper at ICLR 2024
Jan van den Brand. A deterministic linear program solver in current matrix multiplication time. In Proceedings of the Thirty-Fitt Anmual ACM-SIAM Symposium on Discrete Algorithms, SODA '20. pp. 259-278, USA, 2020. Society for Industrial and Applied Mathematics. 18
Xiaosen Wang, Xuanran He, Jingdong Wang, and Kun He. Admix: Enhancing the transferability of adversarial attacks. arXiv:2102.004: [cs], January 2021a. URL 2102. 00436. 3, 15
org/abs
Xiaosen Wang, Jiadong Lin, Han Hu, Jingdong Wang. and Kun He. Boosting adversarial
transferability through enhanced momentum. arXiv:2103. 10609 [cs], March 2021b. URL
http://arxiv. wwriiiiorrr/aaee orgg/bs/2102 10609. 3, 15
Ross Wightman. PyTorch image models. GitHub, 2019. URL https://github..co com/ httphh/aaappptooooooaaa-eeee-eee
6, 15
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang. Zhou Ren, and Alan L. Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019. 3,
Huanrui Yang, Jingyang Zhang, Hongliang Dong, Nathan Inkawhich, Andrew Gard- ner, Andrew Touchet, Wesley Wilkes, Heath Berry, and Hai Li. DVERGE: Di- versifying vulnerabilities for enhanced robust generation of ensembles. In Advances in Neural Information Processing Systems, volume 33, pp. 5505-5515. Curran Asso- ciates, Inc., 2020. URL https: httttpa://proceeedngge neuriiiiisssnnn.nnoooeee..... neurips cc httpp/aper/22//aaaaaaae http:eeebbbeeeaacccccaaoeeeeeeee Abstaaeeeeaeeeeaaaaaataaaaaaaaae html. 3,6, 16
Zhuolin Yang, Linyi Li, Xiaojun Xu, Shiliang Zuo, Qian Chen, Pan Zhou, Benjamin Rubinstein, Ce Zhang, and Bo Li. TRS: Transferability reduced ensemble via promoting gradient diver- sity and model smoothness. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 17642-17655. Curran Associates, Inc., 2021. URL htwps:ttp//cccc http::::///pp/pepppeeeee neurips. /ttp:///220///11e//993300000 18
cbdd- paper.pdf. 3, 6,
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. CutMix: Regularization strategy train strong classifiers with localizable features. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6022-6031, Seoul, Korea (South), October 2019, IEEE. ISBN
doi: 10.1 10.10///CCC2221900000112 URL 16
ieee org/
Jianping Zhang, Weibin Wu, Jen-tse Huang, Yizhan Huang, Wenxuan Wang, Yuxin Su, and Michael R. Lyu. Improving adversarial transferability via neuron attribution-based attacks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 14993-15002. June 2022. 3, 15
Zhengyu Zhao, Hanwei Zhang, Renjue Li, Ronan Sicre, Laurent Amsaleg, and Michael Backes. Towards good practices in evaluating transfer adversarial attacks, November 2022. URL http: /arxiiv. org.org/aaao abs/ /211. 09565. 3, 15
14

---

Published as a conference paper at ICLR 2024
Attack Algorithms
Momentum Augmentation
Feature-Level Non-C Loss
PGD (Madry at,, 2018)
M-PGD (Dong et al., 2018)
Pregradient (Wang et al., 2021b)
/
X
X
Diverse input (DI) (Xie et al., 2019)
X
V
X
invariant (TT) (Dong et al., 2019)
X
V
X
Admix (Wang al., 2021a)
V
NA (Zhang et al., 2022)
/
X
NI-SI-TI-DIM (Zhao ct al., 2022)
X
X
NI-Admix-TI-DDh (Zhao et al., 2022)
AutoPGD-DLR (Croce & Hein, 2020)
AutoPGD-CE (Croce & Hein, 2020)
Table 4: All transfer attack algorithms we experiment with in this paper along with the types of techniques they use improve attack transferability.
A APPENDIX
A.1 DETAILED EXPERIMENT SETUP
A.1.1 FULL LIST OF PUBLIC SOURCE MODELS
Table 5, Table 6, and Table 7 compile all the source models we use in our experiments for CIFAR-10, CIFAR-100, and ImageNet, respectively. For the normal group, model names that begin with t timm/ foon and huyvnphan/ are from the timm library (Wightman, 2019), Chen (2023), and Phan (2023), respectively. Those that ends with _local models that train ourselves locally. These include ConvMixer models and several models on CIFAR- 1-00 where the public models are more difficult find. The remaining models from Hugging Face's model repository (Face, 2023).
There is no &e group for CIFAR-106 and ImageNet because ROBUSTBENCK does not host any f-- adversarially trained models apart from the ones trained CIFAR-10 10. As mentioned in Section 6.1, the models are all chosen to have as much diversity as possible in order to ensure that the defenses evaluated on a wide range of different transfer attacks.
A.1.2 FULL LIST OF TRANSFER ATTACK ALGORITIIMS
All of the 11 transfer attack algorithms experiment with in this paper are listed in Table 4. Two other attacks that we do not experiment with individually but as a part of the other two combined attacks (NI-SI-TI-DIM, NI-Admix- TI-DIM) are the ccale-ivaariant (SI) (Lin et al., 2020) and the Nesterov (NI) (Lin et al., 2020) attacks. We either take or adapt the attack implementation from Zhao al. (2022) (https: : / / thub. com/ / Zhengyu Zhao/ TTrans fe rat ta ck Eval). Then, check our implementation against the official implementation of the respective attacks.
attack implementation. NA attack originally designed to the interme- diate features of ResNet-like models so it lacks implementation or instruction for choosing the layer for other architectures. Therefore, one-fourth of all the layers). For the same reason, we also omit NA attack on the source model Di w.f.eendefee22222. 21:1 ining LRR CARD Deck as it an ensemble of multiple architectures.
Notes
decided to pick an early layer in the network (about
A.1.3 PUBDEF TRAINING
All CIFAR-10/100 models are trained for 200 epochs with a learning rate of 0.1, weight decay of x 10-4, and batch size of 2048. ImageNet models are trained for 50 epochs with a learning rate of 0. weight decay of 1 x 10-4 and a batch size of 512. We have also swept other choices of hyperparameters (learning rate E {0.2, 0.1, 0.05] and weight decay {1 x 10 -4 10-4,3 x 10 10-4) (0-4) but did not find them affect the model's performance significantly. Overall PUBDEF is not
sensitive
to hyperparameter choices anymore than a standard training and is less sensitive than white-box adversarial training. All of the models trained Nvidia A 110 GPUs.
15

---

Published as a conference paper at ICLR 2024
Groups
Model Names
jafoh//#ki1--Ffnetuneed
chenyaofo/zeeene220
chenyaofo/vgglll-ni
chenysaoff//imbb3leeeeex22-----
Normal
clenxxooto/Bhuufleeeeee-==000
chooyyaofo/ccpvygg-a000
nuyyaphaan//Gnschhtttll
huyxopharr/iceeetion-33
convmixer_local
clip
leng22033lstterr_NXX70--66
ku2022kkaplorring_.ww2228--
Loo
1abeneslett12022hhhhg__...11
Sebwag202CHydra
Wangg022BBetter__wnn-----16
aupustiiiccCC@aeerrria..31...... Rony2019Deccouplingg
D.feaadberczz20.ii.nnngggg...d. Deck
xkirer//2llfffccc..enssss.tttg
Corruption
fendayctsee02AAngiiis__eeaaa
Modas22021PR.IMEReseeee
Table 5: All public source models for CIFAR-10.
Data augmentation. Unlike most training procedures, there are two separate places where data augmentation can be applied when training PUBDEF: the first is when generating the transfer attacks and the second is afterward when training the model with those attacks. For the standard data augmentation (pad and then crop) in both places. For ImageNet, we also use the standard augmentation including random resized crop, horizontal flip, and color-jitter in both of the steps. However, in the second step, we make sure that the resizing does not make the images too small as they would have already undergone the resizing once before. For all of the datasets, also add CutMix (Yun et al., 2019) to the second step. No extra data or generated data (e.g., from generative models) are used.
Number of transfer attack instances. For each pair of (S, we generate multiple attacked versions of each training image (by random initialization) to prevent the model from overfitting to specific perturbation. Unless stated otherwise, we use four versions in total, one of which is randomly sampled at each training iteration. That said, using fewer versions does not significantly affect the performance of our defense. With version, the adversarial accuracy drops by 0.5% CIFAR-10 10 and 0.2% on ImageNet. Using cight versions increases by 1.2% on CIFAR-10 10. The clean accuracy seems unaffected by the number of versions. This suggests that when computation storage is bottleneck, we can use only one attack version with minimal loss of robustness.
A.2 ADDITIONAL DESCRIPTION OF THE BASELINES
EAT (Tramèr al., 2018) is an ensemble training technique motivated by the limitations of white-box FGSM training due to their propensity for falling into degenerate global minima that consist of weak perturbations, resulting in weak transfer robustness. EAT is claimed to specifically increase robustness against black-box-based" attacks, where members of an ensemble that serve to augment the data by generating adversarial examples to attack a singular member model.
pre-trained models
DVERGE (Yang et al., 2020) is robust ensemble training technique that leverages cross-adversarial training. In this methodology, ensemble members are subject to training on adversarial examples generated from other models present within the same ensemble. The distinction between DVERGE
16

---

Published as a conference paper at ICLR 2024
Groups
Model Names
aazaspanna//ssinnnninnnneeeeeeee 00
chenyaofo/reeseet20
cheeyyaooovvggll-tt
clexyyaofo/sshafflleeeevv2-----2
Normal
chenyaofo/repvyg-aa0
densonet121_local
sonst18_local
inceptionnvv__lcaal
convmixer_local clip
Nang2203Jettter__NN--00-1
Cui
Hai2023 mmpoo//nnpoo
loc
X 01a20221AS-AT_34_22 Rade2021Keepeer_R1 R18 ddpm
Dabenedet:220221ignt
Diffeengarrerr222liiiiiinng. LRRD Deck
Modaa22021132MEEeessee11.
Corruption
Diffenderrerr200iiiiinngg.....i
Table 6: All public source models for CIFAR-100.
Groups
Model Names
microsoft/reenet-550
googlee/iit-bbee-patchh-----
faceboot/ccommmsstt--n----2
cimm/incceptionn__33
Normal
t L.tm:/reoknet_1000oaaa__l1k
t _imm/Arrett_111.n_________..
tim./oggl1.tv_inik
Mive022ccompreehenniieeees-----
foo
Salman2222DD__50_2
Salman220CDo_R550
T
Tlan2022Deeper_Deiiin
Briichaon22(22Mooiixxxxxxxeeen
Corruption
Hendkycks2020kany
Hendrycks2202aaugMi
Geirhos2018
Table 7: All public source models for ImageNet.
and EAT or adversarial training is the "distillation" loss that encourages a large difference between the intermediate features between models in the ensemble.
17

---

Published as a conference paper at ICLR 2024
TRS (Yang et al., 2021) claims to be improvement over DVERGE. It removes the extra loss term from DVERGE and incorporates fz-based regularization into the loss objective function to enhance model smoothness. This regularization penalizes both the cosine similarity between the gradients of all pairs of models in the ensemble and the Euclidean norm of the gradients. Both TRS and DVERGE evaluated against transfer attacks where they show superior robustness compared EAT as well as single-model adversarial training.
A.3 SIMPLE GAME'S EQUILIBRIUM
The simple game (Section 4.1) can be formulated as the following von Neumann's minimax theorem with bilinear objective function. We refer the interested readers to Roughgarden (2016) for the full derivation.
Theorem 1 (von Neumann's minimax theorem with bilinear function (v. Neumann, 1928)). Given a "simple game" " described above and its payoff matrix R, there exists a mixed strategy T for the attacker and a mixed strategy T for the defender such that
rD
max
min
TD BDAAA=
min TAEA ADEA
max TD R
(5)
ADEA =
where s0-11 is the (sa- 1)-dimensional probability simplex.
This problem can be solved like any other linear program with polynomial time algorithms (van den Brand, 2020).
A.4 PUBDEF'S LOss FUNCTION
Below are all the loss functions we have experimented with when training PUBDEF models. Their comparison is reported in Table 10.
1. Fixed: Here, all n.'s s are fixed to 1/sa. We experiment with two very similar loss functions: (I) ALL: The model trained on all transfer attacks (pairs of (S, A)) simultaneously. This is exactly the same as Eq. (4). (2) RANDOM: Randomly sample one pair of (S, A); at each training iteration.
This scheme modifies ALL by taking, at each iteration of training, the top pairs of (S, A); that maximize the loss on the current defense model being trained. Effectively, in each batch, we attack the current model weights with all s attacks, choose the most effective attacks, and set their n. 's to 1 1/k and the other n; 's to O. For k = 1, this is a minimax problem similar to adversarial training, but the maximization over the choice of transfer attacks instead of the perturbation:
Top-k:
arg min Emyy L(fa(z), v) + max L (fe (Fadv,)) 3) :E(s.a]
(6)
0
3. Dynamic weights: This method can be considered a smooth version of the top-k. Instead of setting each n, to either or O, we dynamically adjust it in proportion to the overall loss or error rate of the attack (S, A);- We call these methods DYNAMICLOSS and DYNAMICACC respectively. We use an exponential moving average μ with a decaying factor a to estimate the loss and the accuracy:
14 = (1 )44 + aL (fo (Iadv,)) .8) 1 [+1 (1 a)44 + allEn v [1 {fe (Zadv.)) = yH
(DyNAMICLOss)
(7)
(DYNAMICACC)
(8)
T - M4
[4g
(Normalize to [0, 1))
(9)
We can normalize the n;'s 's by their sum because both the loss and the error rate are non-negative.
A.5 DEFENDER'S 's SOURCE MODEL SELECTION
In this section, specifically dive deeper into the source model selection heuristic for training PUBDEF. We will investigate various effects on the adversarial accuracy under the TAPM transfer attacks when including/excluding certain models.
Single source model. We begin by first choosing only one source model from each group. The results reveal that training PUB DEF with source model is insufficient for building good defense. In
18

---

Published as a conference paper at ICLR 2024
Normal
E
Corrupt.
Normal
76.2
68.8
41.9
38.0
38.0
Defender's Source Model
80
39.0
74.0
43.1
18.5
18.5
60
b
58.2
84.5
63.2
Corrupt.
82.5
69.2
55.9
79.7
55.9
40
All
88.6
90.4
91.5
90.6
88.6
20
Attacker's Source Model Group
Figure 6: Adversarial accuracy of PUBDEF when trained against only one source model on CIFAR-10 We select one source model from each group, and the adversarial accuracy is also categorized by the attacker's 's source model groups.
Normal
fx
Corrupt
All
Normal
Eg
f
Coruut. All
Defender's Source Model Group
90
Defender's Source Model Group
90
All Normal
85.2
87.8
88.6
87.9
83.8
Normal
83.4
85.5
89.7
68.2
83.4
85
85
All
fx
86.7
84.3
87.9
88..3
84.3
All -0
85.3
80.6
86.1
88.1
80.6
80
80
-6,-
86.8
75.9
60.8
82.9
60.8
75
All Eg-
84.1
76.5
60.1
83.0
60.1
75
-70
Commpt..
85.4
82.0
87.9
76.2
76.2
70
All Corrupt.
H2.2
75..5
68.4
56.9
56.9
-65
All
88.6
90.4
91.5
90.6
88.6
All
88.6
90.4
91.5
90.6
88.6
60
Attacker's Source Model Group
Allacker's Source Model Group
Figure (CIFAR-10) Adversarial accuracy grouped by the attacker's source models (columns). Each row corresponds to a PUBDEF model with one source model from each group removed, but unlike Fig. 7, another source model from different group is added keep the total number of source models constant (four).
Figure 7: (CIFAR-10) Adversarial accuracy grouped by the attacker's source models (columns). Each row corresponds to PUBDEF model with one source model from each group removed except for the last row where all are in-
cluded.
other words, regardless of which source model is chosen, it performs far worse than the default option include four source models ("All"). That said, the model trained against singular source model from the by or the corruption group exhibits surprisingly high resilience even to the best transfer attack, achieving the adversarial accuracy of over 55%.
Excluding one source model group. Fig. 7 is an extension to Table 9 which shows the adversarial accuracy from the best attack from each source model group in addition the overall best. Here, by looking at the diagonal entries, we can see that removing source model from one group during PUBDEF training reduces the robustness against the transfer attack from the same group. Note that the overall best transfer attack always comes from the missing group (i.e., the diagonal entries are equal to the last column). The degradation is minor for the normal and the foo groups but very large for the & and the corruption groups This implies that for certain reasons, adversarial examples generated on source models from the normal and the &xe groups are "subsets" the ones from the other two groups because this generalization phenomenon happens only in one direction but not both.
19

---

Published as a conference paper at ICLR 2024
CIFAR-10
CIFAR-
ImageNet
Defenses
Clean
Adv.
Clean
Adv.
Clean
Adv.
Best white-box adv. training
85.3
68.8
68.8
32.8
63.0
36.2
DUB (all random)
95.2 (+9.9)
62.9 (-5.9)
75.4 (+666)
44.0 (+11.))
DUB (random per group)
95.0 (-9.7)
77.3 (+8.5)
75.3 (4-.5)
44.2 (+11,4)
PUBDEF (default)
96.1 (4-10.8)
88.6 (4 19.8)
76.2 (4-74)
50.8 (4-18.0
78.5 (415.5)
62.3 (4-26.)
Table 8: Clean and adversarial accuracy of PUBDEF against the best attack under the TAPM threat model. As an ablation study, we compare the baseline as well as our default PUB to PUBDEF when the defender's source models are randomly selected.
We conduct another experiment that controls for the fact that the number of source models used to train PUBDEF reduces from four (the default setting where all source models used) to three in the earlier experiments. The results are shown in Fig. 8. Here, in addition to removing one source model, we add another source model from different group the pool. For example, in the first row, the normal model is removed, and then we add a model from each of the remaining groups to create three differ- ent PUBDEF models (ffoo: loc, E,, corrupt.), {foo E2, f2, corrupt.) J. {foo, l2, corrupt., corrupt.)) }). We then compute the adversarial accuracy of all three new models and report the best one. The conclusion above remains unchanged, but the gap in the adversarial accuracy becomes particularly smaller for the corruption group.
A. RANDOM SOURCE MODEL SELECTION
In this section, we provide detailed experiments and results on the random source model selection method for training PUBDEF. We train two new sets of 30 PUBDEF models each where the source models are randomly sampled in two different ways: (A) by model and (B) by group. When randomly sampling by model, all source models have an equal probability of being selected. This means that more normal models may be selected as they are over-represented (12 vs. 4/6 for the other groups). When sampling by group, we uniformly sample one out of four groups and then uniformly sample one source model from the group. This accounts for the difference in group sizes. The total number of source models is still fixed at four for CIFAR-10 and three for CIFAR-100.
Table 8 compares the two random model selection methods to the default one. Here, "all random" corresponds to the sampling by group, and "random per group" the by-group sampling but without replacement so all groups each group (i.c., set B) results a drop in the adversarial accuracy by 11% on CIFAR-10 and 7% on CIFAR-100 average (Appendix A.5). However, they are still 9% and 11 % higher than the baseline. On the other hand, set A performs worse than the baseline by 6% on CIFAR- 10 and similarly to set B on CIFAR- 100. The discrepancy between the two datasets on A can be explained by the fact that the probability of choosing exactly one model from each group is only 7% on CIFAR-10 but 21 % on CIFAR-100.
represented by one model. Choosing random source model from
Next, we use these two sampling methods to perform additional ablation studies. By randomly sampling the source models, we can make a general statement about the effect of each source model group in PUBDEF independent of the exact choice of the source models.
Importance of each source model group. We train 64 PUBDEF models with one, two, three, and four source model groups (16 each). The source models are sampled by groups without replacement. Fig. 9a and Fig. 9b show the adversarial accuracy by the source model group similarly to Fig. but with the random selection described above. Fig. 9a categorizes PUBDEF by which source model group excluded whereas Fig. 9b categorizes by which source model group is included. The result from Fig. 9a very much agrees with Fig. 7 and Fig. 8. Fig. 9b also corroborates with this observation by confirming that PUBDEF models that include the & group outperforms the others quite significantly.
Number of total source models. Fig. 10 displays both the clean and the adversarial accuracy when PU is trained with different numbers of source models. This is similar to Fig. 4 in the main paper as well Fig. 6, but the only difference is that the source models here are randomly selected. Similar to the above result, the random selection allows us to make a more generalized statement about the source model groups without being strictly tied to the exact choice of the source models.
20

---

Published as a conference paper at ICLR 2024
Defender's Source Model Group
Normal
Esc
Corrupt.
All
Defender's Source Model Group
Normal
Ex
Corrupt.
All
80
All Normal
65.1
79.9
70.0
59.4
53.4
+Normal
79.8
76.1
64.5
65.8
60.3
75
70
All p0
78.6
76.5
69.7
71.4
63.6
+
70.4
78.4
62.9
55.1
51.7
70
65
All
---
70.6
73.3
50.1
50.1
42.6
+f2"
78.3
81.1
81.0
75.6
71.5
60
All
Comuut.
67.4
78.4
65.00
51.6
49.9
50
+Corrupt.
80.9
76.6
67.6
73.8
61.7
55
Attacker's Source Model Group
Attacker's Source Model Group
(a) Excluded
(b) Included
Figure 9: (CIFAR-10 10) Adversarial accuracy of PUBDEF with the random source model selection (by group). Similar Fig. 7, transfer attacks by the source model groups. The left plot categorizes PUBDEF by the excluded group while the right plot categorizes by the included group.
100
95.1
95.0
77.3
80
66.8
57.0
Accuracy
60
39.5
40
20
E
Clean Acc.
Adv. Acc.
1x1
2x1
3x1
4x1
Number of (Random) Source Models
Figure 10: Clean and adversarial accuracy under the best attack defense with varying numbers of randomly chosen source models for training PUBDEF. x denotes that we (randomly) pick one model from each of the k categories. The error bars denote (-distribution 95%-conffidenc interval.
This plot demonstrates that the adversarial accuracy against the best transfer attack improves as PUBDEF is trained against more source models, but the return is diminishing. On the other hand, the clean accuracy is completely unaffected and varies in very narrow range.
A.6 ADDITIONAL RESULTS
A.6.1 COMPARISON TO OTHER ENSEMBLE BASELINES
We have described the ensemble defenses including DVERGE and TRS in Section and Ap- pendix A.2. Here, we will introduce another defense we built on top of TRS to make it even more suitable as a defense against transfer attacks. We call this scheme "Frozen TRS." The main idea behind TRS as well as DVERGE is to make the models in the ensemble diverse and different from one another. However, in the TAPM threat model, we want make our defense "different" from the public models as much possible. With this in mind, we apply the regularization terms from TRS to a set of models comprised both the defense we want train and the public source models we want to train against. But unlike TRS, are only training one defense model while holding all the public models fixed (hence "frozen").
Table 11 compares all the ensemble-based defenses including the previously described Frozen TRS. evident that all the ensembles are outperformed by the single-model white-box adversarial training as well PUBDEF. Frozen TRS performs better than the normal TRS but worse than the adversarially trained TRS. We believe that TRS' regularization is effective for diversifying the models' gradient directions, but it takes indirect path improve the robustness against transfer
21

---

Published as a conference paper at ICLR 2024
Defense Loss
CIFAR-10
CIFAR-100
ImageNet
Defender Src. Model Groups
Clean
Adv.
Function
Clean
Adv.
Clean
Adv.
Clean
Adv.
All groups
96.1
88.6
RANDOM
96.1
88.6
76.2
50.8
79.0
58.6
All groups but normal
95.4
83.4 (-52)
TOP-1
95.3
87.0
73.9
50.9
78.2
57.3
All groups but 2o
95.3
80.6 (-80)
TOP-2
95.8
86.1
74.2
51.9
78.4
60.6
groups but
95.0
60.1
(-28.5)
ALL
96.0
86.8
73.5
50.3
78.5
62.3
All groups but corruption
94.9
56.9 (-517)
DYNAMICACC
95.2
88.9
74.0
51.5
78.6
62.8
DYNAMICLOSS
95.6
88.4
73.8
51.1
78.6
63.0
Table 9: Effects on accuracy when excluding
one (out of four) defender's source models from PUBDEF trained CIFAR-10.
Table 10: Clean and adversarial accuracy of our de- fense with different training methods (Section 5).
Defenses
Clean Acc.
Adv. Acc.
Best white-box adversarial training
85.3
68.8
TRS
90.7
30.1
TRS adversarial training
86.9
66.7
DVERGE
88.6
33.4
DVERGE + adversarial training
87.6
Frozen TRS
86.
49.9
PUBDEF
96.1
88.6
Table 11: Comparison of PUBDEF all the ensemble-based defenses CIFAR-10.
attacks. Training the transfer attacks themselves like PUBDEF is more straightforward and so produces a better defense. An interesting future direction to combine TRS (or other forms of gradient regularization) with PUBDEF,
A.6.2 ADVERSARIAL. ACCURACY ON ALL TRANSFER ATTACKS
In this section, we include the figures containing adversarial accuracy against all the transfer at- tacks on CIFAR-10 (Fig. 11) and CIFAR- 100 (Fig. 12), similarly to Fig. 3 for ImageNet in the main paper. NA attacks excluded and marked "n/a" for the ensemble-based model (i.e., Di wfffenderfer222222i...
LRR_ CARD LRR_CARD Deck) because the intermediate feature more difficult to specify. Some interesting observations:
PGD and M-PGD are among the best attack algorithms across all of the datasets despite their simplicity compared to the rest of the attacks.
NA attack the weakest attack both CIFAR-100 and ImageNet but the strongest on CIFAR-10.
The source models from the fo group produce stronger transfer attacks than the other groups. The trend is particularly strong CIFAR- 1 but weaker on CIFAR-10 and ImageNet.
A.6.3 ROBUSTNESS AGAINST QUERY-
QUERY-BASED BASED ATTACKS
We include additional results on the models under query-based attacks. Table 12 shows accuracy under Square attack, soft-label (or score-based query-based attack without any system-level defense in place. Table 13 reports accuracy under hard-label (or decision-based basedd query- attack also without any system-level defense. Here, the white-box adversarially trained models are more robust than PUB However, after applying system-level defense like Qin et al. (2021), PUBDEF has higher accuracy and robustness as shown in Section 7.2.
A.6.4 DETAILS THE ADVERSARIAL SUBSPACE EXPERIMENTS
We start by providing more intuition by visualizing a space of the transfer attack around a training sample z. Fig. 13 compares the attack surfaces under the white-box and the TAPM threat model. The white-bo box attack is inherently more difficult to solve as the defender must train the model to be robust any point in the normr ball of given radius. On the other hand, under the TAPM
22

---

Published as a conference paper at ICLR 2024
narakiivlt baser patch (ec6661-32 20--24--l 224-i221 mk21k finstundd sifart0
(Normal)
Diffeuder Didinneeree222 (e22222 Viningg LRR CARD Desk O
(fa)
(t))
abvanjave aonojavvidceo weaaaaidddeeaaacceeecc VacXy-aa -iy---in------ finnnmes--stee -ifar1
Angasia 2w222 1o 70, 16 cumixx cxtm
Aversaridl 34, 10, eura
Kireev 203 Effeetie Effeetivenee vtveessss RI ALLATAugha
westtttttteeee www.s////ssheeeeeee www.siienn---aaae 221-5-
Vang 302 Better WRN O
Y02t22 Dehunedour222 Light XCIT-L12
Wang 20g22222 Wittee WRN-700166
WRRN-28-11
Hendrycle 302000000000000000 AngMix ResNeXi
Modas WRR.EEEIIIIIII Not
Judks Boltt//Bii -ineruned
cheny (erresste22))
chonyanto wog.ooooggggggg 1-bn
Cheny Chnyyarrre wonpyaafrrrerrrr aistepaa---- 28--0
hay cayyyphantee wwp.anddeeeeee wew.uundddeeeeee--tet 121
Sehwang Myraa
Romy 20 Decaalinng
Can anveinau Jocal
chenyaofi chemyaslee
Robalizioo
(Corrupe)
ali
98
PGD
96
96
966
96
95
95
Transfer Attack Algorithms
M-PGD
Pregradient
96
84
93
95
915
144
955
93
92
344
344
95
95
95
SM4
955
96
DI
97
96
91
T1
95
Admix
97
95
96
99
93
92
96
96
96
96
96
94
NA
NI-SI-TIDIM
95
93
95
95
95
95
93
OX
95
95
91
95
95
NI-Admix- TI-DIM
MA
95
9N
MA
98
92
AwtPGD-DDRR
AutaPGD CT
96
95
95
95
95
95
04
95
93
94
95
94
96
96
95
9h
95
-1
Arg.
93
90
90
92
90
92
89
91
93
92
91
94
92
92
92
92
91
89
Transfer Attack Source Models
Figure 11: Adversarial accuracy of PUB against 264 transfer attacks (24 source models x 11 attack algorithms) on CIFAR-10 + denotes the source models this defense is trained against.
CIFAR-10
CIFAR-100
ImageNet
Models
Sq- L0D
Sq- 1000
Sq-100
1000000
Sq- 10-1
Sq 1000
No Defense
36.6
0.2
9.1
1.8
49.9
17.9
Best White-Box AT
77.9
67.3
55.9
41.0
58.4
57.6
PUBDEF
55.2
8.8
13.5
0.3
55.1
32.3
Table 12:
Accuracy of the models under Square Attack (soft-label -bsed) with 100 and 1000 queries. Here, no system-level defense is applied.
setting, there are only finitely many attacks, and so they will always "occupy" a smaller volume of the ball. Put differently, they only span linear subspace of min(s (8 a, d] dimensions where is the number of all potential transfer attacks defined in Section 3 and d is the input dimension where usually > sa.
Fig. 14 potentially explain why PUBDEF generalizes well to attacks not seen during training. When all the attacks lie in low-dimensional subspace form cluster around a small subsection of the fp-norm ball, the sample complexity is also low. Hence, training on few attacks is sufficient for capturing this adversarial subspace. On the other hand, there can be two potential failure cases of TAPM if the transfer attacks do not actually form low-dimensional subspace. First, the model can overfit to the training attacks and does not generalize the unseen. The second case can happen when the model does not necessarily over to the training attack and tries to learn to be robust in the entire fo-norm ball. this case, TAPM could perform similarly to white-box adversarial training, and we may on our results, it seems that TAPM may experience the first failure case on a subset the samples, creating small generalization gap between the seen and the unseen attacks as shown in Table 2.
benefit it suffers the same pitfalls white-box adversarial training. Based
Cosine similarity. Here, we include additional observations that were not mentioned in Section 7.3. From Fig. 5, there is also relatively strong similarity (0.17) between the for and the & groups
23

---

Published as a conference paper at ICLR 2024
Diffiandetss 2etnanderrre2222122222 2122 Wimning LBn. CARD. Deck o
(Normal)
(t
(Corrup.)
Muzet ManoeAAniiaa wani styy frnetuned cifeeed---fffttaa
Gowal Addepelii2222 Mespelit22222mre www.ttat2222ttteaemmmem fesiient WRN. M I.
www.ooonn.2.2211 Light. XCT--112
wec.k0222Auu Aughuigiii..............iiz ReNNei
hte wexiingg runn Lini
Hinary
cheryani wrwchlletttteee Mw 2-20-5
cleap Saapaaatshhalllnneee 22-0055
Wan Waig Wang 2rng2rr2BBeter Bettey, WRN-T6--
Cocoi Mea.DDDDeepppppp WRN-
Rade Rade Helps R1d. ddpmn
PRIMER PRIMEReNNeT!
Abmua122 2at.aa977555// Cifar
Baiz0z N0120zttmprriine odm
Rict122A AS-AA. $4
deedyafferesseee O
cleayynatt rgst bn
cherpy anyyooforryyrggg..
denvered 221 .beaal
Inceptioa conventte Jonal
23 Jocal
Ditiinnert 2022
Setee Jucall
Henndry Modaciie
city
IGD
M. M-PGD
20
24
G4
65
6G
56
5G
56
G2
56
61
CR
6G
GCI
GSI
Transfer Atlack A Algorithms
Pregradient
C6
56
67
80
75
RO
73
72
CO
70
O8
61
R2
CO
86
62
6S
64
OB
CS
6S
64
GB
70
Admix
76
76
w/a
NL.SL TI-DIM
6G
O7
O7
GP
G9
NI-Admix- TI-DIM
AntoPUID-DUR
75
74
74
74
70
71
73
744
23
76
66
67
62
71
73
70
75
AutePGD-CE
74
72
66
66
20
69
Avg.-
Best
69
70
20
69
64
67
68
66
70
54
55
51
62
54
61
64
66
6O
56
68
Transfer Attack Source Models
Figure 12: Adversarial accuracy of PUBDEF against 264 transfer attacks (24 source models x 11 attack algorithms) on CIFAR- 100. O denotes the source models this defense is trained against.
Models
CIFAR-10 (HSJ)
CIFAR-100 (HSJ)
ImageNet (HSJ)
ImageNet (GeoDA)
No Defense
0.0276
0.0208
0.1995
0.1
Best White-Bi BBxc AT PUBDEF
0.1542
0.1518
0.1969
0.1662
0.0466
0.0397
0.1814
0.1604
Table 13: Mean adversarial perturbation norm (lxo) found by two hard-label query-based attacks, HSJ (Chen et al., 2020a) and GeoDA (Rahmati et al., 2020), with 1000 queries. Higher means the model is more robust.
compared to the other cross-group pairs. This corroborates the earlier result as well as observation made by Croce & Hein (2021) that robustness transfers among different fy-norms. However, this result does not explain generalization among normally trained models, the cosine similarity within the group remains low (0.04). The low similarity might be due to the fact that there are 12 source models in the normally trained group instead of four in the others which implies higher diversity within the normally trained group as well as the transfer attacks generated from them.
PCA. In addition cosine similarity, we fit PCA on the adversarial perturbation in an attempt to verify whether they lie in a now-dimensioma linear subspace. This analysis is inspired by the notion of "aavversarial subspace" done by Tramèr et al. (2017). First, we randomly choose a test sample as well as all 264 adversarial examples generated from it. We put these adversarial examples into five groups by the source models (including all). Then, we fit PCA to the attacks in each of these groups. If the adversarial examples do lie lowedimensional linear manifold. then should expect that most of the variance is explained by only the first few principal components.
Fig. 15 shows the explained variance plot for three random CIFAR-10 samples, and from this figure, we can see some evidence for the low-dimensional linear subspace for the robust training groups In the fxc. &2, and corruption groups, the first principal component already explains about 20% of all the variance. For loo and E,, 90% of the variance is explained by approximately 15 dimensions. This result supports our observation based the cosine similarity: the for and the lg groups seem
24

---

Published as a conference paper at ICLR 2024
P --ror Ball
White-Box Setting
TAPM Setting
Attack #1
Attack #2
Attack
X
X
Figure 13: Schematic comparing TAPM to the white-box threat model. Left: an e--norm ball with radius around clean input sample æ. Middle: the white-box threat model assumes that attacks can lie anywhere inside the ball and so the defense has to protect the entire ball (red highlight). Right: the TAPM threat model expects only a finite set transfer attacks that may lie on a mmall-dimensional manifold.
Good Generalization (what observe)
Failure Case #1: Bad Generalization
Failure Case #2: Just Adversarial Training
Training Attacks
Test Attacks
Figure 14: Schematic explaining the generalization phenomenon of PUBDEF. Left: the observed case where PUBDEF generalizes well unseen attack. Middle: the first potential failure case where PUBDEF overfits to the attacks used during training. Right: the second failure case where PUBDEF does not overfit but does not perform better than adversarial training.
to cluster most tightly (high cosine similarity) followed by the corruption group and lastly by the normal group.
25

---

Published as a conference paper at ICLR 2024
PCA Explained Variance
1.0
Explained Variance
1.0
PCA Explained Variance
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.1
0.1
0.1
0.2
0.2
0.2
0.0
0.0
0.0
120
240
80
PCA Dimension
PCA Dimension
PCA Dimension
Normal Group
Normal Group
Normal Grouu
1.0
0.6
0.4
0.2
0.2
0.0
0.0
20
40
60
80
100
120
0
20
40
60
90
100
120
0
20
40
60
80
100
120
Es Group
Lxp Group
Ev Group
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.4
0.2
0.2
5
10
15
15
20
25
& Group
&g Group
Group
1.0
1.0
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
5
10
15
10
15
10
15
20
Corruption Group
Corruption Group
Corruption Group
1.0
1.0
0.0
5
10
15
20
25
30
35
40
5
10
15
20
25
30
35
25
35
(a) Sample
(b) Sample #2
(c) Sample #3
Figure 15: PCA explained variance function of dimension (number of the principal components) of the transfer attacks for three random CIFAR-10 samples accumulated by the source model groups All attack algorithms were used. The green and the red dashed lines denote 90% and 95% of the variance.
26


Reviews:
[{"review_id": "TNbI4EXSj4", "reviewer": "Reviewer_nHrU", "summary": "The paper argues that the common modality for attack on ML models (image classifiers are used in experiments) is going to be via transfer attacks on public models since companies are likely to keep their model weights protected. The paper then recommends defenders to adversarially train against a reasonable subset of  public models and find that works against L-inf norm attacks that are transferred from public models. They call their technique PUBDEF.\n\nThe authors present the work in a game-theoretic perspective where the attacker only has access to public models.\n\nThe authors acknowledge some of the limitations of the work. In my judgement, they are quite significant. They mention that if an attacker somehow is able to infer model weights (say by training a surrogate model), then they can bypass the defense. For typical classifiers, I think this is a significant concern. One limitation that they do not appear to consider is the possibility of black-box attacks directly on the model (no transfer attack needed) and do not evaluate their defense against a black-box attacker. Thus, the setting is somewhat limited in which the attacker can only do transfer attacks on the protected model and nothing else. \n\n\nOverall, the motivation for the paper, the defense approach used, and evaluation all need to be better.", "questions": "I recommend authors to consider a different setting for their work where transfer attacks are more common due to the complexity of stealing models or doing direct blackbox attacks. The setting would be attacks on generative AI models such as ChatGPT. Recent work shows that transfer attacks are possible on such models (see a 2023 paper by Zico Kolter, Matt Fredrikson and others that  was also mentioned in a recent NYTimes article) where the attacker is able to append a suffix to a prompt to cause the model to output something that can be considered harmful in some way. Unfortunately, though, adversarial training is not the defense approach that is used in such settings (at least until now). So, the approach presented by the authors in this paper is unlikely to work and may require significant changes. But, the setting is likely more realistic for the assumptions they rely on.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The threat model of transfer attacks is well known. The authors assume that this is the main likely modality in practice. That assumption is likely a reasonable assumption only in very complex and large models. In most other cases, including datasets and models that the authors consider, there are other practical attack strategies, including blackbox attacks and model stealing attacks. To authors' credit, they acknowledge some of these limitations (especially model stealing attack via training a surrogate), but that doesn't make the assumption more realistic.\n\nGiven their assumption, their experimental results appear reasonable. The main contribution is that the defender can choose a small subset of public models against which to adversarially train their model. They require that public models be trained on the same task.  In practice, they find that such a model is often robust against a broad range of adversarial attacks that are restricted to using the public models.", "weaknesses": "Weaknesses are unfortunately significant. \n\n-- The assumption underlying the paper is likely unrealistic in that most blackbox models will also permit querying. Thus, transfer attacks are not the only option for an adversary. Blackbox attacks are also a possibility and, in fact, may be the primary attack strategy. The proposed defenses were not evaluated against Square attack and other blackbox attack strategies.\n\n\n[Rebuttal]\n\nI considered the rebuttal provided so far.  The main concern its that the approach is susceptible to blackbox attacks (model stealing attacks are a potential issue as well, but I think the primary attack vector is likely to be blackbox attacks). An attacker will choose the best attack strategy available among many options, including transfer attacks, blackbox attacks, and model stealing attacks. The authors present preliminary results that showed that blackbox attacks would succeed and they would thus either need stateful defenses to work (which were unfortunately recently broken by Feng et al. 2023) or they need to combine their scheme with a noise-based blackbox defense where inputs are combined with noise before doing an inference. The latter combination showed some promise, exceeding the performance of a simple white box defense, but significantly lowering the reported natural accuracy and adversarial accuracy of PubDef by about 10% each in Table 1. \n\nI am raising my score, assuming that the authors are willing to include a deeper analysis of their work against blackbox attacks on CIFAR-10 and CIFAR-100 in the final paper and include a clear acknowledgment something along the following lines in bullet 3:\n\nBlackbox attacks are a potential concern against PubDef. When we started this work, a stateful defense was available, which would have thwarted blackbox attacks and could be combined with our scheme  with no or little change to reported results. Unfortunately, very recently,  Feng et al. broke current stateful defenses, but did not rule out potential new stateful defenses. If new stateful were to be developed, they should be combined with your scheme. In the absence of stateful defenses being available, we present preliminary findings  against a blackbox attacker (see Appendix, section X). They suggest that PubDef, unmodified, would be vulnerable to blackbox attacks and not provide an adequate defense. However, PubDef in combination with a noise-based defense shows some promise (see Appendix, Section X). For example, on the CIFAR-10 dataset, with a sigma of 0.02, PubDef  has an adversarial accuracy of 79.8 against the best attacker, with a natural accuracy of 88.9. This is a drop of adversarial accuracy  and natural accuracy of about 10% each  from the  results in Table 1 that only factored in transfer attacks, but still superior to simply using best white-box adversarial training in both clean and natural accuracy. [And then report additional results on other datasets and include them in the paper.]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "50fycG5ovF", "reviewer": "Reviewer_DtJG", "summary": "The authors approach transfer attacks from the practical perspective and propose training procedure that allows to achieve robustness with a small drop in clean accuracy. They empirically show that their rather simple approach to selecting models allows good performance with respect to SOTA white-box defences. They do this by performing extensive evaluations on a wide range of public models", "questions": "The authors state that PubDef achieves much better results than SOTA models from RobustBench. Where in the main paper can we find the clean and robust accuracy of these SOTA models to compare? I found the presentation of the results a bit confusing.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "-\tThe paper considers a real-life scenario of defending specifically against black-box attacks.\n-\tEvaluation is very extensive (considering different 264 combinations of source models and attack mechanisms)\n-\tSolid results with potentially high relevance for real-life industrial applications", "weaknesses": "The authors state the weaknesses and limitations of their approach quite fully in Section 7.3. I could add that although proposed heuristics and empirical results could be significant in practical applications, the methodological contribution of this work is rather incremental.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NMDP97IDJD", "reviewer": "Reviewer_nPyj", "summary": "This paper proposes a defense against transfer-based evasion attacks. The defense uses publicly-available pretrained models to create adversarial attacks and incorporates these attacks in the training process of the defended model. The process is similar to adversarial training, however it uses adversarial examples from (multiple) other robust models rather than from the trained model itself.", "questions": "**Evaluation is not convincing, missing: white-box evaluation, query-based evaluation, other norms, targeted attacks, strong ensemble transfer attacks**\n\nThe defense should be tested fully, also against white-box evasion attacks. This is motivated by the fact that the authors are trying to make the model robust, hence they should demonstrate the robustness against worst-case adversaries. Otherwise, there is a high risk that the defense might be broken by stronger attacks. As the approach is similar to AT, the authors should demonstrate that the decision boundary of the model is in fact becoming more robust to unseen adversarial attacks and perturbations, thus they should also test gradient-based attacks against the model itself. Additionally, the authors should test against an attack that is generated ensembling the gradients of multiple models, which should improve transferability and also test effectively the robustness claim that the authors are making.\nThe model should also be tested with variations of the attacks, e.g., with PGD with logit loss, or against APGD with DLR loss. The authors should demonstrate that the defense generalizes against other unseen attacks and variations, to really claim robustness. The defense should be robust regardless of the method used to find the adversarial attacks. \nThe model should also be tested against query-based attacks to properly validate the fact that it is not suffering from gradient masking problems.\nFinally, all the parameters of the attacks should be specified to ensure they are conducted properly and they are not suffering from optimization issues.\n\n**No analysis on how expensive the method is, compared, e.g., to adversarial training**\n\nThe authors should discuss how expensive is to train a model with this technique, compared, e.g., with AT. Additionally, they should specify how costly is to add further heuristics.\nFor example, the authors use a heuristic to choose the models to use for creating the transfer attacks. However, this heuristic seems expensive, as it requires to compute the adversarial accuracy (in transfer) against all available models. Does this require to launch a full evaluation against all models?\n\n\n**Clarifications needed**\n\n- the authors state that they propose \"using all available system-level defenses\", however the presented method is clearly a ML-level defense. The authors should clarify this aspect, as it is confusing to read and there is no discussion in the rest of the paper on which system-level defenses are mounted on the model, or if they are mounted at all.\n\n- the authors state that the drawback of AT is that it degrades clean accuracy, however this does not seem to be the case https://robustbench.github.io. The authors should clarify this aspect by detailing this statement.\n\n- The method assumes that the defended is aware of the same set of public models as the attacker. However, with the proliferation of public models (also supported in the introduction of the paper), this might not be realistic to assume that the defender is aware of all possible models available. Furthermore, assuming that the attacker has $s \\dot a$ attack strategies might bring an excessive number of combinations. The authors should clarify these statements and propose solutions for these limitations. In fact, when the authors claim that the proposed approach \"achieves over 90% accuracy against all four transfer attacks\", they should also disclose that this holds only when the set of target models is the same as the source models $T = S$.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ introduction is well written\n+ formulation is clear", "weaknesses": "- evaluation is not convincing\n- no analysis on how expensive the method is, compared, e.g., to adversarial training", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j5IWP9FjBS", "reviewer": "Reviewer_CVr6", "summary": "This paper proposes a pragmatic defence against a newly proposed practical threat model - namely adversaries utilising transfer attacks via publicly available surrogate models (with no ability to fine-tune models or perform query-based attacks). The defence is motivated by a game-theoretic perspective, and evaluated against a range of public models, attack algorithms and datasets.", "questions": "In section 7.3, PUBDEF's vulnerability to white-box and surrogate attacks is mentioned. It would have been good to see some results on this if possible in order to give an idea of the degradation of performance that such weakening of the threat model would provide, i.e. effectively an ablation study on the threat model assumptions.\n\nIs there some value to considering a case in which the set of publicly available models visible to the attacker and defender are different. Maybe there are communities not visible to the defender, or perhaps it could be because of time-based issues (e.g. defender model generated and frozen but subsequent public models appear which are available to the attacker)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "A very well written and presented paper. The game-theoretic motivation is clear and provides a reasoned foundation for the approach taken - though that approach does not fully solve the game-theory problem as posed, the framework at least justifies the concept of the approach as more than just arbitrary/ad hoc.\n\nThorough experimental results including extensive ablation studies. The investigations, for example, of the effects of removing various groups of models on the accuracy was interesting. Further, investigating the cosine similarity of perturbations between attacks was suggestive of further structure to be explored in future work.\n\nThe candidate defence outperforms SOTA by a reasonable margin, and provides a practical mechanism in contrast to SOTA adversarial training. I think because of this practical performance requirement for their approach with still SOTA or better results, it could be a useful technique to use in practise or to consider developing further.", "weaknesses": "The \"slogan\" is not very catchy! :-)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)