Game Theoretic Mixed Experts for Combinational Adversarial Machine Learning
Authors: Ethan Rathbun
Kaleel Mahmood
Sohaib Ahmad
Caiwen Ding
Marten van Dijk
Venue: Submitted to ICLR 2023 (2023)
Keywords: Adversarial Machine Learning
Security

Abstract: Recent advances in adversarial machine learning have shown that defenses considered to be robust are actually susceptible to adversarial attacks which are specifically tailored to target their weaknesses. These defenses include Barrage of Random Transforms (BaRT), Friendly Adversarial Training (FAT), Trash is Treasure (TiT) and ensemble models made up of Vision Transformers (ViTs), Big Transfer models and Spiking Neural Networks (SNNs). It remains an open question, however, as to whether the adversarial examples designed to target one defense will be similarly misclassified by another defense. In this paper, we provide the first adversarial defense transferability study, as well as a game theoretic framework for ensemble adversarial attacks and defenses. Our framework is called Game theoretic Mixed Experts (GaME) and is designed to find the Mixed-Nash strategy for an attacker that can employ compositional adversarial attacks. We show that this framework creates an ensemble of defenses with greater robustness than a combinational defense with a uniform or random probability distribution. Overall, our framework and analyses advance the field of adversarial machine learning by yielding new insights into compositional attack and defense formulations.

Under review as a conference paper at ICLR 2023
GAME THEORETIC MIXED EXPERTS FOR COMBINA- TIONAL ADVERSARIAL MACHINE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances in adversarial machine learning have shown that defenses consid- ered to be robust are actually susceptible to adversarial attacks which are specifi- cally tailored to target their weaknesses. These defenses include Barrage of Ran- dom Transforms (BaRT), Friendly Adversarial Training (FAT), Trash is Treasure (TTT) and ensemble models made up of Vision Transformers (ViTs). Big Transfer models and Spiking Neural Networks (SNNs). A natural question arises: how can one best leverage combination of adversarial defenses to thwart such attacks? In this paper, we provide a game-theoretic framework for ensemble adversarial attacks and defenses which answers this question. In addition to our framework we produce the first adversarial defense transferability study to further motivate a need for combinational defenses utilizing a diverse set of defense architectures. Our framework is called Game theoretic Mixed Experts (GaME) and is designed to find the Mixed-Nash strategy for defender when facing an attacker employing compositional adversarial attacks. We show that this framework creates an en- semble of defenses with greater robustness than multiple state-of-the-arr, single- model defenses in addition to combinational defenses with uniform probability distributions. Overall, our framework and analyses advance the field of adver- sarial machine learning by yielding new insights into compositional attack and defense formulations.
INTRODUCTION
Machine learning models have been shown to be vulnerable adversarial examples Goodfellow et (2014):; |Papernot et al. (2016). Adversarial examples are inputs with small perturbations added, such that machine learning models misclassify the example with high confidence. Addressing the security risks posed by adversarial examples are critical for the safe deployment of machine learning in areas like health care Finlayson et al. (2019) and self driving vehicles Qayyum et al. (2020). However, current defenses and attacks in adversarial machine learning have trended towards cat and mouse dynamic where in new defenses are continually being proposed and then broken Carlini & Wagner (017) Tramer al. (2020): Mahmood et al. (2021a): Sitawarin al. (2022) by improved attacks.
In parallel attack and defense development, studies have also been conducted the transferability of adversarial examples| Liu et al. (2016); Mahmooo et al. (2021b); Xu et al.) (2022). Transferabiltiy refers to the phenomena where adversarial examples generated for one model are also misclassified by different machine learning model. However, to the best of our knowledge no analyses have
been done on the transferability of adversarial examples designed to attack specific defenses. From these observations several pertinent questions arise:
1. Do adversarial examples generated for specific defense transfer to other defenses?
2. Based on adversarial transferability, can a game theoretic framework be developed to de- termine the optimal choices for both attacker and defender?
3. Can randomized defense selection yield higher robustness than a single state-of-the-ar defense?
These are precisely the questions our paper seeks to answer. We break from the traditional dynamic of adversarial machine learning which focuses on the single best attack and defense. We instead take

---

Under review as a conference paper at ICLR 2023
multi-facete approach and develop game theoretic framework answer the above questions. Specifically, we provide the following contributions: Most importantly we formulate a practical, name-theoreti framework for finding the optimal strategies for an attacker and defender who each employ a set of state-of-the-art adversarial attacks and defenses. Motivated by this framework, develop two new white-box attacks called the Momentum Iterative Method over Expectation (MIME) and the Auto Expectation Self-Attention Gradient Attack (AE-SAGA) in order to create stronger adversary. These attacks are necessary for targeting certain randomized defenses and for adapting to multi-defense strategies. Lastly. we analyze the adversarial transferabilitty of cur- rent defenses like Trash is Treasure Xiao & Zheng (2020), Barrage of Random Transforms Raff et al. (2019), Friendly Adversarial Training [Zhang et al. (2020) and other new architectures like SNNS Rathi & Roy (2021b): Fang et al. (2021) and ViT' Dosovitskiy et al.] (2020). We further leverage the low transterability between these classifiers to find those which are best suited for combined, ensemble defense such as the one developed in our came-theoretic framework.
2 ADVERSARIAL MACHINE LEARNING DEFENSES
Here we summarize the state-of-the-ane defenses we analyze in this paper. In the following subsec- tions, we give an overview of each defense and our reasons for choosing said defense. It is important to note our analyses encompass a broad range of different defenses, including ones based on ran- domization, ones based on adversarial training and ones based on exploiting model transferability. In addition, we also consider diverse architectures including Big Transfer models (BiTs), Vision Transformers (ViTs) and Spiking Neural Networks (SNNs). Despite our broad range, we not attempt to test every novel adversarial defense. It is simply infeasible to test every proposed adver- sarial machine learning defense, as new defenses are constantly being produced. However, based on our game theoretic design and open source code (which will be provided upon publication), any new defense can easily be tested and integrated into our proposed framework.
2.1 BARRAGE OF RANDOM TRANSFORMS
Barrage of Random Transforms (BaRT) Raff et al. (2019) utilize a set of image transformations in random order and with randomized transformation parameters to thwart adversarial attacks. Let 1 (xx) represent the jth transformation used in the 5th order in the sequence. A BaRT defense using image transformations randomly alters the input x::
f(x) = pann pla-1
1 (x)
(1)
where w represents the subset of transformations randomly selected from set of N total possible transformations and μ represents the randomized order in which the n transformations are applied. In Equation| the parameters of each image transformation E) (z) are also randomized run time, further adding to the stochastic nature of the defense. In this paper, we work with the original BaRT implementation which includes both differentiable and -on-differentiable image transformations.
Why we selected it: Many defenses are broken soon after being proposed Tramer et al. (2020). BaRT is one of the few defenses that has continued to show robustness even when attacks are specifically tailored to work against it. For example, most recently BaRT achieves 29% robust- ness on CIFAR- 10 against customized white-box attack SSttwaari al. (0022). It remains an open question whether using BaRT with other randomized approaches (i.c. selecting between different defenses) can yield even greater robustness.
2.2 FRIENDLY ADVERSARIAL TRAINING
Training classifiers to correctly recognize adversarial examples was originally proposed in Goodfel- low et al.] (2014) using FGSM. This concept was later expanded to include training on adversarial examples generated by PGD in Madry et al. (2018). In Zhang et al. (220)) it was shown that Friendly Adversarial Training (FAT) could achieve high clean accuracy while maintaining robustness to ad- versarial examples. This training was accomplished by using a modified version of PGD called PGD-K-7. In PGD-K-T, K refers to the number of iterations used for PGD. The variable is hyperparamter used in training which stops the PGD generation of adversarial examples earlier than the normal K number of steps, if the sample is already misclassified.
2

---

Under review as a conference paper at ICLR 2023
Why we selected it: There are many different defenses that rely adversarial training Madry al. 2 Zhang et al. (2019); Wang et al. (2019); Maini et al. (020) and training and testing them all is not computationally feasible. We selected FAT for its good trade off between clean accuracy and robustness, and because we wanted to test adversarial training on both Vision Transformer and CNN models. In this regard, FAT is one of the adversarial training methods that has already been demonstrated to work across both types of architectures Mahmood et al. (2021b).
2.3 TRASH IS TREASURE
One early direction in adversarial defense design was model ensembles Pang et al. (2019). How- ever, due the high transferability of adversarial examples between models, such defenses were shown to not be robust Tramer et al. (2020). Trash is Treasure (TiT) Xiao & Zheng (2020) is a two model defense that seeks overcome the transferability issue by training one model (.(+) on the adversarial examples from another model Cs(-). At run time both models are used:
y = C. (1b(2, Cb))
(2)
where di is an adversarial attack done on model C. with input x and C. is the classifier that makes the final class label prediction the adversarial example generated by with Co.
Why we selected it: TiT is one of the newest defenses that tries to achieve robustness in a way that is fundamentally different than pure randomization strategies direct adversarial training. In our paper, we further develop two versions of TiT. One version is based on the original proposed CNN-CNN implementation. We also model and Vision Transformers to try and leverage the low transferability phenomena described in Mahmood et (2021b).
second mixed architecture version using Big Transfer
2.4 NOVEL ARCIITECTURES
In addition adversarial machine learning defenses, we also include several novel architectures that have recently achieved state-of-the-art or near state-of-the-art performance in image recogni- tion tasks. These include the Vision Transformer (ViT) Dosovitskiy et al.) (2020) and Big Transfer models (BiT) Kolesnikov et al. (2020). Both of these types of models utilize pre-training datasets and fine tuning on smaller datasets to achieve high fidelity results. We also test Spiking Neu- ral Network (SNNs) architectures. SNNs are competitor to artificial neural networks that can be described as a linear time invariant system with a network structure that employs non-differentiable activation functions Xu et al. (2022). A major challenge in SNNs has been matching the depth and traditional deep learning models. Two approaches have been used to over- come this challenge, the Spiking Element Wise (SEW) ResNet Fang et al. (202I) and transferring weights from existing CNN architectures to SNNs Rathi Roy (2021a). We experiment with both approaches in our paper.
larger
model complexity
Why we selected it: The set of adversarial examples used to attack one type of architecture (e.g. a ViT) have shown not be misclassified by other architecture types (e.g. BiT SNN) Mahmood et al. (b021b); Xu et al. (2022). While certain white-box attacks have been used to break multiple undefended models, remains an open question if different architectures combined with different defenses can yield better performance.
3 ADVERSARIAL ATTACKS
In our paper assume a white-box adversarial threat model. This means the attacker is aware of the set of all defenses that the defender may use for prediction. In addition, Yd E the attacker also knows the classifier weights lg architecture and any input image transformations the defense may apply. To generate adversarial examples the attacker solves the following optimization problem:
Σ La(x + 5, y: subject to: ||ő1|p <c
max
RED
where is the set of all possible defenses (models) under consideration in the attack, Lg is the loss function associated with defense D, 8 is the adversarial perturbation, and (ar, y) represents the
3

---

Under review as a conference paper at ICLR 2023
original input with corresponding class label. This is more general formulation of the optimization problem allowing the attacker to attack single or multi-model classifiers. The magnitude of this perturbation 5 is typically limited by certain lp norm. In this paper we analyze the attacks and defenses using the lo norm.
Static white-box attacks such as the Projected Gradient Descent (PGD) Madry et al. (2018) attack often perform poorly against randomized defenses such as BaRT or TVT. In Xiao & Zheng (2020)
they tested the TiT defense against an attack designed to compensate for randomness, the Expec- tation over Transformation attack (EOT) attack Athalye et al. 2018). However, it was shown that the EOT attack performs poorly against TiT (e.g. 20% or worse attack success rate). For attacking BaRT, in Sitawarin et al. (2022) they proposed a new white-bor box attack to break BaRT. However, this new attack requires that the image transformations used in the BaRT defense be differentiable, which is a deviation from the original BaRT implementation.
Attack Contributions: It is crucial for both the attacker and defender to consider the strongest possible adversary when playing the adversarial examples game. Thus, we propose two new white- box attacks for targeting randomized defenses. The first attack is designed to work on individual randomized defenses and is called the Momentum Iterative Method over Expectation (MIME). To the best of our knowledge, MIME is the first white-box attack to achieve a high attack success rate (> 70%) against TiT. MIME is also capable of achieving a high attack success rate against BaRT, even when non-differentiable transformations are implemented as part of the defense. Our second attack, is designed to generate adversarial examples that work against multiple type of defenses (both randomized and non-randomized defenses) simultancously. This compositional attack is called, the Auto Expectation Self-Attenton Gradient Attack (AE-SAGA).
3.1 MOMENTUM ITERATIVE METHOD OVER EXPECTATION
We develop a new attack white-box attack specifically designed work defenses that inherently rely on randomization, like Barrage of Random Transforms (BaRT) Raff et al. (2019) and Trash is Treasure (TiT) Xiao & Zheng 2222)). Our new attack is called the Momentum Iterative Method over Expectation (MIME). The attack "mimes" the transformations of the defender in order to more precisely model the gradient of the loss function with respect the input after the transforma- tions are applied. To this end, MIME utilizes two effective aspects from earlier white-box attacks, momentum from the Momentum Iterative Method (MIM) Dong et al. (2018) attack and repeated sampling Athalyy et al. (2018) from the Expectation Over Transtormation (EOT) attack:
(1-1)
I odd
L addv + Estopg
(4)
where the attack is computed iteratively with addu = x. In Equation 4 ( is the momentum based gradient of the loss function with respect to the input iteration and defined as:
(0)
OL
g00) = (g E
(5)
(4)
where y is the momentum decay factor and t is random transformation function drawn from the defense's transformation distribution T. In Table we show experimental results for the MIME attack on CIFAR- 10 randomized defenses (TiT and BaRT). It can clearly be seen that MIME has
better attack success rate than both APGD Croce & Hein (200)) and MIM Dong et al. 2018
Attack
BaRT-1
BaRT-5
BaRT-10
TTT (BiT/ViT)
TTT (VGG/RN
MIME-10 10
3.18%
15.5%
43.2%
10.1%
24.9%
MIME-50
4.3%
8.22%
23.2%
8.3%
23.3%
MIM
6.7%
39.5%
59.5%
52%
58.9%
APGD
8.9%
47.7%
70.8%
68.2%
40.7%
Clean
98.4%
95.3%
92.5%
90.1%
76.6%
Table Performance of the MIME attack against CIFAR- randomized defenses: Trash Treasure (TTT) and Barrage of Random Transforms (BaRT). In our experiments <mar = 0.031 for all attacks. can clearly be seen that MIME outperforms both APGD and MIM on these two randomized defenses. Further defense and attack implementation details are given in Table ??
4

---

Under review as a conference paper at ICLR 2023
3.2 AUTO EXPECTATION SELF-ATTENTION GRADIENT ATTACK
The use of multi-model attacks are necessary to achieve a high attack success rate when dealing with ensembles that contain both CNN and non-CNN model architectures like the Vision Transformer (ViT) Dosovitskiy et al. (2020) and Spiking Neural Network (SNN) Fang et al. (202T). This is because adversarial examples generated by single model white-box attacks generally do not transfer well between CNNs, ViTs and SNNs Mahmood et al. (2021b); Xu et al. (2022). In addition it is an open question if muulti-mode attacks can be effective against the current tffteoof-the-an defenses. In this paper, we expand the idea of a multi-model attack to include not only different architecture types, but also different defenses, The generalized form of the multi-model attack is found in Equation 3
For single input with corresponding class label y. an untargeted multi-model attack is considered successful if (vd E D, Ca(x + 6) * y) A ([16]I, S e). One formulation of the multi-model attack is the Auto Self-Attention Gradient Attack (Auto SAGA) which was proposed in Xu et al. (2022) to iteratively attack combinations of ViTs, SNNs and CNNs:
(6+1) adv + estep * 4 adv
(1 adv!)
where Estep was the step size used in the attack. In the original formulation of was a weighted average of the gradients of each model d E D. By combining gradient estimates from different models, Auto-SAGA is able to create adversarial examples that are simultaneously misclassified by multiple models. One limitation of Auto-SAGA attack is that it does not account for defenses that utilize random transformations. Motivated by this, we can integrate the previously proposed MIME attack into the gradient calculations for Auto-SAGA. We denote this new attack as the Auto Expectation Belf-Attention Gradient Attack (AE-SAGA). Both SAGA and AE-SAGA use the same iterative update (Equation 6). However, AE-SAGA uses the following gradient estimator:
Gotend
Gotend(zz (o GGtend
(2 ado
(i-1) )
Σ RED
(3) (3) a O
+
(6) (000 (BETT
dL
1) (7)
(4) Oz
Öt 0t(2
ade
PER
In Equation 7 the two summations represent the gradient contributions of sets DVR R and R. respec- tively. Here we define R as the set of randomized defenses and D as the set of all the defenses being targeted. In each summation ø is the self-attention map Abbnar & Zuidema (2020) which is replaced with a matrix of all ones for any defense that does not ViT models. Og and or are the associ- ated weighting factors for the gradients for each deterministic defense l and randomized defense r, respectively, Details of how the weighting factors are derived are given in Xu et al. (2022).
4 TRANSFERABILITY EXPERIMENTS
Adversarial transferability refers to the phenomena in which adversarial examples generated to at- tack one model are also misclassified by a different model. Adversarial transferability studies have been done variety of machine learning models Liu al. (2016); Mahmood et al. (2021b); Xu eet al. (002).. However, to the best of our knowledge, adversarial transferability between different state-of-the-art defenses has not been conducted. This transferability property is of significant inter- est because a lack of transferability between different defenses may indicate new way to improve adversarial robustnes.
In Table 2 we show the different single defenses we analyze ze in this paper and the best attack on each of them Trom the set of attacks (MIM Dong et al. (2018), APGD Croce & Hein (2020) and MIME (proposed in this work). In Figure I we visually show the transferability results of these attacks for CIFAR-10. We give detailed discussions of these results in our supplementary material and briefly summarize of the key takeaways from these experiments.
In general adversarial examples generated using the best attack on one defense do not transfer well to other defenses. For example, only 0.8% of the adversarial examples generated by the BaRT-1 defense transfer to the FAT ViT defense. The average transferability for the 8 different defenses shown in Figure [ is only 21.62% and there is no single model attack that achieves strong perfor- mance (i.e. > 50%) across all defenses. These results in turn motivate the development of a game theoretic framework for both the attacker and defender. For the attacker, this prompts the need to use multi-modes attack like AE-SAGA that was proposed in Section 33 as no single attack (APGD, MIM MIME) is ubiquitous. For the defender these results highlight the opportunity to increase
5

---

Under review as a conference paper at ICLR 2023
Detense
Best Attack
Clean Ace
Robust Acc
B1
MIME
98.40%
3,40%
B5
MIME
95.30%
13.00%
0.3
BIO
MIME
92.50%
43.50%
RF
APGD
81.89%
52.00%
VF
APGD
92.36%
25.00%
ST
APGD
91.54%
0.00%
SB
APGID
81.16%
1.60%
BVT
MIME
90%
8.60%
VRT
MIME
76.60%
26.20%
SoAT- SMIT
sate LOid for
FAT VIT
SNNT
Lovd Attack
SMAT
DARTSS
DARTA
Table 2: Single defense implementations for CIFAR- 10 with the corresponding strongest attack on the defense and the clean accuracy of the defense. The robust accuracy is measured using 1000 adversarial examples. The examples are class wise balanced and also correctly recognized by all the defenses in their original clean form. All attacks done using emaz 0.031. Complete attack and defense implementation details are given in our supplementary material.
Figure 1: Visual representation of the transfcrability of adversarial examples between defenses for C0.AR-10. The blue, green, pink, and purple bars represent adversarial examples generated using the best attack for BaRT, TIT, FAT, and SNN classifiers respectively. Full numerical tables used to generate this figure are given in our supplemental material.
robustness by taking advantage of the low levels transferability between defenses through the implementation of randomized ensemble defense.
GAME THEORETIC MIXED EXPERTS
In this section we derive our framework, Game theoretic Mixed Experts (GaME), for approximating Nash equilibrium in the adversarial examples game. In comparison to other works Meunier et al. (2021) |inoo et al. (2020) Pal & Vidal (2020) Balcan et al. (2022) Le et al. (2022) we take a more discretized approach and solve the approximate version of the adversarial examples game. This ultimately leads to the creation of finite, tabular, zero-sum game that can be solved in polynomial time using linear programming techniques. In relation our work, similar adversarial game framework was proposed in (Sengupta et al.] (2018) but did not include comprehensive defender and attacker threat models. Specifically, we develop our framework under an adversary that can employ state-of-ue oate-offthheart the-art single model and multi-model attacks and a defender that can utilize both randomization and voting schemes.
5.1 THE ADVERSARIAL EXAMPLES GAME
We build upon and discretize the adversarial examples game explored in Meunier (2021). The adversarial examples game is zero-sum game played between two players: the attacker, PA and the defender pn- Let X' be the input space and the output space of pn classifiers, and let O represent the space of classifier parameters. Additionally, let P = {a E X ||>||p S e) be the set of valid adversarial perturbations for norm p and R+
Let A {f θ X y + p.,x)} be the set of all valid attack functions. The goal of pa is to choose a A; which maximizes the expected loss pn 's classifier, 0, given some pair input and ground truth label (x, y) E X y: The goal of pD is to minimize this loss through its choice of E (. We can thus formulate the adversarial examples game as mini-man max optimization problem:
inf sup E(a,))xxy L(x a(0, v). v)
aEA;

---

Under review as a conference paper at ICLR 2023
Due to the vastness of @ and A solving this optimization problem directly is currently compu- tationally intractable. To this end, in the next subsections we will formulate GaME, and GaME, which discretize ④ and A by enlisting set of state-of- ofte-oo-theeart attacks and defenses.
5.2 GAME1
The goal of the GaME framework is to find an approximate solution to the adversarial examples game through the implementation of set of attacks and defenses which will serve as experts for pA and PD respectively.
Let A' A be a subset of all valid adversarial attack functions chosen by PA- Additionally, let D c θ be a set of defense classifiers chosen by PD. We further impose that all a E A' are white-box attacks (see Section 3 for our adversarial threat model) and that A', D are finite, i.c. | N. and [DI & Na for some Na, Na E N.
important to note that each a E A' function of some classifier, 9 E O in addition to the input and ground truth label. Due to this it is possible for Pa to chose to attack defense d E D with attack E A', while pp chooses to evaluate the sample using defense d' E D where + d'. Therefore, for convenience, we will define new, more general set of attack strategies for Pa:
A x - Pc,x) [(, v) (U, v), Oa E A', D
where we extend the definition of A' [ A to attack functions that can take subset of defense parameters U S as input (see Equation for our multi-model attack formulation). This comes into play with multi-model attacks like AE-SAGA. Thus we will let D be the strategy set of pD, and be the strategy set of PA We can then reformulate discretized version of the adversarial examples game follows:
min max [xx,,))xxxx + a axz v) y: d]]
(10)
dED
In the above two formulations we optimize over the set of pure strategies for the attacker and de- fender. However, previously explored in Araujo et al. (2020) Meunier et al. (202I), limiting ourselves to pure strategies severely inhibits the strength of both the attacker and defender. Thus we create the following mixed strategy vectors for PA. PD:
X E fr E [o, 1]|41, ||rlla = 1}, X E fr e [0, 1]]1 ||rlla = =1
(11)
here A and X2 represent the mixed strategies of pA and pD respectively. Let each a E A and d, E D correspond to the ith elements of X and Xo X and Xp respectively. Formally:
(((a, E A::a= a.)) = X P[dd, E DD: d.)) = X
(12)
where a E and E D are random variables. With these mixed strategy vectors we can then reformulate the adversarial examples game mini-max optimization problem over pp 's choice of X and pA's 's choice of A
min max ( X X XA min max (a x D A
(B alx,)) (f d,)l
Σ A >
(13)
a:EA
diED
For continuous and or non-finite X. D, and A solving the above optimization problem is currently computationally intractable. Thus we can instead approximate the mini-max optimization by taking N Monte-Carlo samples with respect to (æj, Uj) E X x )::
N
ΣΣ min max
X
Σ d.EDD
XP[L(1, + (..
(14)
XD AA
j=OGEE
For convenience we will denote Idd.a, = > Eo [T(z + a; v;;d))) Colloquially Ta..a. represents the expected robustness of defense d; when evaluating adversarial samples generated by attack a, From game theoretic perspective, rd,a is the payoff for pD when they play strategy d and Pa plays strategy a. The payoff for pa given strategies d, is -Td.a- Our mini-max optimization problem can then be simplified to:
min max
Σ
X
P Td.
(15)
2 X
a.EA
d.ED
7

---

Under review as a conference paper at ICLR 2023
Using all of this create finite, tabular, zero-sum game defined by the following game- frame in strategic form:
(PA.,P)), (A, O, f)
(16)
where O = {ram Va E A, d E Dj and f is a function A x D - O defined by ʃ(d, a) rd,a- Since this is finite, tabular, zero-sum game, we know that it must have proven in Nash (1951). Let R be the payoff matrix for pD where RI,a = rd,a- It then becomes the goal of pD maximize their guaranteed, expected payoff. Formally, pD must solve the following optimization problem:
Bash-Equilibrium as
max r subject to R (r",..-r") and ||A 1 TTA
(17)
This optimization problem is a linear program, the explicit form which we provide in the sup- plemental material. All linear programs have dual problem, in this case the dual problem finds mixed Nash strategy for This can be done by changing the problem to a minimization problem and transposing R. In the interest of space we give the explicit form of the dual problem in the sup- plemental material as well. These linear programs can be solved using polynomial time algorithms.
5.3 GAME,
GaME, a more general family of games of which GaME; is a special case. In GaMEn, for n > 1, PD calculate their final prediction based upon the output logits of multiple d E D evaluated on the same input x. In order to do this, PD must also choose a function to map the output multiple defenses to final prediction. Formally, the strategy set of pD becomes D D' x F, where F is set of prediction functions and D' is defined as follows.
D' U DUI sn)
(18)
Multi-mode prediction functions can increase the overall robustness of an ensemble by requiring an adversarial sample to be misclassified by multiple models simultaneously Mahmood et al. (2022). In this paper we will focus on two voting functions: the majority vote Raff et al. (2019). and the largest softmax probability vote Sitawarin et al. (2022). We will refer to these as f and respectively:
(x,U) arg max yEy
Σ dEv
1 arg max dx)x))
(19)
jEY
(xx arg max
o(d()))
(20)
yEy
dEc
where 1 is the indicator function, is the softmax function, and (x) represents the jth output logit of defense d when evaluated on input Solving GaME,, is the same as solving GaME, except D
will be much larger. Notationally the mini-max optimization problem terms of raa remains the same as we have simply redefined D, however we can redefine Td,a follows:
T(U. = -I (a. + ax(a,, v1)),,
(21)
1=0
where (U,, E E D' x F. Similarly to GaME,, in GaME,, ((U;,))). represents the expected robust accuracy, i.e. the payoff, for pn if they play strategy (U,, [z) and PA plays strategy ag-
EXPERIMENTAL RESULTS
For our experimental results, we test on two datasets, CIFAR-10 Krizhevsky et al.) and Tiny- ImageNet Le & Yang (2015). For CIFAR-10 solved instances of GaME,, using the following defenses: BaRT-1 l), BaRT-5 (B5), ResNet-1644FB 164-FAT (RF), ViT-L-I6-FAT (VF), SNN Transfer (ST), Backprop SNN (SB), and TYT using VIT and BiT (BVT). For Tiny ImageNet solved in- stances of GaME, utilizing: B BaRT-5, ViT-L-I6-FAT, and TYT using ViT and BiT. Explicit details for experimental setup given in the supplementary material.
Figure 2 shows a visual representation of the performance of our GaME generated defenses when compared single model defenses. Here, and throughout the paper, we measure the robustness
8

---

Under review as a conference paper at ICLR 2023
of each ensemble the lowest robust accuracy achieved by any single attack in our study. From a came-theoretic perspective this can be seen as the minimum, guaranteed utility for the defender. Of all the defenses in study the FAT ResNet-164 164 had the highest robust accuracy 10 at 50%. On Tiny ImageNetBaRT-5 had the highest robust accuracy at 10.62%. Compared to these defenses our GaME generated ensemble achieved 63.5% robustness CIFAR-10 and 44.5% robustness on Tiny ImageNet, leading to a 13.5% increase and 33.88% increase in robustness on each respective dataset.
CIFAR-
In addition to this, our ensemble is able to maintain a high level of clean accuracy in spite of op- timizing for robustness isolation. In particular, the GaME framweork is able maintain 96.2% clean accuracy and 72.6% clean accuracy on CIFAR-10 10 and Tiny ImageNet respectively. This is only out performed by BaRT-1 with a clean accuracy of 98, 98.4% on CIFAR- 10 and the BiT-ViT Trash is Treasure defense with clean accuracy of 76.97% on Tiny ImageNet. It is important to note that both these cases the robustness of the GaME framework is higher.
509
20.30
60.00%
50.00%
Nourrry
38.00%
20.0%
10.0%
591
871
HP
SW9B-1
5V1
GMM
Robest
Clcan
Robust
Occon
Figure 2: (Left: CIFAR-10, 10; Right: Tiny ImageNet) Comparison of the robust and clean accuracy of the top three most robust single model defenses the top two most robust GaME generated ensembles on CIFAR-10 10 and Tiny ImageNet.
We provide more detailed, numerical results for these figures in the supplementary material. Ad- ditionally, provide studies of the effects of n (the maximum ensemble size) and N (the sample number) on GaME, generated ensembles along with an analysis of the computational cost of the framework.
CONCLUSION
The field of adversarial machine learning has begun to cycle between new defenses, followed by new specific attacks that break those defenses, followed by even more defenses. In this paper, we seek to go beyond this cat and mouse dynamic. We consider adversarial defense transferability. multi-model attacks and game theoretic framework for compositional adversarial machine learning.
In terms specific contributions, we develop two new white-box attacks, the Momentum Iterative Method over Expectation (MIME) for attacking single randomized defenses and Auto Expectation Gradient Attack (AE-SAGA for dealing with combination of randomized and non- randomized defenses. We are the first to show the transferability of adversarial examples generated by MIM, APGD, MIME and AE-SAGA on state-of-the-art defenses like FAT, BaRT and TiT.
Lastly, and most importantly, we develop game theoretic framework for determining the optimal attack and defense strategy. Any newly proposed defense or attack can be easily integrated into framework. Using set of state-of-the-an attacks and defenses demonstrate that game theoretic framework can create a compositional defense that achieve a 13.5% increase in robustness on CIFAR-10 1-0 and 33.88% increase in robustness Tiny ImageNet using multi-defense, mixed Nash strategy (as opposed to using the best single defense). Both compostional defenses for CIFAR- 10 and Imag-lm also with higher clean accuracy than the most robust single defenses.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4190-4197, 2020.
Alexandre Araujo, Laurent Meunier, Rafael Pinot, and Benjamin Negrevergne. Advocating for mul- tiple defense strategies against adversarial examples. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 165-177. Sprinnerr 2020.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In International conference on machine learning, pp. 284 293. PMLR, 2018.
Maria-Florina Balcan, Rattana Pukdee, Pradeep Ravikumar, and Hongyang Zhang. Nash equi- libria and pitfalls of adversarial training in adversarial robustness games. arXiv preprint arXiv:2210. /2606, 2022.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten de- tection methods. In Proceedings of the 10th ACM workshop artificial intelligence and security, pp. 3-14, 2017.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference machine learning, pp. 2206- 2216. PMLR, 2020.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Representations, 2020.
Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang. Timothée Masquelier, and Yonghong Tian. Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems, 34:21056-21069. 2021.
Samuel G Finlayson, John D Bowers, Joichi Ito, Jonathan L Zittrain, Andrew L Beam, and Isaac S Kohane. Adversarial attacks on medical machine learning. Science, 16336433)11287-12255 2019.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 2014.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European confer- ence on computer vision, pp. 491-507. Springer, 2020.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. C0 (canadian institute for advanced re- search). URL (ttt::/ wwww://wwwww.ooo es toronto..tooooooo...o edu/-kiiz/cifal html
Trung Le, Anh Tuan Bui, Le Minh Tri Tue, He Zhao, Paul Montague, Quan Tran, and Dinh Phung-
On global-view based defense via adversarial attack and defense risk guaranteed bounds. In Gustau Camps-Valls, Francisco R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 11438- 1 1460. PMLR, 28-30 Mar 2022. URL https:// proceedi ngs .mlg..lr. wwwppprssvvviee. /0l1l//1ecccc html.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial exam- ples and black-box attacks. arXiv preprint arXiv: 161:: 1.11777770. 2016.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
10

---

Under review as a conference paper at ICLR 2023
Kaleel Mahmood, Deniz Gurevin, Marten Dijk, and Phuoung Ha Nguyen. Beware the black- box: On the robustness recent defenses to adversarial examples. Entropy, 23 (310:113359 202
Kaleel Mahmood, Rigel Mahmood, and Marten Van Dijk. On the robustness of vision transformers to adversarial examples. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7838-7847, 2021b.
Kaleel Mahmood, Phuong Ha Nguyen, Lam M. Nguyen, Thanh Nguyen, and Marten Van Dijk. Besting the black-box: Barrier zones for adversarial example defense. IEEE Access, 10:1451- 1474, 2022. doi: A09//SSS 2021 338666.
Pratyush Maini, Eric Wong, and Zico Kolter. Adversarial robustness against the union of multiple perturbation models. In International Conference on Machine Learning, pp. 6640-6650. PMLR, 2020.
Laurent Meunier, Meyer Scetbon, Rafael B Pinot, Jamal Atif, and Yann Chevaleyre. Mixed nash equilibria in the adversarial examples game. In International Conference on Machine Learning. pp. 7677-7687. PMLR, 2021.
John Nash. Non-cooperative games. Annals of Mathematics, 54(2))286-299 1951. ISSN 0003486X. Utth:tt:://wwx jtt.or.org/staaeee1183333
Ambar Pal and René Vidal. game theoretic analysis of additive adversarial attacks and defenses. Advances in Neural Information Processing Systems, 1355, 2020.
Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning, pp. 4p7409944979. PMLR, 2019.
Nicolas Papernt, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.0727) 2016.
Rafael Pinot, Raphael Ettedgui, Geovani Rizk, Yann Chevaleyre, and Jamal Atif. Randomization matters how to defend against strong adversarial attacks. In International Conference on Machine Learning, pp. 7717-7727. PMLR, 2020.
Adnan Qayyum, Muhammad Usama, Junaid Qadir, and Ala Al-Fuqaha. Securing connected & autonomous vehicles: Challenges posed by adversarial machine learning and the way forward. IEEE Communications Surveys & Tutorials, 22(2):998-1026, 2020. doi: 10.1109/C0MS..2000 2975048.
Edward Raff, Jared Sylvester, Steven Forsyth, and Mark McLean. Barrage of random transforms for adversarially robust defense. In 2019 JEEE/CVF Conference on Computer Vision and Pattern doi: 10.1109/CVPR.221990066
Recognition (CVPR), pp. 6521-6530, 2019.
Nitin Rathi and Kaushik Roy. Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems, 2021a.
Nitin Rathi and Kaushik Roy. Diet-snn: low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Transactions on Neural Networks and Learning Systems, 2021b.
Sailik Sengupta, Tathagata Chakraborti, and Subbarao Kambhampati. Mtdeep: boosting the security of deep neural nets against adversarial attacks with moving target defense. In Workshops at the thirty-second AAAI conference on artificial intelligence, 2018.
Chawin Sitawarin, Zachary Gcolan-Strickt and David Wagner. Demystifying the adversarial robust- of random transformation defenses. In International Conference Machine Learning, pp. 20232-20252. PMLR, 2022.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in neural information processing systems, (33), 2020.

---

Under review as a conference paper at ICLR 2023
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Representations, 2019.
Chang Xiao and Changxi Zheng. One man's trash is another man's treasure: Resisting adversarial examples by adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1112-221,22 2020.
Nuo Xu, Kaleel Mahmood, Haowen Fang. Ethan Rathbun, Caiwen Ding, and Wujie Wen. Securing the spike: On the transferabilty and security of spiking neural networks to adversarial examples. arXiv preprint arXiv:2209.088
2022.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent EI Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp. 7472-7482. PMLR, 2019.
Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankan- halli. Attacks which do not kill training make adversarial learning stronger. In International conference on machine learning, pp. 11278 1287. PMLR, 2020.
12


Reviews:
[{"review_id": "uzQtQLYE5n3", "reviewer": "Reviewer_once", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers a set of recent defenses against adversarial attacks on the image domain and uses existing attacks or develops them to capture a game-theoretic interaction between the attacks and defenses. To obtain the utility functions, they evaluate the effectiveness of different attacks on different defenses, thereby conducting a transferability study. Eventually, they consider a mixed strategy over a single-classifier or a less-than-n-ensemble selection for the defender by formulating the game as a zero-sum game and solving for a Nash-equilibria in this game. They show that this is more effective than using single defenses.", "summary_of_the_review": "See above.", "strength_and_weaknesses": "### Things I liked\n\n- I like how the authors concisely described many of the recent defenses, the attacks and highlighted their relevance for this work.\n\n- The authors propose new attacks (eg. MIME) by composing existing attacks, situate its need well in the landscape of existing attacks, and showcase its effectiveness.\n\n### Things that need clarification / improvement\n\n- The study of transferability of attacks on defenses, the formulation of interaction between attacks & defenses, and using game-theoretic equilibrium to come up with a mixed-strategy over defenses is not novel. [1] does all this-- given the white-box threat model, their game-theoretic formulation considers that the attacker may even know the defender's strategy and thus, considers a Stackelberg Equilibria as opposed to a Nash Eq. This paper does the study on more recent attacks and defenses and consider a more relaxed threat-model. Thus, all the 3 novel questions the paper presents on page 1 already have answers. Having said that, I do appreciate the renewed study on recent attacks and defenses but cannot support the novelty claim.\n\n- While the reward function mostly concentrates on the loss function value for the adversarially perturbed examples, it doesn't account for the loss on the non-perturbed examples. I would surely suggest the authors to consider the loss on the latter examples in the reward function, at least for the defender, similar to [1]. Otherwise, the strategy found may over optimize for accuracy/loss on the adversarial examples while losing out on accuracy for the true test distribution.\n\n- Given this is a zero-sum game, the Nash Eq. generates a min-max payoff and can be calculated in polynomial time. Hence the statement \"These linear programs can be solved using weakly polynomial time algorithms like the interior point method Karmarkar (1984), or those developed in Vaidya (1989)\" makes me wonder if the authors are aware of the subtle aspects of their formulation.\n\n- Game_n considers a larger space (that explodes exponentially $~2^n$) of defense actions where the defender can use up to n-classifiers for ensembling. Note that this incurs much larger computational cost as one needs to estimate the utility values for each of the attack-defense pairs using $N$ samples. Hence beyond simply reporting accuracy on the adversarial examples and the test set, the authors should report the time taken to form the game-matrix. Also, the authors should discuss how effective the final strategy becomes as the number of samples used for the utility estimation ($N$), and therefore the cost, increases or decreases.\n\n- Some study on if an attacker can attack the game-theoretic defense strategy is necessary to understand if this too is an effective defense. Based on results in [1], I would assume black-box distillation would not be as effective. Would be interesting to see how this holds for different values of $n$ $GAME_n$ and $N$ in estimating utilities.\n\n[1] Sengupta, S., Chakraborti, T., & Kambhampati, S. (2019). Mtdeep: Moving target defense to boost the security of deep neural nets against adversarial attacks. In International Conference on Decision and Game Theory for Security.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ubJYQVssazc", "reviewer": "Reviewer_ZYXb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a game-theoretic analysis of adversarial robustness. In the adversarial examples game the authors propose, attackers choose an attack method (including both the optimization method and the defense to attack) while defenders choose a defense. This game is connected to the study of transferability of adversarial examples across diverse attacks. With respect to transferability, often attacks which are strong against one attack produce adversarial examples that do not transfer well to another defense. The authors then show that one can determine an optimal randomized strategy for attacker and defender by considering a matrix game where the payoffs are the success rates of how well each attack transfers to each defense (the values called $r_{d, a}$ in the paper). An optimal defense strategy is computed for CIFAR-10 and Tiny-ImageNet given several defenses and it is shown that it has higher robust accuracy than any single one of the defenses.", "summary_of_the_review": "Overall, I found the paper to be an interesting analysis of adversarial transferability and its connection to randomized defenses via game theory. I would encourage the authors to work on the presentation of their ideas so that others can more easily appreciate their results.", "strength_and_weaknesses": "In general, I think the idea of the paper is quite interesting and the theory and experiments are explained clearly. As far as I know, nobody has explored an optimal strategy for randomizing over multiple defenses by using game theory. The experiments seem to support the theoretical analysis and there is a nice result that the randomized strategy is more robust than any single of its members. The connection between the game theoretic analysis and transferability of adversarial examples is also quite interesting.\n\nI think that the main weakness of the paper is that the presentation could be improved. I found that a lot of the paper seemed to focus too much on just the methods, mathematical analysis, and algorithms, and not enough on the motivation for them. This meant that I didn't appreciate the ideas behind the paper until I had more fully internalized the methodology sections. I worry that readers will just read the title, abstract, or introduction and not appreciate all the contributions.\n\nIn particular, I think there might be too much emphasis on the parts of the paper dealing with transferability of adversarial examples. This idea has been explored a lot in the past and a casual reader might assume that the paper is just another study on transferability. Instead, I think the paper might be more impactful if the authors emphasize the game-theoretic analysis and the idea of coming up with better randomized defenses. A title that might be more attention-grabbing could be something like \"Optimal Randomized Adversarial Defenses via Game-Theoretic Analysis of Attack Transferability.\" If the authors focus on the goal of picking a good randomized defense and use the analysis of transferability more as an argument to justify that goal rather than as a primary contribution, I think more people will find the paper novel and interesting. If more space is needed to add more motivation, I think some of the details of the new attacks could be moved to the appendix since I think the game-theory ideas and results are more interesting than the attacks. But I'm just one person so maybe others will think the attack/transfer part of the paper is more interesting!\n\nBesides the presentation, one objection to the method in the paper (and randomized defenses in general) is that a defender could just re-submit the same adversarial example until the defender randomly picks the vulnerable model. For instance, say an attacker is trying to post offensive pictures by fooling a content-filtering system on social media. Then, they could just resubmit the pictures until the content-filtering system randomly picks a vulnerable classifier. It would be good to discuss this limitation somewhere in the paper.\n\nSmaller issues/suggestions\n * On page 9, you say \"For CIFAR-10 instance r* = .573, meaning the worst expected performance of the ensemble against\nthese attacks is to get a robust accuracy of 57.5%.\" Shouldn't it be 57.3%? Also, seems like there is a typo and it should be \"For CIFAR-10 for instance\" or just \"For CIFAR-10\".\n * This seems like a relevant paper to cite, although your approach is definitely quite different: Balcan et al. Nash Equilibria and Pitfalls of Adversarial Training in Adversarial Robustness Games. (Also I realize that it came out after you submitted, so use your judgement as to whether you think it should be cited.)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oKKF0txQ_r", "reviewer": "Reviewer_hEcL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how adversarial examples crafted for one defensive model transfer on other models, and correspondingly how random defensive models help adversarial robustness.", "summary_of_the_review": "As is.", "strength_and_weaknesses": "# Strength \nThe studies topic is an interesting and practical one. According to my knowledge, such topic has not been discussed in the past. The theoretical results seems sufficient to me. However, I have not checked its every detail. \n\n# Weakness\nThe empirical verification is not very convincing nor sufficient. According to the theory established by this work, I expect the author to construct a state-of-the-art defense model by randomly choosing the prediction model from an ensemble of models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dD29LHz16X", "reviewer": "Reviewer_rrfN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers a finite set of attackers and a finite set of defenders and views them in a game-theoretic framework to choose the optimal choices for attacker and defender.\n\nSpecifically, it solves a min-max problem to find the optimal weights for attackers and defender.\n\n\n", "summary_of_the_review": "- The paper is not well-organized and well-written. The notions used are confusing. \n- The theoretical setting is not feasible and practical.\n- The contribution is not clear and significant.\n", "strength_and_weaknesses": "Strength\n-  The paper mentions to many methods in adversarial machine learning.\n-  Some experiments about the behaviors are pretty interesting though not new.\n\nWeaknesses\n-  The paper is not well-organized and well-written. The sections are not really logically linked and coherent each other.\n-  Section 5 is messy with inconsistent notions.\n   - $\\theta(x + a(\\theta, x ,y))$: $\\theta$ is generally for a model parameter and it is strange to write $\\theta(x + a(\\theta, x ,y))$.\n   - The notations $p_a$ and $a$ are confusing because it is easy to think $p_a$ depends on $a$. Similarly for $p_d$ and $d$.\n   - No clear definition for $a_i(U,x,y)$. Here $U$ is a set of defenders, but why $a_i(U,x,y)$ returns a common perturbation for every defender in $U$.  \n-  The assumption about finite number of defenders is not feasible. We might interpret $d_i$ as a defender type (e.g., CNNs, ViTs). However, it seems not sufficiently rich to represent the model parameters. \n- The main experiment in Section 6 is very humble.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject