Towards Evaluating the Robustness of Neural Networks Learned by Transduction
Authors: Jiefeng Chen
Xi Wu
Yang Guo
Yingyu Liang
Somesh Jha
Venue: ICLR 2022 Poster (2022)
Keywords: adversarial robustness
transductive learning
test-time defense
dynamic defense
attacking model spaces

Abstract: There has been emerging interest in using transductive learning for adversarial robustness (Goldwasser et al., NeurIPS 2020; Wu et al., ICML 2020; Wang et al., ArXiv 2021). Compared to traditional defenses, these defense mechanisms "dynamically learn" the model based on test-time input; and theoretically, attacking these defenses reduces to solving a bilevel optimization problem, which poses difficulty in crafting adaptive attacks. In this paper, we examine these defense mechanisms from a principled threat analysis perspective. We formulate and analyze threat models for transductive-learning based defenses, and point out important subtleties. We propose the principle of attacking model space for solving bilevel attack objectives, and present Greedy Model Space Attack (GMSA), an attack framework that can serve as a new baseline for evaluating transductive-learning based defenses. Through systematic evaluation, we show that GMSA, even with weak instantiations, can break previous transductive-learning based defenses, which were resilient to previous attacks, such as AutoAttack (Croce and Hein, ICML 2020). On the positive side, we report a somewhat surprising empirical result of "transductive adversarial training": Adversarially retraining the model using fresh randomness at the test time gives a significant increase in robustness against attacks we consider.

Published as a conference paper at ICLR 2022
TOWARDS EVALUATING THE ROBUSTNESS OF NEURAL NETWORKS LEARNED BY TRANSDUCTION
Jiefeng Chen Wu Yang Guo
Yingyu Liang
Somesh Jha 1.3
University of Wisconsin-Madise 2 Google XaiPient (jiefeng, yguo, yliang, jha] wisc. edu
wuxi wuxi wixi@google.com
ABSTRACT
There has been emerging interest in using transductive learning for adversarial robustness (Goldwasser et al., NeurIPS 2020; Wu et al., ICML 2020; Wang et al., ArXiv 2021). Compared to traditional defenses, these defense mechanisms "dynamically learn" the model based on test-time input; and theoretically, attacking these defenses reduces solving bilevel optimization problem, which poses difficulty in crafting adaptive attacks. In this paper, we examine these defense mechanisms from a principled threat analysis perspective. We formulate and analyze threat models for transductive-leaniin important subtleties. We propose the principle of attacking model space for solving bilevel attack objectives, and present Greedy Model Space Attack (GMSA), an attack framework that can serve as new baseline for evaluating transductive- learning based defenses. Through systematic evaluation, even with weak instantiations, can break previous transductive-learnin based defenses, which positive side, we report a somewhat surprising empirical result of "transductive adversarial training": Adversarially retraining the model using fresh randomness the test time gives a significant increase in robustness against attacks we consider.
based defenses, and point out
show that GMSA,
resilient previous attacks, such as Atto-Attak.. On the
INTRODUCTION
Adversarial robustness of deep learning models has received significant attention in recent years (see Kolter& & Madry (2018) and references therein). The classic threat model of adversarial robustness considers an inductive setting where a model is learned at the training time and fixed, and then at the test time, an attacker attempts to thwart the fixed model with adversarially perturbed input. This gives rise to the adversarial training (Madry et al., 2018; Sinha et al., 2018; Schmidt et al., 2018; Carmon et al., 2019) to enhance adversarial robustness.
Going beyond the inductive threat model, there has been emerging interest in using transductive learning (Vapnik, 1998)' for adversarial robustness (Goldwasser et al., 2020; Wu et al., 2020b; Wang et al., 2021). In essence, these defenses attempt leverage batch of test-time inputs, which is common for ML pipelines deployed with batch predictions (bat, 2021), to learn an updated model. The hope is that this "test-time learning" may be useful for adversarial robustness since the defender can adapt the model to the perturbed input from the adversary, which is distinct from the inductive threat model where model is fixed after training.
This paper examines these defenses from a principled threat analysis perspective. We first formulate and analyze rigorous threat models. Our basic I-round threat model considers a single-round game between the attacker and the defender. Roughly speaking, the attacker uses an objective maxy'eN(V) La (T(")., V') (formula (2)), where V is the given test batch, N(V) is neighborhood around V, Ln, is loss function for attack gain, I is the ctasduutivv-learminn based defense, and U' = -'lx, the projection of V" to features, is the adversarially perturbed data for breaking r. This
Our code is available https: / htwthub.bbuuttgggg/gg com/ f eeval-transee iireedduttiiieieee robus tness. We note that this type of defense goes under different names such as "test-time adaptation" or "dynamic defenses". Nevertheless, they all fall into the classic transductive learning paradigm (Vapnik, 1998). which attempts to leverage test data for learning. We thus call them transductive- based defenses. The word "transductive" is also adopted Goldwasser al. (2020).

---

Published as a conference paper at ICLR 2022
objective is transductive as U², the attacker's 's output, appears in both attack (V' in La) and defense (U" in I). We extend this threat model to multiple rounds, which is necessary when considering DENT (Wang et al., 2021) and RMC (Wu et al., 2020b). We point out important subtleties in the modeling that were unclear or overlooked in previous work.
We then study adaptive attacks, that is leverage the knowledge about I to construct attacks. Com- pared to situations considered in BPDA (Athalye et al., 2018), a transductive learner is even further from being differentiable, and theoretically the attack objective is a bilevel optimization (Colson et al., 2007). To address these difficulties, our key observation is to consider the transferability of adversarial examples, and consider a robust version of (2): maxy" minTEN La (IU),, V')
(formula (6)), where we want to find single attack set U" to thwart family of models, induced by U "around" U" This objective relaxes the attacker-defender constraint, and provides more information in dealing with nondifferentiabiliy.. To solve the robust objective, we propose Greedy Model Space Attack (GMSA), a general attack framework which attempts to solve the robust objective in a greedy manner. GMSA can serve as new baseline for evaluating transductive-leaninn
based defenses.
We perform a systematic empirical study on various defenses. For RMC (Wu et al., 2020b), DENT (Wang et al., 2021), and URejectron (Goldwasser et al., 2020), show that even weak instantiations of GMSA can break respective defenses. Specifically, for defenses based on adversari- ally training, we reduce the robust accuracy to that of adversarial training alone. We note that, under adaptive attack for the inductive threat model,
AutoAttack (Croce & Hein, 2020a), the state-of-the some of these defenses have claimed to achieve substantial improvements compared adversarial training alone. For example, Wang et al. show that DENT can improve the robustness of the state-of-the-art adversarial training defenses by more than 20% absolutely against AutoAttack on 10. However, under our adaptive attacks, DENT only has minor improvement: less than 3% improvement over adversarial training alone. Our results thus demonstrates significant differences between attacking transductive-leaming based defenses and attacking in the inductive setting, and significant difficulties in the use of transductive learning to improve adversarial robustness. On the positive side, we report a somewhat surprising empirical result of transductive adversarial training: Adversarially retraining the model using fresh private randomness on a new batch of test-time data gives significant increase in robustness against all of our considered attacks.
RELATED WORK
Adversarial robustness in the inductive setting, Many attacks have been proposed to evaluate the adversarial robustness of the defenses in the inductive setting where the model is fixed during the evaluation phase (Goodfellow et al., 2015; Carlini & Wagner, 2017; Kurakin et al., 2017; Moosavi- Dezfooli et al., 2016; Croce & Hein, 2020b). Principles for adaptive attacks have been developed in Tramer et al. (2020) and many existing defenses are shown to be broken based on attacks developed from these principles (Athalye et al., 2018). A fundamental method to obtain adversarial robustness in this setting is adversarial training (Madry et al., 2018; Zhang et al., 2019). A state-of-the-an attack in the inductive threat model is Attakt (Croce & Hein, 2020a).
Adversarial robustness via test-time defenses. There have been various work which attempt to improve adversarial robustness by leveraging test-time data. Many of such work attempt to "sanitize" test-time input using a non-differentiable function, and then send it to a pretrained model. Most of these proposals were broken by BPDA (Athalye et al., 2018). To this end, we note that a research agenda for "dynamic model defense" has been proposed in Goodfellow (2019).
Adversarial robustness using transductive learning. There has been emerging interesting using transductive learning to improve adversarial robustness. In view of "dynamic defenses", these proposals attempt to apply transductive learning to the test data and update the model, and then use the updated model to predict on the test data. In this work we consider three such work (Wu et al., 2020b; Wang et al., 2021; Goldwasser et al., 2020).
3 PRELIMINARIES
Let F be a model, and for a data point (z, y) E X x V, a loss function E(F:z, y) gives the loss of F on the point. Let V be set of labeled data points, and let L(F,V) M Eeeyee E(Fiz, y) denote
2

---

Published as a conference paper at ICLR 2022
the empirical loss of F on V. For example, if we use binary loss C (F; y)
1[F(x) + y). this
gives the test error of F on V. We use the notation Vix to denote the projection of V to its features, that is {z,, y1) );)];= + [=1. Throughout the paper,
N()) to denote neighborhood function for perturbing features: That is, N(x) = {a' d(æ' x) << } is a set of examples that are close in terms of distance metric d (e.g.. d(x'
|| x pl). Given U {zr 1 let N (U) = {</ d(z; x;) < E, O,... m). Since labels are not changed for adversarial
examples, we also use the notation (V) to denote perturbations of features, with labels fixed.
4 MODELING TRANSDUCTIVE ROBUSTNESS
In this section we formulate and analyze threat models for transductive defenses. We first formulate a threat model for single-round game between the attacker and the defender. We then consider exten- sions of this threat model to multiple rounds, which are necessary when considering DENT (Wang et al., 2021) and RMC (Wu et al., 2020b), and point out important subtleties in modeling that were not articulated in previous work. We characterize previous test-time defenses using our threat models.
1-round game. In this case, the adversary "intercepts" a clean test data V (with clean features U === Vlx. and labels Vly), adversarially perturbs it, and sends a perturbed features U' to the defender. The defender learns a new model based on U". A referee then evaluates the accuracy of the adapted model on U². Formally:
Definition 1 (I-round threat model for transductive adversarial robustness). Fix an adversarial perturbation type (e.g.. Eo perturbations with perturbation budget E). Let Px.y be data generation distribution. The attacker is an algorithm A, and the defender is a pair of algorithms (T,1") where T is supervised learning algorithm,, and T is transductive learning algorithm. A (clean) training set D is sampled ii.dd from Px.y. A (clean) test set V is sampled i.i.d. from Px.y-
(Training time, defender] The defender trains optional base model F T(D), using the labeled source data D.
[Test time, attacker] The attacker receives V, and produces an (adversarial) unlabeled dataset U':
1. On input 1, F, D, and V, A. perturbs each point (z. y) E V to (x', .)) (subject to the agreed attack rype), giving V = A(T, F, D, V) (that is, V E N())).
2. Send U = V (the feature vectors Vf' to the defender.
[Test time, defender] The defender produces a model F ((F,D,U")
Multi-round games. The extension of --runnd games to multi-round contains several important considerations that were implicit unclear in previous work, and is closely related to what it means by adaptive attacks. Specifically:
Private randomness. Note that I' uses randomness, such random initialization and random restarts? in adversarial training. Since these randomness are generated after the attacker's 's move, they are treated private randomness, and not known to the adversary.
Intermediate defender states leaking vs. Non-leaking. In a multi-roumd game, the defender may maintain states across rounds. For example, the defender may store test data and updated models from previous rounds, and use them in new round. If these intermediate defender states are "leaked" to the attacker, we call it intermediate defender states leaking, or simply states leaking, otherwise call it non states-leaking, or simply non-leaking. Note that the attacker cannot simply compute these information by simulating on the training and testing data, due to the use of private randomness. We note that, however, the initial pretrained model is assumed to be known by the attacker. The attacker can also of course maintain arbitrary states, and are assumed not known to the defender.
Adaptive vs. Non-adaptive. Because transductive learning happens after the attacker produces U', the attacker may not able to directly attack the model I' produced. Nevertheless, the attacker assumed to have full knowledge of the transductive mechanism I, except the private randomness. In this paper we call an attack adaptive if it makes explicit use of the knowledge T.
Naturally ordered Adversarially ordered. Both RMC and DENT handle batches of fixed sizes. An intuitive setup for multi-round game is that the batches come in sequentially, and the attacker
When perturbing data point during adversarial training, one starts with random point in the neighborhood.
3

---

Published as a conference paper at ICLR 2022
must forward perturbed versions of these batches in the same order to the defender, which we call the "naturally ordered" game. However, this formulation does not capture an important scenario: An adversary wait and pool large amount of test data, then chooses "worst-case" order of perturbed data points, and then sends them in batches one at a time for adaptation in order to maximize the breach. We call the latter "adversarially ordered" game. We note that all previous work only considered naturally-orders ordered game, which gives the defender more advantages, and is thus our game is evaluated for DENT in Appendix A.7.
focus in the rest of the paper. Adversarially-ordees
Modeling capacity of our threat models. Our threat models encompass a large family of defenses. For example. without using I, the threat model degenerates the classic inductive threat model. Our threat models also capture various "test-time defenses" proposals (e.g., those broken by the BPDA (Athalye et al., 2018)), where a mionddiffereniiabee" function which "sanitizes" " the test data, instead of updating the model, before sending them to a fixed pretrained model. Therefore, in partieular, these proposals are not transductive-leane learning based. Below we describe previous defenses which we study in the rest of this paper, where I' is indeed transductive learning.
Example 1 (Runtime masking and cleansing). Runtime masking and cleansing (RMC) (Wu et al., 2020b) is a recent transductive-lerriigy defense. For RMC, the defender is stateful and adapted from the model learned in the last round, single test point (UU) = 1): The adaptation objective is F = arg minp ((a.))eN' L(F, vyy, where I is the test time feature point, and N' N') is the set of examples in the adversarial training dataset D' that are top-K nearest to I in a distance measure. RMC paper considered two attacks: (1) Transfer attack, which generates perturbed data by attacking the initial base model, and (2) PGD-skip attack, which at round p 1, runs PGD attack on the model learned at round p. In our language, transfer attack is stateless (i.e. the adversary maintains state) and non- -da-adeptive, PGD-ski skip attack is state-leaking, but still non-adaptive.
Example 2 (Defensive entropy minimization (DENT (Wang el., 2021))). DENT adapts the model using test input, and can work with any training-tine time learning procedure. The DENT defender is stateless: It always starts the adaptation from the original pretrained model, fixed at the training time. During the test-time adaptation, only the affine parameters in batch normalization layers of the base model are updated, using entropy minimization with the information maximization regularization. In this paper, we show that with strong adaptive attacks under the naturally ordered setting, we are able to reduce the robustness to be almost the same as that of static models (see Section 6). Further, under the adversarially ordered setting, we can completely break DENT.
Example 3 (Goldwasser et al.'s transductive threat model). While seemingly our threat model is quite different from the one described in Goldwasser et al. (2020), one can indeed recover their threat model naturally as a I-round game: First, for the perturbation type, we simply allow arbitrary perturbations in the threat model setup. Second, we have a fixed pretrained model F, and the adaptation algorithm I' learns a set S which represents the set of "allowable" points (so Fls yields a predictor with redaction, namely it outputs 1 for points outside of S). Third, we define two error functions (5) and (6) in Goldwasser et al. (2020):
U (rI(Fls. = 1 {2 U' +
U U\S |0|
rej(s)
where f is the ground truth hypothesis. The first equation measures prediction errors in U' that passed through S, and the second equation measures the rejection rate of the clean input. The referee evaluates by measuring two errors: L(Fls, V') = (ertu reju
ADAPTIVE ATTACKS IN ONE ROUND
In this section we study a basic question: How to perform adaptive attacks against a transductive- learning based defense in one round? Note that, in each round of multi-round game, independent batch of test input U is sampled, and the defender can use transductive learning to produce a model specifically adapted to the adversarial input U", after the defender receives it. Therefore, it is of fundamental interest to attack this ad-hoc adaptation. We consider white-bor attacks: The attacker knows all the details of I, except private randomness, which is sampled after the attacker's move.
We deduce a principle for adaptive attacks in one round, which we call the principle of attacking model space: Effective attacks against transductive defense may need consider atlacking a set
4

---

Published as a conference paper at ICLR 2022
of representative models induced in the neighborhood of We give concrete instantiations of this principle, and show in experiments that they break previous transductive-leamming based defenses.
Attacks in multi-round. If the transductive- repeat one-round attack multiple times. If it is stateful, then we need to consider state-lcaking setting non-leaking setting. For all experiments in Section 6, we only evaluate non-leaking setting, which is more challenging for the adversary.
based defense is stateless, then simply
5.1
GOAL OF THE ATTACKER AND CHALLENGES
To start with, given a defense mechanism r, the objective of the attacker can be formulated as:
max VV
where La the loss function of the attacker. We make some notational simplifications: Since D is constant, in the following we drop it and write r(U"). Also, since the attacker does not modify the labels in the threat model, abuse the notation and write the objective
La LT((")))
(3)
A generic attacker would proceed iteratively follows: It starts with the clean test set V, and generates sequence of (hopefully) increasingly stronger attack sets ( = Vlx, U,, (7) (U(*) must satisfy the attack constraints at U, such as fo bound). We note several basic but important differences between transductive attacks and inductive attacks in the classic minimax threat model:
(D1) T(U") is not differentiable. For the scenarios we are interested in, r is an optimization algorithm to solve an objective F* E arg minp (F, D. This renders (3) into bilevel optimization problem (Colson et al., 2007):
max V'EN(V) L.(F", V') subject to: F E arg La(F, D,U'),
(4)
F
In these cases, I' is in general not (in fact far from) differentiable. A natural attempt is to approximate I' with differentiable function, using theories such as Neural Tangent Kernels (Jacot et al., 2018). Unfortunately no existing theory applies to the transductive learning, which deals with unlabeled data U' (also, as we have remarked previously, tricks such BPDA (Athalye al., 2018) also does not apply because transductive learning is much more complex than test-time defenses considered there).
(D2) U appears in both attack and defense. Another significant difference is that the attack set U also appears as the input for the defense (i.e. I(U")). Therefore, while it is easy to find U to fail IT)) for any fixed U, it is much harder to find good direction to update the attack and converge to an attack set U* that fails an entire model space induced by üself: T(U*).
(D3) ((U") can be random variable. In the classic minimax threat model, the attacker faces fixed model. However, the output can be random variable of models due to its private randomness, such as the case of Randomized Smoothing (Cohen et al., 2019). In these cases, successfully attacking a single sample this random variable does not suffice.
Algorithm 1 FIXED POINT ATTACK (FPA)
Require: A transductive learning algorithu I, an optional training dataset D. a clean test set V, an initial model F(0), and an integer parameter 2 (the number of iterations).
1: for i 0, 1,..,, T do 2: Attack the model ohtained in the last iteration get the perturbed set:
V4
arg max La (P(() V') VEN(V)
(5)
where L, a loss function. Set ( (+)
lx
3: Run the transductive learning algorithm I" to get the next model: F(+1) ((D,U0)
4: end for
5: Select the best attack set (t()) as k = arg maxosis L(F(+1), v()).
6: return (i))
5

---

Published as a conference paper at ICLR 2022
Fixed Point Attack: A first attempt. We adapt previous literature for solving bilevel optimization in deep learning setting (Lorraine & Duvenaud, 2018) (designed for supervised learning). The idea is simple: iteration i + 1, we fix (769) and model space F()) )( and construct (7(++1) to fail it. We call this the Fixed Point Attack (FPA) (Algorithm I), one hopes that this process converges to a good fixed point U*. Unfortunately, we found FPA to be weak in experiments. The reason is exactly (D2): ( failing F(3) may not give any indication that it can also fail p(++) induced by itself. Note that transfer attack is special case of FPA by setting T O.
5.2 STRONG ADAPTIVE ATTACKS FROM ATTACKING MODEL SPACES
To develop stronger adaptive attacks, we consider key property of the adversarial attacks: The transferability of adversarial examples. Various previous work have identified that adversarial examples transfer (Tramer et al., 2017; Liu et al., 2016), even across vastly different architectures and models. Therefore, if U' is good attack set, would expect that U' also fails I(U) for U close to U.. This leads to the consideration of the following objective:
max min La UT()) U' UEN(U")
(6)
where N-)) is neighborhood function (possibly different than N). It induces family of models (T(T) U E CUU)), which we call a model space. (in fact, this can be a family of random variables of models) This can be viewed as a natural robust version of (3) by considering the transferability of U². While this is seemingly even harder to solve, it has several benefits: (1) Considering model space naturally strengthens FPA. FPA naturally falls into this formulation as a weak instantiation where we consider a single U = U6), Also, considering a model space gives the attacker more information in dealing with the nondifferentiabiiity of I' (D1). (2) It relaxes the attacker-defende constraint (D2). Perhaps more importantly. for the robust objective, we no longer need the same U to appear in both defender and attacker. Therefore it gives natural relaxation which makes attack algorithm design easier.
In summary, while "brittle" U that does not transfer may indeed exist theoretically, their identification can be challenging algorithmically, and its robust variant provides a natural relaxation considering both algorithmic feasibility and attack strength. This thus leads us the following principle:
The Principle of Attacking Model Spaces. An effective adaptive attack against a transductive transductive-learning based defense may need to consider a model space induced by proper neighborhood of U.
Algorithm 2 GREEDY MODEL SPACE ATTACK (GMSA)
Require: A transductive learning algorithm 1' an optional training dataset D, a clean test set V, an initial model F(0), and integer parameter T 2 (the number of iterations).
1: for i = 0, 1,..., T do
2: Attack the previous models to get the perturbed set:
vii arg max LOMBA LOMMA(( ((FU))) (3 V") V'EN(V)
(7)
where LOMSA is a loss function. Set (+()
V Ix
3: Run the transductive learning algorithm I to get the next model: ((++1) = T(D,U())
4: end for
Select the best attack (K) as k arg maxp L(F(+)), v(9).
6: return (k).
An instantiation: Greedy Model Space Attack (GMSA), We give a simplest possible instantiation of the principle, which we call the Greedy Model Space Attack (Algorithm 2). experiments we use this instantiation to break previous defenses. In this instantiation, the family of model spaces to consider is just all the model spaces constructed in previous iterations. LoMSA (f ((F() Jj=o= V' loss function that the attacker uses to attack the history model spaces. We consider two in- stantiations: (1) LVGG ((FG)) = V') = * Σj (F'D), V'), (2) L (( } V') =
MIN mingsjs: La (FU), V'), where LGMSA gives attack algorithm GMSA-AVG, and GMSA gives attack algorithm GMSA-MIN. We solve (7) via Projected Gradient Decent (PGD) (the implementation
details of GMSA can be found in Appendix 1.1.3).

---

Published as a conference paper at ICLR 2022
Accuracy
Robustness
Base Model
Dataset
Static
RMC
Static
RMC
AA
AA
PGD
FPA
GMSA-AVG
GMSA-MIN
Standard
99.50
99.00
0.00
97.70
98.30
0.60
0.50
1.10
MNIST
Madry et al.
99.60
97.00
87.70
95.70
96.
59,50
61.40
58,80
Standard
94,30
93.10
0.00
94.20
97.60
8.50
8.00
8.10
CIFAR-10
Madry et al.
83.20
90..0
44.30
77.90
71.70
40.80
42.50
39.60
Table 1: Results of evaluating RMC. We also evaluate the static base model for comparison. Bold numbers are worst results.
Accuracy
Robustness
Statie
DENT
Model
Static
DENT
AA
DENT-AA
AA
FGD
TPA
GMSA-AVG
GMSA-MIN
er (2020a)
85.70
86.10
58.00
78.80
64.40
59.50
59.30
59.60
59.60
Carmon ct al. (2019)
88.00
87.40
57.30
80.10
61.70
58,40
58,40
58,50
58.50
Schwag et al. (2020)
87.30
86,90
54.90
76.50
59.60
55.80
55,80
55.80
55.80
Wang ot (2020)
86.60
85.60
53.60
75.90
61.30
55,90
55,80
36.10
56,10 10
Hendryeks et (2019)
85.80
85.50
$1.80
77.20
58.40
54.30
$4.40
54,20
54,20
Wong et al. (2020)
81.20
81.00
42.40
69.70
48.90
44.10
44.30
44,50
44.
Ding ct al. (2020)
82,40
82,40
39.70
62.80
44.80
39.90
39.40
39.10
39.20
Table 2: Results of evaluating DENT on CIFAR-10. Bold numbers are worst results.
We also evaluate the static base model for comparison.
EMPIRICAL STUDY
This section evaluates various transductive-lcarnigg based defenses. Our main findings are: (1) The robustness of existing transductive defenses like RMC and DENT is overestimated. Under our evaluation framework, those defenses either have little robustness or have almost the same robustness as that of the static base model. To this end, we note that while AutoAttack is effective in evaluating the robustness of static models, it is not effective in evaluating the robustness of transductive defenses. contrast, our GMSA attack a strong baseline for attacking transductive defenses. (2) We experimented novel idea of applying Domain Adversarial Neural Networks (Ajakan et al., 2014), an unsupervised domain adaptation technique (Wilson & Cook, 2020), as a crnsductivv-learming based defense. We show that DANN has nontrivial and even better robustness compared to existing work, under AutoAttack, PGD attack, and FPA attack, even though it is broken by GMSA. (3) We report somewhat surprising phenomenon transductive adversarial training: Adversarially retraining the model using fresh private randomness on a new batch of test-time data gives a significant increase in robustness, against all of our considered attacks. (4) Finally, we demonstrated that URejectron, while enjoying theoretical guarantees in the bounded- VC dimensions situation, can be broken in natural deep learning settings.
Evaluation framework. For each defense, we report accuracy and robustness. The accuracy is the performance on the clean test inputs, and the robustness is the performance under adversarial attacks. The robustness of transductive defenses is estimated using Auto/Atack (AA)², PGD attack, FPA, GMSA-MIN and GMSA-AVG, We use PGD attack and AutoAttack in the transfer attack setting for the transductive defense: We generate adversarial examples by attacking static model (e.g. the base model used by the transductive defense), and then evaluate the transductive defense on the generated adversarial examples. Accuracy and robustness of the static models are also reported for comparison. We always use AutoAttack to estimate the robustness of static models since it is the state-of-the the-art art for the inductive setting. For all experiments, the defender uses his own private randomness, which is different from the one used by the attacker. Without specified otherwise, all reported values are percentages. Below we give details. Appendix A gives details for replicating the results.
Runtime Masking and Cleansing (RMC (Wu et al., 2020b)). RMC adapts the network at test time, and was shown achieve state-of-the-art robustness under several attacks that are unaware of the defense mechanism (thus these attacks are non-adaptive according to our definition). We follow the setup Wu al. (2020b) perform experiments on MNIST and CIFAR-10 evaluate the
We use the standard version of AutoAttack: https: com/ Fra31 aatto
7

---

Published as a conference paper at ICLR 2022
Accuracy
Robustness
Dataset
Standard
Madry et al.
DANN
Standard
AA
Madry
AA
AA
PGD
FPA
DANN
GMSA-AVG
GMSA-MIN
MNIST
99.42
99.16
99.27
0.00
88.92
97.59
96.66
96.81
79.37
6.17
CIFAR-10
93.95
86.06
0.00
66.61
60.54
53.98
5.53
8.56
Table 3: Results of evaluating DANN. Bold numbers are worst results.
robustness of RMC. On MNIST, we consider Loo norm attack with 0.3 and on CIFAR-10, 10, we consider Loo norm attack with e = 8/255. The performance RMC is evaluated on sequence of test points x (2)))... .... (x) randomly sampled from the test dataset. So we have a n-round game. The FPA and GMSA attacks are applied on each round and the initial model F(O) used by the attacks at the (k: + 1) round is the adapted model (with calibration in RMC) obtained at the k-th round. To save computational cost, we set n = 1000. The robustness of RMC is evaluated on a sequence of adversarial examples " (") generated by the attacker on the sequence of test points x²)),.. ... (()), We evaluate the robustness of RMC in the non-state leaking setting with private randomness (both are in favor of the defender).
Results. The results are in Table 1. RMC with the standard model is already broken by FPA attack (weaker than GSMA). Compared to the defenns-unawwre AutoAttack. our GMSA-AVG attack reduces the robustness from 77% to 0.50% on MNIST and from 94.20% to 8.00% on CIFAR-10. 10. Further, RMC with adversarially trained model actually provides worse adversarial robustness than using adversarial training alone. Under our GMSA A MMIN attack, the robustness is reduced from 96% to 58.80% on MNIST and from 71.70% to 39.60% on CIFAR-10.
Defensive Entropy Minimization (DENT (Wang et al., 2021)). DENT performs test-time adap- tation, and works for any training-ting time learner. was shown that DENT improves the robustness of the state-of- -fate-of-the-arr adversarial training defenses by 20+ points absolute against AutoAttack on CIFAR-10 under Tox norm attack with = 8/255 (DENT is implemented as a model module, and AutoAttack is directly applied to the module, and we denote this as DENT-AA). Wang et al. also considers adaptive attacks for DENT, such as attacking the static base model using AutoAttack to generate adversarial examples, which is the same as the AutoAttack (AA) in our evaluation.
We evaluate the best version of DENT, called DENT+ Wang et al., under their original settings on CIFAR-10: DENT is combined with various adversarial training defenses, and only the model adaptation is included without input adaptation. The model is adapted sample-wise for six steps by AdaMod (Ding et al., 2019) with learning rate of 0.006, batch size of 128 and no weight decay. The adaptation objective is entropy minimization with the information maximization regularization. To save computational cost, we only evaluate on 1000 examples randomly sampled from the test dataset. We consider Los norm attack with 8/255. We design loss functions for the attacks to generate adversarial examples with high confidence (See Appendix A.3 for the details).
Results. Table 2 shows that both DENT-AA and AA overestimate the robustness of DENT. Our PGD attack reduces the robustness of DENT to be almost the same as that of the static defenses. Further, our FPA, GMSA-AVG and GMSA-MIN have similar performance as the PGD attack. The results show that AutoAttack not effective in evaluating the robustness of transductive defenses.
Domain Adversarial Neural Network (DANN (Ajakan et al., 2014)). We consider DANN as transductive defense for adversarial robustness. We train DANN on the labeled training dataset D (source domain) and unlabeled adversarial test dataset U (target domain), and then evaluate DANN on U'. For each adversarial set U', we train a new DANN model from scratch. We use the standard model trained on D as the base model for DANN. We perform experiments on MNIST and CIFAR-10 10 to evaluate the adversarial robustness of DANN. On MNIST. we consider Loc norm attack with 0.3 and on CIFAR-10. 10. consider Lo norm attack with 8/255.
Results. Table 3 shows that DANN has non-trivia ann-trivial robustness under AutoAttack, PGD attack and FPA attack. However, under our GMSA attack, DANN has little robustness.
Transductive Adversarial Training (TADV). We consider simple but novel wransuutive-Ieariing based defense called transductive adversarial training: After receiving set of examples at the test time, we always adversarially retrain the model using fresh randomness. The key point of this transduction is that private randomness is sampled after the attacker's move, and the attacker
8

---

Published as a conference paper at ICLR 2022
Accuracy
Robustness
Dataset
Madry al.
TADV
Madry et al.
TADV
AA
AA
FGD
FPA
GMSA-AVG
GMSA-MIN
MNIST
99.01
99.05
86.61
96.07
96,48
95.47
94.27
95,48
CIFAR-10
87.69
88.51
45.29
72.12
59.05
58.64
54.12
$7.77
Table 4: Results of evaluating TADV. Bold numbers are worst results.
URejectron under FGD Attack
URejectron under CW Attack
URejectron under image Corruption
Eir on
Ert un
ErT
0.8
Rej
0.8
0.8
0.6
0.2
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.4
0.0
1.0
Relection on
Rejection rate on
Rejection rate on.
(a)
(b)
(c)
Figure 1: URejectron in three settings. contains "normal" examples which the elassifier can have high accuracy. @ includes and consists of a mix of 50% "normal" examples and 50% adversarial examples. In (a), the normal examples are clean test inputs and the adversarial examples are generated by PGD attack. In (b), the "normal" examples are still clean test inputs but adversarial examples are generated by CW attack. In (c), the "normal" examples are generated by image corruptions (adversarial examples are generated by PGD attacks).
cannot directly attack the resulting model in the inductive case. Specifically, for GMSA attacks, we attack (with loss LAYG or MIN an ensemble of T = 10 models, adversarially trained with independent randomness, and generate perturbed test set U". Then adversarially train another model from scratch with independent randomness, and check whether U' transfers to the new model (this thus captures the scenario described earlier). Somewhat surprisingly, show that U does not transfer very well, and the TADV improves robustness significantly.
Results. Table shows that transductive adversarial training significantly improves the robustness of adversarial training (Madry et al., 8). On MNIST, the robustness is improved from 86.61 % to 94.27%. On CIFAR-10, the robustness is improved from 45.29% to 54.12%.
URejectron in deep learning settings. URejectron performs transductive learning for defense, and has theoretical guarantees under bounded VC dimension case. We evaluated URejectron on GTSRB dataset using ResNet18 network. We used the same implementation by Goldwasser et al..
Results. Figure ((a) shows that for transfer attacks generated by PGD attack (Madry al., 2018), URejectron can indeed work as expected. However, by using different attack algorithms, such as CW attack (Carlini & Wagner, 2017), we observe two failure modes: (1) Imperceptible adversarial perturbations that slip through. Figure 1(b) shows that one can construct adversarial examples that are very similar to the clean test inputs that can slip through their URejectron construction of (in the deep learning setting), and cause large errors. (2) Benign perturbations that get rejected. Figure I(c) shows that we can generate "benign" perturbed examples using image corruptions, such as slightly increased brightness, but URejectron rejects all.
7 CONCLUSION
In this paper, we formulate threat models for transductive defenses and propose an attack frame- work called Greedy Model Space Attack (GMSA) that can serve as a new baseline for evaluating transductive defenses. We show that GMSA can break previous transductive defenses, which were resilient to previous attacks such as AutoAttack. On the positive side, we show that transductive adversarial training gives significant increase in robustness against attacks consider. For the future work, one can explore transductive defenses that can be robust under our GMSA attacks, and can also explore stronger adaptive attacks that are effective in evaluating transductive defenses.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
The work is partially supported by Air Force Grant FA9550- 1100-18-1-0166 the National Science Foun- dation (NSF) Grants CCT-FMidf-1836978 IIS-2008559, and CCF-16521442 and ARO grant number W91 1991N--11 1--1--77----- 177-10005. Jiefeng Chen and Somesh Jha are partially supported by the DARPA-GARD problem under agreement number 885000.
CC--Frontiers-1804666 CCF-2046710 10
ETHICS STATEMENT
We believe that our work gives positive societal impact in the long term. In the short term potentially some services deploying existing transductive defenses may be broken by adversaries who leverage our new attacks. However, our findings give a necessary step to identify transductive defenses that really work and deepened our understanding of this matter. It also gives positive impact by advancing the science for trustworthy machine learning and potentially how deep transductive learning works.
9 REPRODUCIBILIIY STATEMENT
We have included enough experimental details to ensure reproducibility in Section 6 and Appendix A. Also, the source code for the experiments is submitted as supplemental materials.
REFERENCES
Online versus batch prediction. https: //cludd 2o0o://...oo/aa-ppllttffooorrrre on iine vs batch- predi ct on, 2021.
Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. Dommain-adversarial ncural networks. stat, 1050:15, 2014.
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In Jennifer G. Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssan, Stockholm, Sweden, July I0-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 274-283. PMLR, 2018
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy, SP 2017, San Jose, CA, USA, May 22-26, 2017, pp. 39 57, IEEE Computer Society, 2017, doi: 10. 10.1100/SP.201..44 49. URL https: // doi. org/ / 1109 2017 49.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C. Duchi, and Percy Liang. Unlabeled data improves adversarial robustness. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrlPS 2019, 8-14 December 2019, Vancouver. BC. Canada, pp. 11190- 12011, 2019.
Ching- Yao Chuang, Antonio Torralba, and Stefanie Jegelka. Estimating generalization under distribution shifts via domain invariant representations. In Proceedings of the 37th International Conference on Machine Learn- ing. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1984-1994. PMIR, 2020. URL http:: proced ngs m press / vl1 9/ chuang20a hhtml.
Jeremy M. Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing. In Kamalika Chaudhuri and Ruslan Salakhutdinov (cds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 1310-1320. PMLR, 2019.
Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization, 2007.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fic free attacks. In /CML, 2020a.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fiee free attacks. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020. 13-18 July 2020, Virtual Event, volume 1 19 of Proceedings of Machine Learning Research, pp. 2206-2216. PMLR, 2020b. http:t ppocced ngs. ml r. www.prss/veee v 9 croce20b. html.
10

---

Published as a conference paper at ICLR 2022
Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. MMA training: Direct imput space margin maximization through adversarial training. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: http::///cpcrreviwww net/ forum? wwmmmlddddkeeeae dod-kkeyXBCF...
Jianbang Ding, Xuancheng Ren, Ruixuan Luo, and Sun. An adaptive and momental bound method for stochastic learning. CoRR, Abss/910..1249, 2019. URL http: http///axii// org/ hat///b//1999 /1910. 12249.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky.
training of neural networks. J. Mach. Learn. Res., 17:59:1-59:35 2016. URL http: /mm1 r.. .rrg/ papers / 7 5- 233. html.
Shafi Goldwasser, Adam Tauman Kalai, Yael Tauman Kalai, and Omar Montasser. Beyond perturbations: Learning guarantees with arbitrary adversarial test examples. CoRR, abs/2007.051155 2020. URL ht pps // //aaxiv.coo orgg abs/ 20bs/2000 2007. 051 45.
Ian Goodfellow. A research agenda: Dynamic models to defend against correlated attacks. CoRR, 0bs/1993.06293, 2019. URL pettt:://rxivve org/ wtt.////910 /ttp//9...00000 06293.
lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamessing adversarial examples.
In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: //arxiv. org/ abs / 111111111222 6572.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pp. 770-778, IERE Computer Society, 2016. doi: 10. 10.11009CVP..2006..90 URL https: //dai.
11t.111/9..0222 2016. 90.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In 7th International Conference on Learning Representatiins, ICLR 2019, New Orleans, LA. USA, May 6-9, 2019. OpenReview.net, 2019. URL 6J CqYm.
net/ wtttttorem/tteeeeeeeeeeeeeteeeee
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training improve model robustness and uncertainty. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2712 2721. PMLR, 2019. URL http: http:///poooeeeeeeeeeeeee m1 r.rrrse///// htteas//v7///eee/e hh//hhnnddycssss .html.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kemne:: Convergence and gen- eralization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL htcpp::/pproceediee
neur www.wps..eeeeeeeeeeeeeee....eeee httpaipps.epaapee/1//////// -aperr.
Zico Kolter and Aleksander Madry. Adversarial Robustness adversarial-ml--ae otreiaalllrro.rooooooooeeeeeeeee
Theory and Practice. https://
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net, 2017. URL https: httpt:://ppeeeeeeeeeeeeeee /wwp/reveeeeeeee////////ee net forumrrrmmmm id= HJGU3RodL.
Yann LeCun. The mnist datahase of handwritten digits. http://yann. lecun. com/exdh/nnist// 1998.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. CoRR, abs/161 1. 102770, 2016. URL http:// //tp/v//a/vvv//////////eeaaaaavr org/ abs / /1611.111111 02770.
Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hypernetworks. CoRR, his/1802.09411 2018. URL http: //bbp uni trier.r................ wwwrrierrdddd/b.e. journal rnal a/ wtt.//orrrrr///// 802. html 4abs-
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
11

---

Published as a conference paper at ICLR 2022
Seyed-Mohsen Moosavi-Dezfooli. Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2574-2587 IEEE Computer Society, 2016. doi: 10.1109/CVPR.2011 www.CCVPR..211.28888 2016.282. URL https: /oii org/ 10. 1109 CCV.. 0.066.2 282.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adversarially robust generalization requires more data, In Advances in Neural Information Processing Systems, pp. 5014-5026, 2018.
Vikash Sehwag, Shiqi Wang, Prateek Mittal, and Suman Jana. HYDRA: pruning adversarially ro- bust neural networks. In Hugo Lamochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balean, and Hsuan-Tien Lin (eds.). Advances Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 12, 2020, virtual, 2020. URL https:/ httpst//proceediaat www.eeeiigg...uur.......oo. www.nnnuu..... wttps/.cppppp/////ee 20000/ hash/ e3a e3a72c7911 2e222799669689 7005ea 7742e04 (el22e04430ee 430ed- Abstract. html.
Aman Sinha, Hongseok Namkoong, and John C. Duchi. Certifying some distributional robustness with principled adversarial training. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, 32:323-332, 2012. doi: 10.1016/j. 201.0e1201..02.006. URL https: / /doi..ioo/// org/1c. 10161016666 6/ neunet. 2012. 02. .016.
Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In Hugo Larochelle. Marc' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Conference on Neurul Information Processing Systems 2020, NewrIPS 2020. December 6-12, 2020, virtual, 2020, URL https : /ttp:p::/pprrooeee pttpp/pocccnng.....cc/p neurips tttccc//ppp// paappr//2200 //aah// 1at////8888-bbbaaaccceaaa .html.
Florian Tramèr, Nicolas Papernot, Ian Goodffllow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv, 2017. URL https: / / arxiv wwt.i//..rr./////e/eo/e/oossss//
org abs 1704. 03453.
Vladimir Vapnik. Statistical learning theory. Wiley, 1998. ISBN 978-0-471-03003-
Dequan Wang, An Ju, Evan Shelhamer, David Wagner, and Trevor Darrell. Fighting gradients with gradients: Dynamic defenses against adversarial attacks. arXiv preprint arXiv:2105.088: 2021.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversar- ial robustness requires revisiting misclassified examples. In 8th International Conference on Learning Representaations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL. htttp:://openeviii net/
Garrett Wilson and Diane J Cook. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology (TIST), 11(5):1 46, 2020.
Eric Wong, Leslie Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In 8th International Conference on Learning Representations, ICLR Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.nec, 2020. URL https://///... pttttps://peereev//// http///ppnee..ecceoe/oeeeeooe.wc net/ fwwummdd=30x00000000
Dongxian Wu, Shu-Tao Xia. and Yisen Wang. Adversarial weight perturbation helps robust generalization. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Bal- can, and Hsuan-Tien Lin (cds.), Advances in Neural Information Processing Systems 33: An- mual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020a. URL https: 2ee.2220eee111225924ee11 wtw.::eaat...-bttattttt........ httml
pttpaper//02///aaeeeeeeepp hashh/
Yi-Hsuan Wu, Chia-Hung Yuan, and Shan-Hung Wu. Adversarial robustness via runtime masking and cleansing. In Proceedings the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 10399-10409, PMLR, 2020b. URL http: / http:////ccceeeeeeeeeeeeee
mwr.preeeeee / vl19 9 wu20f. html.
Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoreti- cally principled trade-off between robustness and accuracy. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 7472-7482. PMLRR 2019. URL http:/ttp/// /proceee//////eeee/// ngs r. www.s//assseee.... /ataas//97/ v97/ /zhagg 9p. html.
12

---

Published as a conference paper at ICLR 2022
Supplementary Material
Towards Evaluating the Robustness of Neural Networks Learned by Transduction
A EXPERIMENTAL DETAILS
A.1
GENERAL SETUP
A.1.1 COMPUTING INFRASTRUCTURE
We run all experiments with Py Torch and NVIDIA GeForce RTX 2080Ti GPUs.
A.1.2 DATASET
We use three datasets MNIST, CIFAR-10 and GTSRB in experiments. The details about these datasets are described below.
The MNIST (LeCun, 1998) is large dataset of handwritten digits. Each digit has 5,500 training images and 1,000 test images. Each image a 28 x 28 grayscale. We normalize the range of pixel values to [0, 1].
MNIST.
The CIFAR-10 10 (Krizhevsky et al., 2009) is a dataset of 32x32 color images with ten classes, each consisting of 5,000 training images and 1,000 test images. The classes correspond to dogs, frogs, ships, trucks, etc. We normalize the range of pixel values to [0, 1].
CIFAR-10.
The German Traffic Sign Recognition Benchmark (GTSRB) (Stallkamp et al., 2012) is dataset of color images depicting 43 different traffic signs. The images are not of a fixed dimensions and have rich background and varying light conditions as would be expected of photographed images of traffic signs. There are about 34,799 training images, 4,410 validation images and 12,630 test images. resize each image to 32 32. The dataset has a large imbalance in the number of sample occurrences across classes. We use data augmentation techniques to enlarge the training data and make the number of samples each class balanced. We construct a class preserving data augmentation pipeline consisting of rotation, translation, and projection transforms and apply this pipeline to images in the training set until each class contained 10,000 examples. We also preprocess images via image brightness normalization and normalize the range of pixel values to [0, 1].
GTSRB.
A.1.3 IMPLEMENTATION DETAILS OF THE ATTACKS
We use Projected Gradient Descent (PGD) (Madry et al., 2018) to solve the attack objectives of PGD attack, FPA, GMSA-AVG and GMSA-MIN. For GMSA-AVG, at the i-th iteration, when applying PGD on the data point to generate the perturbation 6, we need to do one backpropagation operation for each model in {F == per PGD step. We do the backpropagation for each model sequentially and then accumulate the gradients to update the perturbation since we might not have enough memory to store all the models and compute the gradients at once, especially when i is large. For GMSA-MIN, we find that it requires more PGD steps to solve the attack objective at the i-th iteration where we need to attack i + 1 models simultaneously. Thus, we scale the number of PGD steps at the i-th iteration by a factor of i + for GMSA-MIN.
A.2 SETUP FOR RMC EXPERIMENTS
We follow the original settings in Wu et al. (2020b) perform experiments on MNIST and CIFAR- 10 datasets to evaluate the adversarial robustness of RMC. We consider two kinds of base models for RMC: one is the model trained via standard supervised training; the other is the model trained using the adversarial training (Madry et al., 2018), We describe the settings for each dataset below.
A.2.1 MNIST
Model architecture and training configuration. We use a neural network with two convolutional layers, two full connected layers and batch normalization layers. For both standard training and
13

---

Published as a conference paper at ICLR 2022
adversarial training, train the model for 100 epochs using the Adam optimizer with a batch size of 128 and a learning rate of 10-3, training with perturbation budget of 0.3, a step size of 0.01, and number of steps of 40.
3, We use the L norm PGD attack as the adversary for adversarial
RMC configuration. We set K = 1024. Suppose the clean training set is D. Let D' contain [D] clean inputs and [T] adversarial examples. So N D/ We generate the adversarial examples using the Lo norm PGD attack with a perturbation budget of 0.3, a step size of 0.01, and number of steps of 100. We extract the features from the penultimate layer of the model and use the Euclidean distance in the feature space of the model to find the top-K nearest neighbors of the inputs. When adapting the model, we use Adam as the optimizer and set the learning rate to be x 10 4. We train the model until the early-stop condition holds. That is the training epoch reaches 100 or the validation loss doesn' 't decrease for epochs.
Attack configuration. We use the same threat model for all attacks: Loe norm perturbation with a perturbation budget c of 0.3. Cross entropy loss is used as the loss function for PGD attack, FPA, GMSA-AVG and GMSA-MIN: La(F, V) = TV Laayeer - log (zr), where f f(x is the softmax output of the model F. We use PGD with a step size of 0.01, the number of steps of 100, random start and no restarts. We set T = 9 for FPA, GMSA-AVG and GMSA-MIN.
A.2.2 CIFAR-10
Model architecture and training configuration. We use the ResNet-32 network (He et al., 2016). For both standard training and adversarial training, we train the model for 100 epochs using Stochastic Gradient Decent (SGD) optimizer with Nesterov momentum and learning rate schedule. We set momentum 0.9 and Eg weight decay with a coefficient of 10-4 The initial learning rate is 0.1 and it decreases by 0.1 50. 75 and 90 epoch respectively. The batch size is 128. We augment the training images using random crop and random horizontal flip. We use the T norm PGD attack as the adversary for adversarial training with a perturbation budget of 255 step size of 255 of steps of 10.
and number
RMC configuration. We set K = 1024. Suppose the clean training set is D. Let D' contain [D] clean inputs and 4|D] adversarial examples. So N' = 5|D], We generate the adversarial examples 8
255. using the Loe norm PGD attack with a perturbation budget e of 255 a step size of 1 and number of steps of 40. We extract the features from the penultimate layer of the model and use the Euclidean distance in the feature space of the model to find the top-K K nearest neighbors of the inputs. We use Adam the optimizer and set the learning rate to be 2.5 10-5.
Attack configuration. We use the same threat model for all attacks: Los norm perturbation with a Cross entropy loss is used as the loss function for PGD attack, FPA,
perturbation budget of 255 GMSA-AVG and GMSA-MIN: (F,F, V) = - Eiewev - log fz) where f fzx is the softmax output of the model F. We use PGD with step size of 255 the number of steps of 40, random start and no restarts. We set = 9 for FPA, GMSA-AVG and GMSA-MIN.
8
A.3 SETUP FOR DENT EXPERIMENTS
DENT configuration. We perform experiments to evaluate the best version of DENT (DENT+ in Wang et al. (2021)) on CIFAR-10 following the experimental settings Wang et al. (2021).
We use the pre-trained robust models CIFAR- 10 under the Loo norm perturbation threat model from RobustBench Model Zoo as the static base models for DENT, including mod- els with model ID Wu2020Adversariill extra (Wu et al., 2020a), Carmon 20w.on00eennnneeeeee (Car- mon et al., 2019), Sehwag2020Hydra (Sehwag et al., 2020), Wang20201mproving (Wang et al., 2020), Hendrycks20199se Ding2020MMA (Ding et al., 2020). For the test-time adaptation, only the affine scale 7 and shift parameters in the batch normalization layers of the base model are updated. DENT updates sample-wise with different affine parameters (x, 3r) for each input 2, The input adaptation of Σ is not used as suggested in Wang et al. (2021). The model is adapted for six steps by AdaMod (Ding et al., 2019) with learning rate of 0.006, batch size of 128 and no weight decay. The adaptation
(Hendryeks al., 2019), Wong2020rast (Wong al., 2020), and
hhttps://gi thub. http.///ooootstteeecceeeeeeeeeee
14

---

Published as a conference paper at ICLR 2022
objective is entropy minimization with the information maximization regularization:
6 Σ
min 9,
Σ
(ox;00) log f(z;
Σ Σ f(z 0:). log Σ (zx
(8)
where is the batch size, C is the numher of classes and f(0;;; 0c) is the softmax output of the model with the affine parameters 0: (7r, (i) for the input 2;-
Attack configuration. We use the same threat model for all attacks: Ls norm perturbation with a perturbation budget of 255 For PGD attack, FPA, GMSA-AVG and GMSA-MIN, use the following loss function to find adversarial examples with high confidence: (F,, V)
[V] E(e,))ev maXxyyy (z)). where (xx the softmax output of the model F. However, it is hard to optimize this loss function. Thus, we use two alternative loss functions to find adversarial examples. One is the untargeted CW loss (Carlini & Wagner, 2017): (FF, V) [V -(a.)evv-2 -Z(x),
maxkey 7 (xx) k. where (x) is the logits of the model F (the output of the layer before the softmax layer). The other the targeted CW loss: L (F, V) IV E(E.vjEv Z(x), + Z()),. where is the targeted label and t * y. For each attack, we use 14 PGD subroutines to solve its attack objective, including PGD subroutines using the untargeted CW loss L. with different random restarts and 9 PGD subroutines using the targeted CW loss T.2 with different targeted labels. So for each clean Among these adversarial examples, we select the one that maximizes the attack loss with the loss function Li (F,, V) the final adversarial example x' for We use the same hyper-parameters for all PGD subroutines:
test input these PGD subroutines will return 14 adversarial examples x',-- ....... ,
the step size is 255 the number of steps is 100, and the random start is used. We set T = 2 for FPA, GMSA-AVG and GMSA-MIN.
A.4 SETUP FOR DANN EXPERIMENTS
We perform experiments on MNIST and CIFAR-10 datasets. We describe the settings for each dataset below.
A.4.1 MNIST
Model architecture. We use the same model architecture as the one used in Chuang et al. (2020),
which is shown below.
Encoder
nn. mnn.Conv2lt 64, kernel size=S)
Normh
nn. MaxPool22)))
nn. ReI.U
nn. Co442 128, kernel size=5)
nn. BatchNorm2d
nn. Dropour2d
nn. ma.MaxXool2d(2)
nn. ReLU
nn.Conv 2d(128, 128, k size=3, padding=1) No.BatchNorr2d
nn.ReLU
x2
15

---

Published as a conference paper at ICLR 2022
Predictor
nn. C1nCoon22,,12,, 128, kernel size=3, padding=1)
Discriminator
nn. oonBha
mn.Conv2d( (288 128, size=,, padding=1)
nn ReLU
an Rn.
x
X5
flatten
Flatten
nn LinLannar22048, 256) an.BatchNormid
Linear (2044 nn nn.ReLU
(048,, 256)
nn.ReLU
on.Linear(2555 2)
nn. Linear((2256, 10)
mn.Softmax
nn.Softmax
Training configuration. We train the models for 100 epochs using the Adam optimizer with a batch size of 128 and learning rate of 10-3, For the representation matching in DANN, we adopt the original progressive training strategy for the discriminator (Ganin et al., 2016) where the weight a for the domain-invariiat loss is initiated at and is gradually changed to 0.1 using the schedule 10-p 1) 0.1, where p is the training progress linearly changing from 0 to 1.
Attack configuration. We use the same threat model for all attacks: Loo norm perturbation with perturbation budget e of 0.3. Cross entropy loss is used as the loss function for PGD attack, FPA, GMSA-AVG and GMSA -MIN: La (F, V) A Ei - logg f(z where f f(x is the softmax output of the model F. We use PGD with a step size of 0.01, the number of steps of 200, random start and no restarts We set T 9 for FPA, GMSA-AVG and GMSA-MIN.
A.4.2 CIFAR-10
Model architecture. We use the ResNet-18 network (He et al., 2016) and extract the features from the third basic block for representation matching. The detailed model architecture is shown below.
Encoder
nn. 2d(3, 64, kernel size=3)
nn. Bo.BatthNormdd
nn. ReLU
BasicBlock(in planes=64, planes=2, stride=1)
BasieBllokk(n_planessse planes=2, stride=2)
BasichHookKin_pllnes=225 planes=2, stride=2)
Predictor
Discriminator
BasicBlook(in_plaal 5pt0:/ii___aass5111222000 planes=2, stride=2)
BasicBlack(in Masiccllckkinpllane=5112 51anes=512, planes=2, stride=2)
avg_pool2d
wwg_pool2d
flatten
flatten
nn. mm....nnar(512, 10)
nn.
Im..aincar511, 2)
Softmax
nofttxx
Training configuration. We train the models for 100 epochs using stochastic gradient decent (SGD) optimizer with Nesterov momentum and learning rate schedule. We set momentum 0.9 and l2 weight decay with a coefficient of 10-4, The initial learning rate 0.1 and decreases by 0.1 at 50, 75 and 90 epoch respectively. The batch size is 64. We augment the training images using random crop and random horizontal flip. For the representation matching in DANN, we adopt the original progressive training strategy for the discriminator (Ganin et al., 2016) where the weight for the domain-invariant
2 loss is initiated at and gradually changed to using the schedule a = 1+exp(-10-p) 10.p) 1, where is the training progress linearly changing from 0 to 1.
Attack configuration. We use the same threat model for all attacks: Loo norm perturbation with perturbation budget of 255 Cross entropy loss is used as the loss function for PGD attack, FPA, GMSA-AVG and GMS MIN: (F, V) -V= Eiawiev log f(x),, where f(x) is the softmax output of the model F. We use PGD with step size of 256. the number of steps of 100, random start and restarts. We set T = 9 for FPA, GMSA-AVG and GMSA-MIN.
16

---

Published as a conference paper at ICLR 2022
A.5
SETUP FOR TADV EXPERIMENTS
We perform experiments on MNIST and CIFAR-10 10 datasets. We describe the settings for each dataset below.
A.5.1 MNIST
Model architecture and Training configuration. We use the LeNet network architecture. We train the models for 100 epochs using the Adam optimizer with a batch size of 128 and a learning rate of 10-$. We use the Loo norm PGD attack as the adversary to generate adversarial training examples with a perturbation budget e of 0.3, a step size of 0.01, and number of steps of 40. We train on 50% clean and 50% adversarial examples per batch.
Attack configuration. We use the same threat model for all attacks: Lo norm perturbation with a perturbation budget of 0.3. Cross entropy loss is used as the loss function for PGD attack. FPA, GMSA-AVG and GMSA-MIN, L. (F, V)= = iv Σ(awiey - loog fz)),, where fzx is the softmax output of the model F. We use PGD with a step size of 0.01, the number of steps of 200, random start and no restarts. We set T 9 for FPA, GMSA-AVG and GMSA-MIN.
A.5.2 CIFAR-10
Model architecture and Training configuration. We use the ResNet-20 network architecture (He et al.. 2016). We train the models for 110 epochs using stochastic gradient decent (SGD) optimizer with Nesterov momentum and learning rate schedule. We set momentum 0.9 and l2 weight decay with coefficient of x 10-1 The initial learning rate is 0.1 and it decreases by 0.1 at 100 and 105 epoch respectively. The batch size is 128. We augment the training images using random crop and random horizontal flip. We use the Loo norm PGD attack as the adversary to generate adversarial training examples with perturbation budget of 255 step size of 2 and number of steps of 10. We train on 50% clean and 50% adversarial examples per batch.
8
255
Attack configuration. We use the same threat model for all attacks: Lo norm perturbation with perturbation budget e of 255. Cross entropy loss is used as the loss function for PGD attack, FPA, GMSA-AVG and GMSA-MIN: (F, V) = TV [envevv log f (x)),, where f(x) is the softmax I the number of steps of 100, random start
output of the model F. We use PGD with step size of 255 and no restarts. We set T = 9 for FPA, GMSA-AVG and GMSA-MIN.
A.6 SETUP FOR UREJECTRON EXPERIMENTS
We use a subset of the GTSRB augmented training data for our experiments, which has 10 classes and contains 10,000 images for each class. We implement URejectron (Goldwasser et al., 2020) on this dataset using the ResNer18 network (He al., 2016) in the transductive setting. Following Goldwasser et al. (2020), we implement the basic form of the URejectron algorithm, with T = 1 iteration. That is we train a discriminator h to distinguish between examples from P and Q, and train a classifier F randomly split the data into training set Duain containing 63,000 images,
on P. Specifically. validation set Drat containing 7,000 images and a test set Diesi containing 30,000 images. We then use the training set Durai to train classifier F using the ResNet18 network. We train the classifier F for 10 epochs using Adam optimizer with a batch size of 128 and a learning rate of 10-3 The accuracy of the classifier on the training set Dirain is 99.90% and its accuracy on the validation set D is 99.63%. We construct a set ह consisting of 50% normal examples and 50% adversarial examples. The normal examples in the set 5 form a set z. We train the discriminator on the set Dirain (with label 0) and the set ă (with label 1). We then evaluate URejectron's 's performance on ă: under certain threshold used by the discriminator h, we measure the fraction of normal examples in a that are rejected by the discriminator h and the error rate of the classifier F on the examples in the set ă that are accepted by the discriminator ht The set z can be Diest or a set of corrupted images generated on Drest We use the method proposed in Hendrycks & Dietterich (2019) to generate corrupted images with the corruption type of brightness and the severity level of 1. The accuracy of the classifier on the corrupted images is 98.90%. The adversarial examples in H are generated by the PGD attack (Madry et al., 2018) or the CW attack (Carlini & Wagner, 2017). For PGD attack, we use Lox norm with perturbation budget
8 and random initialization. The number of iterations is 40 and the step .
17

---

Published as a conference paper at ICLR 2022
Robustness
Base
Static
DENT
Model
AA
AA
PGD
FPA
GMSA-AVG
GMSA-MIN
Wu al. (2020a)
58.00
58.00
50.40
50.30
50,40
50.40
Carmon et al. (2019)
57.30
58.00
51.80
51.80
51,80
51.80
Schwag ct al. (2020)
54.90
55,90
50.10
49.80
50.00
50.00
Wang et al. (2020)
53.60
55.90
49.00
49.10
48,90
48,70
Hendrycks al. (2019)
51.80
53.10
48.10
48.20
48.30
48.30
Wong al. (2020)
42.40
44.60
40.10
39.90
40.00
40.00
Ding et al. (2020)
39.70
42.10
36.20
35.70
35.30
35.00
Table Results evaluating DENT on CIFAR-10 under the adversarially ordered game. Bold numbers are
worst results.
Accuracy
Robustness
Base Model
Dataset
Static AA
RMC
Static
RMC
GMSA-AVG
GMSA-MIN
Standard
99.40110.15
98.62.10.23
0.00110.00
0.5410005
0.80.10.19
MNIST
Madry et al.
99.24.10.22
9..01.0.91
87.86
57.1415..83
59.48.14.52
Standard
93.96110.38
93.5610.34
0.00
8.50111229
8.92.10.33
CIFAR-10
Madry et al.
83.70.11.11
91.4610.71
43.5811..3
38.5012.07
39.00
Table 6:
Results of evaluating RMC. We also evaluate the static base model for comparison. We report the mean and standard deviation of the accuracy robustness (meanistd) over the five random of the experiment.
size is 255 The robustness the classifier under the PGD attack 3.66%. For CW attack, we use La norm as distance measure and c and 0. The learning rate is 0.01 and the number of steps is 100. The robustness of the classifier under the CW attack is 0.00%.
A.7 EVALUATE DENT UNDER THE ADVERS GAME
We evaluate the robustness of DENT under the adversarially-orde game where the adversary can choose a "worst-case" order of perturbed data points after receiving large amount of test data and then sends them in batches one time to the defender. Specifically, each time the attacker will generate adversarial examples up 256 data points, and then sort the adversarial examples by their labels from lowest to highest, and finally send the sorted adversarial examples in batches one at time the defender. Other experimental settings are the same those described in Appendix A.3. The results in Table 5 show that under the adversarially-oreee game, we can reduce the robustness of DENT be lower than that of static base models.
A.8 MULTIPLE RANDOM RUNS OF THE RMC EXPERIMENT
In Section 6, we describe the experimental setup for evaluating RMC. The performance of RMC is evaluated on sequence of test points x .... !!!" randomly sampled from the test dataset. We repeat this experiment five times with different random seeds and report the mean and standard deviation of the results over the multiple random runs of the experiment. When evaluating the robustness of RMC, we only use the GMSA-AVG attack and the GMSA-MIN attack since they are the strongest attacks. From Table 6, we can see that the results don't vary much across different random runs and the conclusion that the proposed GMSA attacks can break RMC still holds.
18


Reviews:
[{"review_id": "uCy5MRo9BQB", "reviewer": "Reviewer_WKKJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My biggest concern is about the importance of the studied topic in this paper. Specifically, the so-called transductive defenses are not very popular and have not been fully recognized by the community so far. In fact, I happen to have read the paper of RMC when it was published in ICML and thought that the authors of RMC should have tested some adaptive defense-aware attacks in their work, which they did not. After all, breaking random or dynamic models for attacking purpose has been investigated by BPDA and EOT. Meanwhile, I also served as a reviewer for Dent, which was rejected at the time and has not been accepted by other conferences according to my knowledge. In the review opinions for Dent, many reviewers also questioned that it failed to conduct some defense-aware and adaptive evaluations. Thus, to me, it seems that the proposed attack GMSA origins from the obfuscated gradient work and only breaks a few defenses that are somewhat not canonical. ", "summary_of_the_paper": "This paper intends to break a specific type of adversarial defenses. These defenses choose to dynamically modify the model for the purpose of robustness and defenses. This paper proposes to counter such defenses with a new optimization algorithm.", "summary_of_the_review": "I give a weak reject score for now. I may change my score after I exchange my opinions with other reviewers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "tbmzu8uifRT", "reviewer": "Reviewer_2Lpz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\n1. Experimental results seem to suggest the proposed GMSA is effective (though there are some places not clear and require clarification, see below questions)\n2. The proposed idea (greedy model space attack) is simple yet seemingly effective\n\n## Weakness\n1. The writing needs to be improved. The current draft is written in a complicated way, which is hard to parse, e.g. the notations, the problem setting explanation etc.\n2. The are some questions/places not clear and require clarifications from the authors, see below.\n\n### Questions\nQ1: How practical is the problem setting of adversarial robustness in transductive learning? Also, it's a bit not clear about how exactly the adversary and the defender interact. I assume it works as follows:\nadversary generate a batch of adv examples (adv_0) based on model_0 (original model) --> defender get adv_0 and update the model to model_1 --> adversary generate a new batch of adv example (adv_1) based on model_1 --> defender get adv_1 and update the model_1 to model_2 --> repeat to the end\n\n- Are my above understanding correct? If so, then is it practical to assume that the adversary and defender will interact in this way? \n\n- Perhaps a more difficult setting is when the adversary have no ideas of when the defender actually update the model? In this case, would the proposed method work (it seems that the proposed method is a white-box attack setting)? \n\nQ2: it's a bit not clear to me regarding the \"private randomness\". \n- What happens if the adversary also know about the randomness in $\\Gamma$? The attacker should be able to develop a stronger attack?\n- The authors mention that the $\\Gamma$ can be randomized smoothing type of defense, but I didn't see the experiment results compared with randomized smoothing? Can the authors show some experiments for this? \n\nQ3: Table 1, Table 2\n- Table 1: It's not clear to me how exactly is the difference between the AA under static and AA under RMC? Using my understanding in Q1, does it mean that, AA (static) refers to attacking model_0, while AA (RMC) refers to attack model_i, while i = {1, 2, 3, ...} ? Then why AA (RMC) does not work? \n- Table 2: can the authors explain what's the difference between DENT-AA and AA under DENT? why they have a large gap (~20%) in robust accuracy? \n- Can the authors explain again what's the difference between DENT and RMC? It looks like the proposed GSMA is more effective on RMC compared to AA (RMC) while not so effective compared to AA (DENT)? \n\nQ4: the authors mention the setting of adversarially ordered and naturally ordered data. But from the Algorithm 1 and 2, the data order doesn't seem to matter in GMSA? Why DENT and RMC need the naturally ordered data? \n\n", "summary_of_the_paper": "This paper study the adversarial robustness in transductive learning, where the defenders can update the models during test time by interacting with the adversary in multiple rounds. The authors propose \"Greedy Model Space Attack\" (GSMA), where the key idea is to general adversarial examples considering all the history models in the past rounds. The result show that the proposed attack can decrease robust accuracy more than other baseline (the static AA, DENT AA).  ", "summary_of_the_review": "In summary, this paper propose a simple yet effective GSMA attack to evaluate adversarial robustness of models in the transductive learning setting. Some results are promising but there are a few places require further explanation/clarification. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "oEb4H8Pjrm", "reviewer": "Reviewer_q4Dr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- I appreciate the formalism of the adversarial game for transductive learning introduced by the paper, which also seems novel.\n- The paper covers a wide range of topics: formalizing the transductive attacks and defenses, breaking transductive defenses thought to be strong, experimenting with domain adaptation as defense against the attack proposed in the paper.\n- The proposed attack seems to perform well, defeating existing defenses. The experimental section seems to follow standard practices for robustness evaluation.\n- The implementation is provided.\n- The paper is well-written.\n\nWeaknesses:\n- The topic of the computation cost of the proposed attack is not discussed.\n- The proposed method seems to be a variation of PGD (limited novelty), but considering that the paper has other contributions and GMSA works well in practice, this work is arguably worth publishing.\n\nOther comments / questions:\n- The paper could benefit from additional proofreading.\n- The transferred adversarial samples produced with PGD and AutoAttack might be weaker than some generated with the proposed attack, thus not resulting in a proper evaluation.", "summary_of_the_paper": "This paper proposes a new attack strategy for adversarial examples in the case where transductive defense strategies are employed. The proposed attack is based on projected gradient descent and manages to break existing transductive defenses that were thought to perform well. A new strategy for adversarial training is also proposed yielding more robust models in the transductive setting than existing defenses. Experiments are performed on MNIST, CIFAR-10 and GTSRB, against both naturally trained and robust models.", "summary_of_the_review": "Good formalism for adversarial machine learning in transductive setting, plus other smaller contributions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Kr3-gkIXnm", "reviewer": "Reviewer_7duz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n- The paper is generally well-written and clear. It clearly articulates issues with the previous evaluation of transductive learning algorithms for learning and lays out the threat model that should be used in the future when evaluating these algorithms. It also delves into the various aspects of the multi-round game that results from an attack on transductive robust learning.\n- The empirical study is fairly thorough, tackling 4 different types of robust learning algorithms across two standard datasets, using the two different attacks proposed in the paper.\n**Weaknesses**\n- While the paper does a good job of articulating the challenges for an adaptive attacker on transductive algorithms, including that the problem results in a bilevel optimization problem for the attacker, it does not use particularly novel methods to tackle this problem. Both proposed methods just turn the problem into a static optimization problem which just use PGD to solve it. There is a plethora of literature on end-to-end poisoning attacks (see [1], [2]) that uses methods such as gradient unrolling or replacing the model with an approximation derived from the Neural Tangent Kernel. The GSMA attack that is proposed merely uses the history of all the trained models, and is essentially still a static attack. I would urge the authors to understand the methods used in the poisoning literature for circumventing the challenges of bilevel optimization.\n- Some aspects of the manner in which the final robustness numbers are determined for the proposed attacks are unclear. Specifically, when the transductive model is run for T iterations, is the robustness reported using the best adversarial example set over all iterations, but evaluated with respect to the final model with all test data? Further, for DENT, the value of T is set to 2, so does that mean the numbers reported are only with respect to two samples from the test set? I read through the paper and appendix multiple times for clarity on this point, but it did not seem to be addressed.\n- The strawman TADV approach is not evaluated with sufficient rigor. The private randomness can be circumvented by the attacker simulating adversarial training themselves with multiple instantiations of the initialization and adversarial example generation, and then generating universal adversarial perturbations to be effective across all these locally generated models. It would be interesting to see how the defense performs under that setting.\n\n**References**\n[1] Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization, https://arxiv.org/abs/1708.08689\n[2] Neural Tangent Generalization Attacks , https://proceedings.mlr.press/v139/yuan21b.html\n", "summary_of_the_paper": "This paper proposes adaptive attacks against transductive robust learning methods. These methods aim to improve the robustness of models to adversarial examples by updating the model using unlabeled test data. However, this paper notes that previous evaluation of transductive robust learning has not considered attacks that are truly adaptive, i.e. the attacker should be able to utilize its knowledge of the transductive learning algorithm to craft better examples. Two types of adaptive attacks against transductive learning algorithms are proposed, a single step method and a greedy method taking into account all previous instances of the model learned during transductive robust learning. The experimental results on two existing training methods and two newly proposed ones, one of which is a strawman, demonstrate that the proposed adaptive attacks render transductive robust training to be no more robust than standard robust training. The paper's conclusion is that the previously claimed improvements in robustness were an artifact of the use of non-adaptive attacks.", "summary_of_the_review": "Overall, while I found the paper to elucidate an important shortcoming in previous work on evaluating the robustness of transductive robust learning, the attack methods used lacked novelty. In fact, in a number of cases the simple FPA algorithm had the best performance. I urge the authors to consider improved methods to circumvent the bilevel optimization problem.\n\n+++++++++++++++++++\n\nThe clarifications have improved the paper. While I still believe more innovation is possible for the attack, I have decided to raise my score to a 6 since the paper is likely to lead to interesting follow-up work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper formalizes the adversarial attack problem for transductive defenses, where the model is sequentially updated with a batch of (adversarial) test inputs. The paper comes up with a quite generic attack scheme and their instantiation of this scheme shows that RMC and DENT are not robust respectively not more robust than the underlying adversarially robust base model.
  
Positive 
- formal treatment of attacks on transductive defenses including discussion about different types of attacker knowledge
- the attack model is quite generic and could work for future transductive defenses and thus is a useful baseline attack which could be suggested to be used by future transductive defenses for robustness evaluation. In particular, as the standard AutoAttack is not designed for transductive defenses and thus can overestimate adversarial robustness

Negative
- the description is sometimes overly technical and some (important) details had to be clarified
- the technical novelty of the attack is limited
- the overall accuracy but also robust accuracy depends on the chosen batch. Therefore the authors should report mean and standard deviation over several different random draws of batches 
- the Transductive Adversarial Training Defense seems to consist of adversarial retraining from scratch after each incoming batch. This is excessively costly and not practical.

Minor:
- The batch size is an important parameter which apparently is assumed to be known in this work

The paper is borderline. Two reviewers argue for rejection, two for acceptance. Only one reviewer engaged in the discussion. 
In my point of view the positive point of having a reference for correct evaluation of adversarial robustness of transductive defenses weighs more than the raised negative points which can be fixed (at least partially). Thus I think that this paper is a valuable contribution to the field of adversarial robustness.
Decision: Accept (Poster)