Test-Time Adaptation and Adversarial Robustness
Authors: Xi Wu
Yang Guo
Tianqi Li
Jiefeng Chen
Qicheng Lao
Yingyu Liang
Somesh Jha
Venue:  (2021)
Keywords: test-time adaptation
adversarial robustness
unsupervised domain adaptation
threat model
maximin game

Abstract: This paper studies test-time adaptation in the context of adversarial robustness. We formulate an adversarial threat model for test-time adaptation, where the defender may have a unique advantage as the adversarial game becomes a maximin game, instead of a minimax game as in the classic adversarial robustness threat model. We then study whether the maximin threat model admits more ``good solutions'' than the minimax threat model, and is thus \emph{strictly weaker}. For this purpose, we first present a provable separation between the two threat models in a natural Gaussian data model. For deep learning, while we do not have a proof, we propose a candidate, Domain Adversarial Neural Networks (${\sf DANN}$), an algorithm designed for unsupervised domain adaptation, by showing that it provides nontrivial robustness in the test-time maximin threat model against strong transfer attacks and adaptive attacks. This is somewhat surprising since ${\sf DANN}$ is not designed specifically for adversarial robustness (e.g., against norm-based attacks), and provides no robustness in the minimax model. Complementing these results, we show that recent data-oblivious test-time adaptations can be easily attacked even with simple transfer attacks. We conclude the paper with various future directions of studying adversarially robust test-time adaptation.

Under review as a conference paper at ICLR 2021
TEST-TIME ADAPTATION AND ADVERSARIAL Ro- BUSTNESS
Anonymous authors Paper under double-blind review
ABSTRACT
This paper studies test- cost-time adaptation in the context of adversarial robustness. We formulate an adversarial threat model for test-time adaptation, where the de- fender may have a unique advantage as the adversarial game becomes maximin game, instead of minimax game as in the classic adversarial robustness threat model. We then study whether the maximin threat model admits more "good solutions" than the minimax threat model, and is thus strictly weaker. For this purpose, we first present provable separation between the two threat models in a natural Gaussian data model. For deep learning, while we do not have a proof, we propose candidate, Domain Adversarial Neural Networks (DANN), an algo- rithm designed for unsupervised domain adaptation, by showing that it provides nontrivial robustness in the test-time maximin threat model against strong trans- fer attacks and adaptive attacks. This is somewhat surprising since DANN is not designed specifically for adversarial robustness (c.g., against norm- noom- attacks). and provides no robustness in the minimax model. Complementing these results, we show that recent data-oblivious test-time adaptations can be easily attacked even with simple transfer attacks. We conclude the paper with various future di- rections of studying adversarially robust test-time adaptation.
INTRODUCTION
There is a surge of interest to study test-time adaptation to help generalization to unseen domains (e.g., recent work by Sun et al. (2020); Wang et al. (2020); Nado et al. (2020)). At the high level, a generic test-time adaptation can be modeled as an algorithm I which accepts an (optional) labeled training dataset D, an (optional) model trained on (usually used as starting point), and an unlabeled test feature set U, outputs a model F = [[F, U), in order to achieve high test accuracy on U. For large test set U. test-time adaptation can be viewed as form of transductive learning (Joachims (1999); Vapnik (1998)) (i.e., using D, U to train a model to predict specific instances in U) which argued to be easier than more traditional inductive learning.
This paper studies test-time adaptation in the context of adversarial robustness (i.c.. there is an active agent who tries to fool the test-time adaptation by perturbing the input so that F gives wrong predictions). There are several motivations in pursuing this direction. First, this question of practical interest: Many practical ML pipelines run in a butch mode', where they first collect set of unlabelled data points, and then send them to a model (c.g. Nado et al. (2020).. In such cases, data in the batch may have been adversarially perturbed, and it is natural question whether we can leverage the large batch size and test-time adaptation to enhance adversarial robustness. Second, from purely theoretical point of view, since test-time adaptation is form of transductive learning. is intriguing to ask whether transductive adversarial learning can be easier, given that traditional adversarial robustness is formulated in the inductive learning setting (e.g. Madry et al. (2018)). To this end, a recent work by Goldwasser et al. (2020) shows that, with transductive learning, one can achieve nontrivial guarantees for classes of bounded VC dimension with arbitrary train and test distributions. The current work complements their paper in the setting of deep learning.
To study these questions, we formalize a threat model. which we call (test-time) maximin threat model, for the adversarial robustness of test-time adaptation. Recall that the classic adversarial
For Inssple. collects large hatch of photos before sending them model tag people.

---

Under review as a conference paper at ICLR 2021
robustness game is minimax game ming Ey [maxy, L(F, V)], where V is random sampled data, V is the perturbed data generated from V by the adversary, and L(F, V) is the loss of the model F on V. By contrast, in the maximin threat model, we allow V to be sampled from different domain, and the game is maximin: Ev [maxy ming L(F,, V)] (where U is the perturbed features of V. subject to the attack type, and V is the labeled perturbed data, see Definition 2). By the maximin inequality, it follows that this threat model is no harder than the minimax model (to allow source and target domains to be different, we need to generalize the classic minimax model, see Definition 3).
We then move to the focus of this work: Whether the maximin threat model is "strictly weaker" than the minimax threat model. We note that any good defender solution (a robust model) in the minimax game induces good defender solution in the maximin game (an adaptation algorithm that outputs that robust model), thus intuitively, the good defender solutions of the minimax model is a subset of the good defender solutions of the maximin threat model. We ask whether such a contain- ment is proper: That is, whether there exists a defender solution that is good in the maximin threat model, but is bad in the minimax threat model. The existence of such a defender will demonstrate that the maximin threat model admits more good solutions. Besides theoretical interest, this ques- tion is also of practical importance since these "new" solutions may possess desirable properties that good solutions in the minimax threat model may lack. For example, one such property is that the defender solution is attack agnostic (Goodfellow (2018) (pp.30)): That is, the solution is not to directly optimize the performance measure for particular type of perturbation".
To this end, we first present a provable separation between the maximin and minimax threat mod- els in natural Gaussian data model. In fact, the separation holds even when U only contains single point, indicating the power of transductive learning. We then move to deep learning. While empirically examine Domain Adverarial Neural Networks
we do not have provable guarantees, (DANN) (Ganin et al. (2017)), an algorithm designed for unsupervised domain adaptation (UDA), as candidate for the separation. Specifically, we demonstrate that DANN provides nontrivial test- time adversarial robustness against both transfer attacks and adaptive attacks, in both homogeneous and inhomogeneous cases. This is somewhat surprising as DANN is attack agnostic as we mentioned above, and has not been considered for adversarial robustness. Not surprisingly, as we hypothesized for separation, the accuracy becomes very low when evaluating F in the minimax model.
Complementing the above result, we explore the maximin robustness of the recent data-oblivious adaptation algorithms (namely, the adaptation algorithms do not use D. but just the pretrained model F and unlabeled test set U). Specifically, we consider Test-Time Training (TTT) by Sun et al. (2020)² We show that TTT can be easily attacked using simple transfer attacks. While this is not surprising as authors of Sun et al. (2020) have cautioned that TTT is not designed for adversarial robustness, the situation is in sharp contrast our results with DANN.
The rest of the paper is organized as follows: Section 2 presents the setup. In Section 3 we define threat models. In Section 4 we present theoretical results about separation, and examine DANN as a candidate separation in the deep learning. Finally, Section 5 explores the maximin robustness of oblivious test-time adaptation, and concludes the paper with future directions.
2 PRELIMINARIES
Let F be a model, for data point (z.v) E X' XV, a loss function e(F;2, v) give the loss of F on æ given the true label y. Let V be a set of labeled data points. We use the notation I.(F, V) = [V Σ (ix)eev ((F; ar, y) to denote the empirical loss of F on V. For example, if we use binary loss
go. (Fi = [=)()) + y). this gives the test error of F on V. We use the notation Vix to denote the projection of V to its features, that is {(,,, y:) + [Z
Threat model for classic adversarial robustness. To formulate the threat model for test-time adaptation, we first present threat model for the classic adversarial robustness. Although the classic adversarial robustness can be written down succinetly as a minimax objective, namely
2 Another consideration, which is heyond the scope of this paper, is the computational fcasibility of finding a good solution, given the hardness of minimax optimization Katz et al. (2017); Daskalakis et al. (2020). While TTT does not use training data 1 at the test time, it has special self- self-training component, and the joint architecture is sttrucctue, A more domain agnostic approach is discussed in Wang et al. (2020).
2

---

Under review as a conference paper at ICLR 2021
ming ((x,)))(Xx))) m (a
(N(x) is neighborhood function of determined by the attack type), a threat formulation will help us develop more nuanced models.
((FFF'',))) y)]]
Definition 1 (Threat model for classic adversarial robustness). Attacker and defender agree on
particular attack type, Attacker is an algoritlum A, and defender is supervised learning algorithm T.
Before game starts
(labeled) training set sampled i..d from from (X, Y).
Training time
(Defender) Train a model F on D as F T(D).
Test time
(labeled) natural test set V is sampled ii.d from (X. Y) (Attacker) input F, D, and V, A perturbs each point (ar, y) E to (x'.y) (subject to agreed attack type), giving V A(F, V.V).
Evaluation:
Evaluate the text loss of F on V. L(F.V) V.. Attucker's goal is to maximize the test loss, while the de- fender's 's goal is to minimize the test loss.
We stress that the i.dd sampling of V is important (which is also present in the expectation in the minimax objective): Otherwise an attacker can pick any point that fools F and repeat it arbitrarily many times. (we refer readers to Goodfellow (2019) for more discussions along this line).
Notations for models and attacks. In this paper we mainly use the PGD attacks (Projected Gradient Descent attacks) with borm-based perturbations Madry al. (2018). For example, given model F, we use the notation PGD(F, V) to denote PGD attacks against F, on data V (the attack type is specified in the context). We adopt the following notations:
Notation
Meaning
A target model trained on the labeled target data V.
AdvT S AdvS PGD(..)
An adversarially trained target model using the labeled target data V.
A source model trained on the labeled source data D.
An adversarially trained source model using the labeled source data D. PGD Attacks on model and data. For example, PGD(AdvT, V' means to use PGD attacks on the model AdvT and data V.
Test-time defenses and BPDA. Various previous work have investigated test-time defenses where a pretrained model is fixed and there is a "preprocessing procedure" to preprocess an input before sending it to the model. Several such defenses were described and attacked in Athalye et al. (2018), by the BPDA technique (Backward Pass Differentiable Approximation), While syntactically one can fit these defenses into our framework, they only form some very special cases of our framework, which reuses a fixed pretrained model and focuses on inpus sanitization. As we will show later in the paper, for both of our provable separation and deep learning results, the adaptation algorithms train new models (beyond sanitizing inputs); and theoretically attacking these adaptations becomes a bilevel optimization. In these cases, it is unclear how to apply BPDA, and indeed is an intriguing direction to further study attacking unsupervised domain adaptation algorithms, such as DANN.
THREAT MODELS
3.1 TEST-TIME MAXIMIN THREAT MODEL
The intuition behind the test-time maximin threat model is as follows: After receive the adver- sarially perturbed data to classify (at test time), the defender trains a model based on U, and we evaluate the test accuracy only for U. (i.e., for different test set U may have different models and different test accuracy.) This perspective of training a model using labeled data D and unlabeled data U, and only test on U. is essentially the transductive learning by Vapnik (1998) (however, we consider it in an adversarial setting). Formally, we have the following definition:
Definition 2 (Maximin threat model for test-time adaptation). Fix adversarial perturbation type. We have source domain (X", Y"), and target domain (X", Y*). Attacker an algoritlum A, and
3

---

Under review as a conference paper at ICLR 2021
defender is a pair of algorithms (T,I), where T is supervised learning algorithm, and is test-time adaptation algorithm.
Before game starts A (labeled) truining set D is sampled Li.d. from (X*, (XxY))
Training time
(Defender, optional) Train model F trained on D as F T(D).
Test time
(labeled) natural test set sampled ii.d from (X',Y)), sent the attucker.
(Attacker) Produce an unlabeled dataset U as follows:
1. On input I, F, D, and V, A perturbs each point (x, v) E V to (x', .u) (subject to the agreed attack type), giving = A(T,F, D,D, DVV).
Send U = V|x (that is, the feature vectors of VV to the defender.
(Defender) Produce an adapted model as F D,U,
Evaluation: Evaluate the test loss of F on V. L(F.V).
On the adversary: While the adversary knows the adaptation algorithm, I' may use some private ran- domness, such as random initialization. Since defender proceeds after the attacker to apply I, these private randomness is assumed not to be known by the adversary.
On notations: If there is no need for a pretrained model, we use the notation I'd,,
Modeling differences. (1) The first obvious distinction is that source domain may differ from the target domain. (2) This is called test-time threat model since F is trained based on U = Vlxx and the test error is only evaluated on V (the perturbed test set). This is in sharp contrast with the classic minimax threat model, where the model F is trained on D, which independent from V and so V. (3) This threat model enables us to study whether a large test set (i.e. large (U]) can benefit a defender for adversarial robustness. This is in sharp contrast with the classic minimax threat model, where the attacker is granted much power, and can pick points in a one-by-one, or "online", fashion (i.e., sample a point from the nature, perturb it, and send to the defender). (4) This is called a maximin threat model since the adversary must move first to submit the challenge set U, and the defender then moves based on it. fact, this can be written as a maximin game Ey [maxy min& test err(F V)]. different from the minimax game in the classic threat model.
Same security goal. However, in the case where these two domains coincide, the maximin threat model is still nontrivial. In fact, from the security perspective, both threat models are about the same end-to-end security goal: Correct predictions on adversarially perturbed data.
Attacking a model ys. Attacking an algorithm. In the classic threat model, a model F is fixed after training, and the test time the adversary attacks the model. However, in the maximin threat model, the adversary must attack an adaptation algorithm l.
Restrictions of the threat model. Astute readers may realize that have intentionally left the definition of the test-time adaptation I' to have a lot of freedom. For example, I' can leverage the labeled training data D, which may be practically test-time computations. We consider the following possibilities of restrictions:
available computationally infeasible for
Homogeneity. If the source and target domains equal, call it the homogeneous maximin model. This setting is directly related to the classic threat model (Definition 1), where we have a single do- main (X, , Y). ). but an adversary can perturb the test instances. From security perspective, maximin threat model gives relaxed threat model, but the same end-to-end security guarantees. (i.e., correct predictions against adversarially perturbed input)
Data-obliviousnees. In this case, the adaptation algorithm I cannot leverage the labeled source data D. In other words, the agent can only leverage the pretrained model and the unlabeled data set U.
3.2 ADVERSARIAL SEMI-SUPERVISED MINIMAX THREAT MODEL
An unsatisfying aspect of the theory thus far is that the classic minimax threat model only defined in the homogeneous case, but the maximin threat model can be inhomogeneous, which makes the
4

---

Under review as a conference paper at ICLR 2021
direct comparison difficult. To bridge the gap, provide a generalization of the classic minimax threat model that lies between the classic minimax threat model (Definition 1) and the test-time maximin threat model (Definition 2).
Definition (Adversarial semi-supervised minimax threat model), Fix an adversarial perturbation rype. We have a source domain (X*,Y"), and a target domain (X Y*). The attacker is a pair of algoritims A1), and the defender is pair of algorithms (T T",, where TT is supervised learning algorithm, and I is semi-supervised learning algorithm.
Before game starts
training set sampled ii.dd from (X*, Y*).
semi- supervisio set V is sampled i..d from (X*, Y²), ), sent to the attacker.
Training time
(Defender, optional)
Train model F trained on D F = T(D).
(Attacker) Produce an unlabeled dataset U as follows:
On input F. D, and V, .A perturbs each point (x, v) (x',y) (subject to the agreed attack type), giving V = Ao(T, F, D.V).
Send U V(x (that is, the feature vectors of V) ) to the defender.
(Defender) Produce final model as : F I,F,D,U).
Test time
(labeled) natural set V is sampled ii.dd. from (X*, Y+). (Attacker) On input F. D. and V", Ai perturbs each point (æ,y) V to (x',u) (subject the agreed attack type), giving V A1 (F, D, D,V").
Evaluation: Evaluate the test loss of F V1, (.). The goal of the adversary (Ao.Ai) is to (jointly) maximize L(F, V').
Modeling differences. (1) This threat model is semi-supervised, because the defender (learner) receives an unlabeled data set U from the target domain. Note that the procedure to produce training time of this threat model is exactly the same as the procedure to produce F' in the test-time maximin threat model. The key difference is that, once F is trained, we evaluate it on V", which is the adversarially perturbed data on independently sampled V'. This is thus inductive learning, and minimax game. (2) This threat model is adversarial, because the attacker can adversarially perturb the clean semi-superviee supissuperviioon set V/ produce U.
the
Classic minimax model as a special case. The classic minimax threat model is a special case of this threat model, by putting source and target domains equal, and choosing trivial I: [(F, D,,,D,U) F TTD). We list several other specializations of this threat model in Appendix B.3. Therefore, without loss of generality, for the rest of the paper, by minimax model we mean the adversarial semi-supervises minimax threat model.
I: One algorithm, two interpretations. We note that for algorithm r (F.D,U) + F, one can have now two interpretations: In the maximin threat model, we interpret it as a adaptation algorithm, because we are only going to evaluate F on V; in the minimax threat model, we interpret it semi- sumeivsuerrised learning algorithm, because SEPARATING MAXIMIN AND MINIMAX
going to evaluate F on unseen points V
We now move on to the focus of this work: the maximin threat model "strictly weaker" than the minimax threat model?
4.1 VALUATION OF THE GAMES
Proposition (Homogeneous maximin vs. Classic minimax threat model). Let k 2 be natural number, and F be the hypothesis class. For a given V, the domain of V is well-defined function of V (e.g., lox ball around V). We have that: Ev~(XY) maxy min [ Vi} s
min PEr EV~(X,Y)
maxp (L(F [V]
5

---

Under review as a conference paper at ICLR 2021
The proof (A.1) does not rely on the homogeneity condition, and holds verbatim to the more general remi-supervised threat model. We also note that, in fact, if the concept class has unbounded VC dimension, then good models always exist that can fit both D and V perfectly. So the valuation of the maximin game is actually always 0:
Proposition 2 (Good models exist with large capacity). Consider binary classification tasks and that the hypothesis class F. has infinite VC dimension. Then the valuation of the maximin game
Ev maxy ming V)} is 0. That is, perfect models always exist to fu U.
This thus gives first evidence that that transductive advesarial learning is strictly easier. We mark that transductive learning here is essential (differnet models are allowed for different U). We conclude this section by noting the following:
Proposition 3 (Good minimax solution is also a good maximin solution). Suppose T is a super- vised learning algorithm which trains a model F T (D), where its adversarial gain in the ad- versarial semi-supervised minimax model bounded by is (i.e. Ey (mr(mx L(F", L(F*,V) S K.) Then in the maximin threat model, the adversarial gain of the strategy (T*,T*). where I (F*, D,U)= F = TT"(D) is also upper bounded by K.
However, clearly, the valuation of the game does not answer a key question whether there is "real" adaptation algorithm, which can only leverage unlabeled data U, that separates between the two theat models. In other words:
Is there a such that:
As a test-time adaptation algorithm in the maximin model, it provides robustness, but As a learning algorithm in the minimax model, the model produces has no robustness.
The existence of such a I' would provide separation between the minimax and the maximin threat models, and indicates that the maximin threat model admits more good solutions. Despite theoretical interest, this question is of practical importance since these "new" solutions may possess desirable properties that good solutions in the minimax threat model may lack. For the rest of this section, we consider one such desirable property that the defender solution is attack agnostic (Goodfellow (2018) (pp.30)): That is, The defender strategy is not to directly optimize the performance measure. (e.g., we know the attack -nom attacks, and we directly train for it).
4.2 PROVABLE SEPARATION OF THE MINIMAX AND MAXIMIN MODELS
In this section we provide problem instance (i.e., data distributions and number of data points) and prove that that maximin threat model is strictly easier than the minimax threat model for the problem: In the minimax model no algorithm can achieve nontrivial error, while in the maximin model there are algorithms achieving small errors. Since the maximin model is no harder than the minimax model for all problem instances and there is problem instance where the former is strictly easier, we thus formally establish separation between the two models. Furthermore, the problem instance we considered is on Gaussian data. The fact that maximin is already strictly easier than minimax in this simple problem provides positive support for potentially the same phenomenon on more complicated data.
Data distributions and the learning task. We consider the homogeneous case (the source and tar- get are the same distribution) and the fox attack. We consider the Gaussian data model from Schmidt et al. (2018); Carmon et al. (2019b): binary classification task where X' = IRd and y = {+1. -1), y uniform on ) and xly ~ (yj,, o²1) l) for a vector μ E Rd and coordinate noise variance o² > 0. In words, this is mixture of two Gaussians, one with label +1, and one with label -1. We will consider the following parameters. First, fix an integer ny > 0, and an e E (0, 1/2), then set the following parameter values:
(1) For both threat models, the datasets D = {(x; v:)) and V = (ar, v)}. J. In particular, V only has one data point. In the maximin threat model, we let x' denote the perturbed input obtained from x by the loc. attack with bounded norm e > 0, i.c., x' = x v with || S e. Put V = [(x,, y)} and U {a'}. We prove the following:
11a112= = d, 0² =2=vdom dno-

---

Under review as a conference paper at ICLR 2021
Theorem 1 (Separation of Maximin and Minimax). Let K be a (sufficiently large) constant (which used for control classification error). Consider the Gaussian data model above with 2²
La no 2 K, vd/no 2 32K logd, and 2K S S no To (the second holds for all suffi-
ciently large d, which then determines all that falls into the range). Then the following are true.
(1) In the semi- superviie minimax threat model, the learned model F I(T(D), D,U) by any algorithms TT and I must have large error: E. { ((F,V')) 2 (1--d=)), where the expectation is over the randomness of D, V and possible algorithm randomness.
(2) In the maximin threat model, there exist attack-agnostic algorithms T and I' such that for some absolute constant e O, the adapted model F (TT()), D) has a small error: E ((F.V)) Se-ck,
4.3 SEPARATION IN DEEP LEARNING: A CANDIDATE ALGORITHM OF DANN
We now consider deep learning. While we are not able to prove the existence of such a defender so- lution in this setting, present somewhat surprising connection between transductive adversarial robustness and unsupervised domain adaptation. In particular, we propose Domain Adversarial Neu- ral Networks (DANN) candidate algorithum for the separation, and provide empirical evidence that it provides the desired separation. The experiment design is as follows:
(1) We use the DANN algorithm with random initialization (RI), that is, DANN(RI, ")), as test-time adaptation. There are several motivations of choosing DANN: (A) DANN fits our framework as designed for unsupervised domain adaptation. (B) DANN is however not designed for adversarial robustness. Thus is it a very interesting question whether DANN can provide test-time adversarial robustness against attacks (e.g. norm-based attacks) that it is not specifically designed for. (C)
DANN algorithm leverages source data D, which could benefit the maximin robustness.
(2) We generate adversarially perturbed V, and check whether DANN can provide robustness.
(3) Note that DANN(RI, can be viewed as semi-supervised learning algorithm in the adver- sarial semi-supervised minimax threat model. Therefore, we check whether the adapted model F DANN(RI, D, U) is robust in the minimax model.
(4) (2) and (3) show that DANN(RI, ... significantly more robust in the test-time maximin threat model than in the minimax model, then the experiment goal is achieved.
Attacks. We use for attacks in this section (note that DANN is not designed for &oo attacks). We report more results about & attacks in Appendix E.3. We consider two categories of attacks:
Transfer attacks. Transfer attacks are a common class of attacks, where we transfer attacks on one model to the target model (in context, produced by a maximin defender strategy). In this paper we will mainly apply PGD attacks on adversarially trained models to generate transfer attacks.
Adaptive attacks. We also consider adaptive attacks', where adversary leverage the knowl- edge of the adaptation algorithm. To this end, we notice that test-time adaptation is typically an optimization objective (us(F, F, D,D, which gives rise to the following bilevel optimization ob- jective for the attacker:
maximize VEN(V)
(2)
subject to: F arg arg (FF,D,U-VV
To solve this bilevel optimization, we generalize the work of Lorraine & Duvenaud (2018) to an algorithmic framework, called Fixed Point Alternating Method (Algorithm 1 and 2). We consider two instantiations in this section: (1) FPAM Loww which is standard instantiation, where the outer CoAsN.
objective is L(F, V), and (2) J-FPAM
Cpawn where the outer objective is the exact same DANN objective LDANN. This is also natural instantiation where the adversary sets out fail the DANN
*Please refer Section for derivations.
7

---

Under review as a conference paper at ICLR 2021
optimization. Note that in this mization J-FPAM, because the objective becomes a more traditional minimax objective.
naturally apply the more traditional alternating opti-
Datasets. For the homogeneous case, consider MNIST and CIFARIO (i.e.. for both source and target domains). The homogeneous case represents the security problem considered in the classic threat model. For the inhomogeneous case, consider MNIST-MNNSTT (Ganin et al., 2017), and CIPAR10-20 PERIOO--CFAIOO-FOO0 (Hendrycks & Dietterich, 2019). MNIST-M is a standard dataset in unsupervised domain adaptation, and CIFARIOc is recent benchmark for evaluating neural network robustness against common corruptions and perturbations. For CIFAR 10c-fog, all 5 levels are combined and we perform 80-20 train-test split, which gives training set of size 40000 and test set of size 10000. (for space reasons, we combine all corruption levels for experiments in this section, results where we separately study different levels are reported in Section E.2).
Models. For MNIST, we use the original DANN architecture from Ganin et al. (2017) with slight modifications (e-g- adding batch normalization and dropout layers). For CIFARIO, we use pre- activation ResNets from He et al. (2016) for the prediction branch of DANN, and for the domain prediction branch we use the architecture from Sun al. (2020) (convolution layer). This architec- ture is slightly different from the the typical DANN architecture used for MNIST, which assumes vectorized feature and fully connected domain prediction branch.
Homogeneous (S 7)
Inhomogeneous (S + T)
Data MNIST
Defender
Accuracy
Data MNIST -MNISTMM
Defender
Accuracy
DANN
98. 98.79%
DANN
62.73%
MNIST
AdvS
98.14%
MNIST SRIST--MNTE -NIITTM
AdvS
35.95%
CIFARIO 10 CIFARIO
DANN
19.47%
LIFARIO-CIIAAL 10c-fog DANN
52.28%
AdvS
46.39%
CIFARIO-CCIFARIC,
AdvS 17.22%
Table I: Adversarial robustness under transfer attacks with &oc attacks. For MNIST we perturbation perturbation budget 8/255 (we found robustness degrades fast MNIST-
budget 0.3. For MNIST-M M. even adversarially trained models, so we choose CIFAR experiments we use perturbation budget 8/ /2555. For transfer attacks, DANN remained robust (albeit in a weaker threat model). Note that the numhers on adversarial training are only provided to qualitatively indicate the robutness level of DANN. The main goal is to provide evidence for separation.
number to illustrate relative performance). For all
Experiment results. (A) As sanity check first, we evaluate the accuracy of the adapted DANN models in the minimax threat model. Not surprisingly. the accuracy becomes very low (close to 0%),
Effect Target Size of DANN for CIFAR10 CIFARIO-edY
MNIST
CIFARIO->
MNIST
CIFARIO
MNIST-M
CIFARI0c-FOS fog
1
98.57%
56.37%
63.65%
67.99%
0.4
98.85%
55.62%
83.62%
56.06%
3
99. 99.20%
55.54%
83.52%
50.12%
2 g033-
4
99.42%
52.87%
88.01%
49.52%
S
98. 98.23%
52.31%
88.18%
55.09%
98.71%
53.49%
88.52%
57.25%
0.1
99.35%
52.57%
88.24%
55.85%
40
60
800
100
8
68%
52.16%
87.91%
54.89%
Epoch
Figure 1: Effect of test size on CIFAR10. We plot accuracy with DANN during the adaptation, for dif- ferent target size. Adversarial accuracy smaller target size gets lower accuracy.
Table 2: Adversarial robustness under FPAM EpanN adaptive attacks, with F = AdvT (i.c., the attacker initializes the initial model as the adversarially trained target model). DANN exhibits significant robustness even for large k.
train more epochs
Homogeneous (S T)
Inhomogeneous ($ T)
Data
Attacker
Accuracy
Data
Attacker
Accuracy
CIFAR10
J-FPAMS
Crann
42.22%
CIFAR 10 SCIFAC 10c-fog
J-FPAM &
31.70%
Table 3: Adversarial robustness under J-FPAM
Coanm adaptive attacks for CIFAR10 and CIFAR10c-fog
tasks. For FPAM& Zoww we use & = 20 (carly stopping according to Table 2). For J-FPAM 2 k = 100. Joint FPAM by far our most effective for DANN, though DANN still achieves non-trivial test time robustness, and demonstrate separation between maximin and minimax threat models.
2pAny" we use
8

---

Under review as a conference paper at ICLR 2021
which shows that DANN provides robustness in the minimax threat model. (B) Then in the max- imin threat model, Table summarizes the results under transfer attacks. In the homogeneous case, the adversarial accuracy DANN provided in the maximin model is comparable to the adversarial ac- curacy an adversarially trained model provided in the minimax model. And the adversarial accuracy becomes significantly higher in the inhomogeneous case (compared to using an adversarially trained source model). (C) In the maximin threat model, Table 2 summarizes the results under FPAM EpwnN attacks. Similar to the transfer attack case, DANN provides noticeable robustness. Note that since the defender moves after the attacker, he or she always applies adaptation "one more time". (D) In the maximin threat model, Table summarizes the results under J-FPAM Eoxmn far our most effective attacks against DANN, which decreases the robustness to 40% in the homo- geneous case on CIFARIO, 10, and to ~ 30% in the inhomogenous case on CIFARI 10 O-~CIFAR fo- Nevertheless, DANN still provides nontrivial robustness, and thus provides positive evidence to our hypothesis that DANN separates maximin and minimax threat models. (E) Finally. Figure I gives the robustness results under different target size. As we can see. the robustness of DANN degrades as the target size decreases, confirming our intuition that large target size benefits test-time robustness.
CoANN attacks. This is by
ROBUSTNESS OF OBLIVIOUS ADAPTATION AND FUTURE DIRECTIONS
We briefly explore the adversarial robustness of the recent data-oblivious test-time adaptation algo- rithms. Specifically, we focus on the TTT algorithm by Sun et al. (2020). Recall that a test-time adaptation algorithm is data-oblivious, it does not use the labeled training data D at the test time. TTT algorithm uses a specially trained (with self training) pretrained model, which we denote as PrTTT. Similar our DANN experiment, we conduct experiments on CIFARIO-CIFARIOO- (1) We found that transfer attacks against PrTTT can already break TTT algorithm (close to 0% accuracy). While this is not surprising as authors of Sun et al. (2020) have cautioned that TTT is not designed for adversarial robustness, this is in sharp contrast to our results with DANN. (2) We then use an adversarially trained model as the pretrained model for TTT. We found that this indeed increases the maximin-robusie robusttnes of TTT. However, the robustness is roughly the same (about 1% difference) as directly using the pretrained model. This indicates that the robustness mainly comes from the adversarially trained model, and TTT algorithm provides no robustness. This is again in sharp contrast to DAÑN. We list several implications as future directions:
Is transductive adversarial deep learning easier? Both the current work and the work of Gold- wasser et al. (2020) have given indication that adversarial robustness may be easier in the transduc- tive learning setting. To this end, we ask: Can we devise more effective algorithms against DANN, or we can prove its robustness in the maximin threat model? To this end, either confirming (the- oretically, for example) or refuting the maximin robustness of DANN is intriguing. To make this concrete, we propose the following (informal, qualitative) conjecture:
Conjecture (Transductive robustness of DANN (informal, qualitative)). If DANN can suc- cessfilly adapt from source domain (X*, Y²)) to (Xt, Yt) (no adversary), then DANN can provide test-time maximin robustness for bounded norm attacks on "X',Y")
Specifically, for a successful refutation, one must provide two (natural) domains where DANN can successfully adapt from the source to the target, but DANN fails to provide nontrivial transductive robustness for bounded norm attacks on the target domain.
The surprising power of distribution matching for robustness. It is somewhat surprising that a dis- tribution matching algorithm, such as DANN, can provide adversarial robustness (albeit in a weaker threat model), even when the target domain is already corrupted domain such as CIFAR 10c-fog- This means that for practical applications where we run an ML model in a batch mode on some collected unlabeled data, one can consider first apply UDA to produce an adapted model and then classify the batch, and this may provide adversarial robustness.
Does robust test-time adaptation always need access source data? We note that while DANN provides maximin robustness, it needs to access the labeled training data D at test time, and while TTT does not need D. it provides no maximin robustness. Can we achieve the best of both of worlds and get adversarially robust oblivious test-time adaptation algorithms?

---

Under review as a conference paper at ICLR 2021
REFERENCES
Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. Domain-adversarial neural networks. stat, 1050:15, 2014.
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 274-283. PMLR, 2018.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory learning from different domains. Machine learning, 79(1-2):511-75,
2010.
John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman. Learning bounds for domain adaptation. In Advances in neural information processing systems, pp. 129- 136, 2008.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C. Duchi, and Perey Liang. Unlabeled data improves adversarial robustness. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Alché-Bur Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 11190-11201, 2019a.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems, pp. 11190- 1201, 2019b.
Nicolas Courty, Rémi Flamary, Amaury Habrard, and Alain Rakotomamonjy. Joint distribution optimal transportation for domain adaptation. In Advances in Neural Information Processing Systems, pp. 3730-3739, 2017.
Constantinos Daskalakis, Stratis Skoulakis, and Manolis Zampetakis. The complexity of constrained min-max optimization, 2020.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversarial training of neural networks. Gabriela Csurka (ed.), Domain Adaptation in Computer Vision Applications, Ad- vances in Computer Vision and Pattern Recognition, pp.
Springer, 2017.
Shafi Goldwasser, Adam Tauman Kalai, Yael Tauman Kalai, and Omar Montasser. Beyond pertur-
bations: Learning guarantees with arbitrary adversarial test examples. CoRR, abs/2007.05145, 2020. URL https: http///aavvvv.e///e//eeeeaaae org abs/2007.00./////// 05145.
Ian Goodfellow. Defense against the dark arts: An overview of adversarial example security research and future research directions. CoRR, abs/1 1806.04169, 2018. URL http:://aaaaet
org/abs/ 1806. 04169.
lan J. Goodfellow. A research agenda: Dynamic models to defend against correlated attacks. CoRR, abs/1903.06293, 2019. URL http: btttp://arxee /ttp///aivvvv// org/ abs/ 1903. 06293.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings deep residual networks. In European conference computer vision, pp. 645. Springer. 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. Proceedings of the International Conference on Learning Represen- tations, 2019.
Thorsten Joachims. Transductive inference for text classification using support vector machines. In Ivan Bratko and Saso Dzeroski (eds.), Proceedings of the Sixteenth International Conference on Machine Learning (ICML 1999), Bled, Slovenia, June 27 30, 1999, pp. 200-209. Morgan Kaufmann, 1999.
10

---

Under review as a conference paper at ICLR 2021
Guy Katz, Clark W. Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. Reluplex: An efficient SMT solver for verifying deep neural networks. In Rupak Majumdar and Viktor Kun- cak (eds.), Computer Aided Verification 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I, volume 10426 of Lecture Notes in Computer Science, pp. 97-117. Sprige. 2017.
Daniel Kifer, Shai Ben-Davice and Johannes Gehrke. Detecting change in data streams. In VLDB, volume 4, pp. 180-191. Toronto, Canada, 2004.
Mingsheng Long, Jianmin Wang, Guiguang Ding, Jiaguang Sun, and Philip S Yu. Transfer joint matching for unsupervised domain adaptation. In Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 1410-1417, 2014.
Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. arXiv preprint arXiv:/502.0279) 2015.
Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hypernet- works. CoRR, 0t4/1802.09411 2018.
Alcksander Madry, Alcksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Confer- ence Track Proceedings. OpenReview.nec 2018.
Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Multiple source adaptation and the rényi divergence. In Proceedings of the Twenty-Fifhe Conference on Uncertainty in Artificial Intelligence. pp. 367-374, 2009.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning. IEEE Trans. Pattern Anal. Mach. Intell., 1978) 2019.
Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-tin time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Multi-adversarial domain adap- tation. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data. In Advances in Neural Information Processing Systems, pp. 5014-5026, 2018 8.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. Wasserstein distance guided representation learning for domain adaptation. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Rui Shu, Hung H. Bui, Hirokazu Narui, and Stefano Ermon. A DIRT-T approach to unsupervised domain adaptation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April -0 3, 2018, Conference Track Proceedingy. OpenReview.net, 2018.
Aman Sinha, Hongseok Namkoong, and John C. Duchi. Certifying some distributional robustness with principled adversarial training. In 6th International Conference on Learning Representa- tions, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceed- ings. OpenReview.net, 2018
Yu Sun, Xiaolong Wang, Liu Zhuang, John Miller, Moritz Hardt, and Alexei A. Efros. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020.

---

Under review as a conference paper at ICLR 2021
Jonathan Uesato, Jean-Baptiste Alayrac, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? In Hanna M. Wal- lach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 12192-12202, 2019.
Vladimir Vapnik. Statistccll learning theory. Wiley, 1998. ISBN 978-0-471-03000--4
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno A. Olshausen, and Trevor Darrell. Fully test-time adaptation by entropy minimization. CoRR, abs/2006. 10726, 2020.
Han Zhao, Shanghang Zhang. Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J
Gordon. Adversarial multiple source domain adaptation. In Advances in neural information processing systems, pp. 8559-85777 8570, 2018.
A PROOFS
A.1 PROOF OF PROPOSITION
Let F the family of models F can choose from. From the maximin inequality, we have that max min (L(F, V) FEF
min max (L(F, V)
FEF
Note that for the minimax, the max over V is also constrained to perturb features (as we want to find adversarial examples). we take expectation over V, we have then
max min (L(F, 5)) FEF
min max V)) FeF V
E
Note that
min max [)]]
< min E FEF V max Lmax [1}
E
FEF
which completes the proof.
A.2 PROOF OF PROPOSITION 3
In the maximin threat model, let take the strategy (T [*) as defined. Therefore the maximin adversarial gain is
max LT) m)] = max L(F", T)
E max LT))
which, by assumption, is bounded by
B MORE ON THREAT MODELS
B. ON THE MODELING
Why is the adversary constrained attack each point in Definition and Definition 3? This set up the rule of the threat model, because if not, adversary can pick any point that fails the model and repeats it arbitrary number times, which trivializes the model. In fact, such one-by-one attack is implicit in the original minimax objective, and is also the common practice of evaluating adversarial robustness (i.e., typical evaluation is to take test set, attack the points one by one, and then evaluate adversarial accuracy). Our threat model formulation makes this hidden assumption explicit. We refer readers to Goodfellow (2019) for more discussion one wants to break the i.i.d. sampling assumption.
12

---

Under review as a conference paper at ICLR 2021
Why in the minimax modeling, we do not assume A in Definition or A1 in Definition 3 to know the training algorithms? For example, T is not in the input of A in Definition 1. This is the case because in the inner most expression L(F, V) only depends on F, and F. is fixed after the training. Therefore, bringing in the training algorithm gives no additional information (note that this is also the typical practice of evaluating adversarial robustness in the classic setting, where we just attack the model). By contrast, this is not the case for the maximin threat model because attacker moves first and the model is not fixed yet, so in that case, a strong white-box attacker may need to take into consideration the adaptation algorithm, not just the pretrained model.
Why do we need to have two algorithms Ao, A1 in Definition 3? Ao attacks the semi-supervised learning algorithm, and thus has as input. On the other hand, A, attacks a model, so neither I nor T is in the input. Therefore, these two algorithms are different and are separated in the definition.
Note that the joint goal of Ao, A1 is fail the final model on V Ag can be totally different from A in order to fail the training. In this respect, Ao is like data poisoning the unlabeled data set in the semi-supervised learning. Unlike data poisoning though, we have the constraint that it must attack each point in V.
There are two algorithms Ao. in Definition 3, wouldn't Ag help the defender because leaks information about the target distribution. We have shown that the defender can choose to ignore the unlabeled data set, and this degenerates to the classic threat model the homogeneous case. Otherwise, the semi-supervised learning setting may indeed help the defender, because the information about the target domain (or more unlabeled data we are in the homogencous case) is leaked to the defender. This is, however, exactly why in these cases that this threat model lies between the classic threat model and the maximin threat model. In the maximin threat model, the attacker leaks U to the defender, and the defender is only required to predict U. In the semi- supervised threat model, things becomes more challenging, because finally we want to evaluate the model on independent samples.
In the experiments, we essentially instantiated the semi-supervised threat model, for example, as (An = FPA, Ai = PGD). Is that too weak? A strong adversary of course may want use strong attack Ao to fail the training, and thus fail the final model, and choosing FPA may indeed be a weak choice. However, we note that our goal in Step (3) of the experiment design is show that we can break the adapted model, and (FPA, PGD) already full-fills this purpose. As another note, in the maximin threat model, currently FPA is the most natural adaptive attack we can think of. Exploring stronger attacks is an intriguing future direction.
B.2 MORE EXTENSIONS AND RESTRICTIONS FOR MAXIMIN THREAT MODEL
Singleton. In the standard model, we allow U to be a batch which consists of many point. In the singleton restriction, U restricted to have only one point (for example, the test-time training algorithm by Sun et al. (2020) works in this setting).
Definition 4 (Online version). In the online maximin model, assume that we will sequentially receive batches of test samples, one after another, and we allow to leverage the previous samples received. We assume, for simplicity of this work however, that the batches are sampled independently from each other.
For the online maximin model, data obliviousness means that the agent cannot directly leverage the previous unlabeled samples he or she received. However, the agent can save the previous model parameters and use that.
B.3 SPECIALIZATION OF THE ADVERSARIAL SEMI-SUPERVISED MINIMAX THREAT MODEL
Adversarial training considered in Miyato et al. (2019); Shu et al. (2018). The setting becomes special case of our general minimax threat model by:
Instantiating An as a trivial adversary that does not do any perturbation. That is the defender receives set of clean set U of unlabeled features.
13

---

Under review as a conference paper at ICLR 2021
The defender adaptation strategy does adversarial training the unlabeled data, given by
(oylll))ppy((z +r,,)))))))))))
where r" = argmax ="arm=""==a"ammmpp))bb milrlss
Unsupervised Adversarial Training Uesato et al. (2019) This similar the above except that the source and target domains are equal.
C RELATED WORK
Adversarial robustness. Adversarial robustness of deep learning has received significant atten- tion in recent year. By far a standard approach is to do adversarial training directly for the attack type Madry et al. (2018); Sinha et al. (2018). Several recent works have studied adversarial training in a semi-supervised setting Carmon et al. (2019a); Uesato et al. (2019). As we have discussed, these work can be viewed instantiations in the adversarial semi-supervised minimax threat model. Our work can be viewed as one step further to study adversarial robustness in the transductive setting for deep learning.
Test-time adaptation and transductive learning, Transductive learning is a long line of research (for example, see the classic work of Transductive Support Vector Machine Joachims (1999)), where one attempts to leverage unlabeled data U to predict on specific instances of U Vapnik (1998). On the other hand, transductive learning has not received much attention in the deep learning setting. The recent proposals of rest-time adaptation Sun et al. (2020); Wang et al. (2020); Nado et al. (2020) seems to be an reincarnation of this idea, with the additional twist of leveraging a pretrained model. Our work considers test-time adaptation in an adversarial deep learning setting, which to the best of our knowledge, is new.
Unsupervised Domain Adaptation. A classic approach for analyzing domain adaption is based on H-divergence Kifer et al. (2004); Blitzer et al. (2008); Ben-David al. (2010). That theoretical framework is the basis for line of methods that uses adversarial training with neural networks to learn representations that are indistinguishable between source and target domain, in particular domain adversarial neural network (DANN) Ajakan et al. (2014); Ganin et al. (2017) and related techniques Pei et al. (2018); Zhao et al. (2018). Some other approach used different divergence notions, such as MMD Long et al. (2014; 2015), Wasserstein distance Courty et al. (2017); Shen et al. (2018), and Rényi divergence Mansour et al. (2009).
D ADAPTIVE ATTACKS AND BILEVEL OPTIMIZATION
In this section we present details of our considerations of adaptive attacks in the test-time adaptation setting. To start with, in the maximin threat model, for fixed D (labeled training set) and V (labeled test set), the game is maxy minz L(F,, V) (the adversary can only modify features, namely norm-bounded attacks, even though the test-time
U. without modifying the labels). We focus defense strategy can be agnostic to the attack type (i.c., the adaptation algorithm does not explicit leverage the attack type information). More specifically, the constraint of the adversary is that he can only perturb the feature of each point in V using norm-based perturbation: Namely for each (ar, y) the adversary can generate (a', .) where ||. - x'1l < E. we use the notation N(V) denote the neighborhood of V which includes all such legitimate perturbed datasets.
In our settings, the adaptation algorithm I is typically an optimization objective (which revises the model, we will demonstrate an instantiation using DANN below). We assume that this objective is Lua (F. F, D. U) where F is the we want to solve by optimization, F is (fixed) pretrained model (which can be 1 if a pretrained model not needed, see the DANN instantiation below), D is the labeled data set from source domain, and U = V|x is the test features (of V) from the target domain. Under these conditions, the maximin game can then be written as a bilevel optimization as follows:
14

---

Under review as a conference paper at ICLR 2021
maximize VEN(V)
L(F", V)
subject to: F E arg min Lma (,,,F, Vlx)
B
To incorporate more objectives, such as that the adversary can attack the inner minimization special case, we generalize the loss function of the outer optimization to the form of L(F, F, D, V). Note that compared to Lma we allow labeled data V for the attacker). This gives rise to the following objective (we have specifically generalized the outer maximization objective):
C(F",F,,VV)
maximize VEN(V)
(4)
subject to: F E arg Ci (F,F,D,U VIx)
In words, the outer optimization is the adversarial game where the adversary tries to find V to fool the defense, while the inner minimization solves the test-time adaptation objective Lus leveraging the information of U Vlx. in order to derive revised model 1 for predictions.
Example: An objective of maximin game with DANN as an defender. If the inner test-time defense DANN, then we know that F can be written as o o, and we can write the following bilevel optimization objective, where the inner minimization the DANN objective Ganin al. (2017):
maximize VEN(V)
L(F,V)
subject to: F arg min { (If D) + D) (o(o)))
o.
where d is distance function, which is realized domain discriminator (network). We refer readers to Ganin al. (2017) for more details. We remark that this objective with DANN gives evidence that: (1) Defenses using test-time adaptation are significantly different from test-time de- fenses discussed in Athalye et al. (2018). (2) The game is harder for the adversary because he or she needs to solve a harder optimization problem. To this end, we note that the test-time defenses discussed in Athalye et al. (2018) do not amount to bilevel optimizations. This is because that those defenses merely about sanitizing input x given fixed pretrained model, which is known to the adversary at the attack time and can thus be differentiated. On the other hand, the use of DANN as test-time adaptation algorithm makes bilevel optimization essential to the adversarial game.
Algorithm FIXED POINT ALTERNATING MEthOd FPAME,, [k, F]]
Require: A training dataset D, natural test V, an (optional) pretrained model F for test-time adaptation, and an integer parameter 2 0 (the number rounds).
I: If the pretrained model F equals 1, set Un = V|x. Otherwise, attack the pretrained model F on V. by fix- ing F' and solves the objective maxyeNce L(F, V) (i.e., the standard test loss of F), to get adversarially perturbed examples Vo. Set Uo Volx.
2: for 1, 1,2,..., k do
Solve inner minimization objective = arg ming Les F,D,Ui..). Solve outer maximization objective V. argmax ((F,F,D, Set Vixx. end for return Ug.
Solving the bilevel optimization (4). To solve the bilevel optimization, we propose two algorithm frameworks. Fixed Point Alternating Method (FPAM, Algorithm 1), and Joint Fixed Point Alter- nating Method (J-FPAM, Algorithm 2). These two algorithms generalize the work of Lorraine & Duvenaud (2018) (specifically, their Algorithm 2, "optimization of hypernetwork, then hyperpa- rameters"), ) to solve (4). Specifically, the joint optimization be effective in the case where the outer and inner objectives are similar to each other, in which case the objective degnerates to a more traditional minimax objective for optimization.
15

---

Under review as a conference paper at ICLR 2021
Algorithm JOINT FIXED POINT ALTERNATING METHOD J-FPAM [K,
Require: A training dataset D, natural test V, an (optional) pretrained model F for test-time adaptation,
integer parameter 2 0 (the number of rounds).
I: If the pretrained model I" equals 1, set Uo = Vlx. Otherwise, attack the pretrained model I on V. by fix- ing F and solves the objective max EN(V) L(F, V) (i.c, the standard test loss F), to get adversarially perturbed examples Vo. Uo = Volx.
2: for i= = 1, ..2.....k. do
3: for minibatch Va C do
4: Perform PGD the outer maximization objective: V argmax (EN(V,))) ((F.F,D,V) CCE
5: Set Vix-
6: Perform an SGD step on the inner minimization objective: F aV (F,,FF,,,,.
7: end for
8:
end for
return UR.
Instantiations of F with DANN. We now instantiate this framework by considering Lu as the DANN objective (with random initialization). In this case, there is no pretrained model F, the inner test-time adaptation objective simplifies to CDANN (F, U) We consider two instantiations of the outer maximization:
Loam Zoam Outer objective L(F, V). This is the most standard instantiation, where for the outer maximization, the adversary directly search for F to maximize the test loss of F on V.
FPAM
J-FPAM COANN" Outer objective is LDANN, the same DANN objective. This is natural and standard instantiation where the adversary uses the same DANN objective for both inner and outer objectives. In this case, J-FPAM can be naturally applied an alternating optimization based solver.
E MORE EXPERIMENTS
E.1 BASELINES: ACCURACY OF ADVERSARIALLY TRAINED MODELS.
Attack distance
Training PGD
Test PGD steps (A)
Nat.
Adv.
Data
steps (T)
test acc
test acc
CIFARIO
8/255
20
78.31%
46.39%
CIFAR1O-fO
8/255
20
71.39%
29.
MNIST
0.3
40
100
98.81%
18%
MNIST-M
8/255
100
96.50%
80.88%
Table 4: Accuracy of adversarially trained models in the minimax threat model. For the Nat. test ace column we do not use the adversary .A. Note that for the MNIST-M row is the case that we directly do adversarial training the clean labeled target data. This cannot be achieved in threat model because defender only sees perturbed unlabeled target data. Nevertheless, this row gives the best adversarial accuracy
hope for.
See Table for the results.
E.2
TRANSFER ATTACKS ON DIFFERENT CORRUPTION LEVELS
Corruption
AdvS on PGD(AdvS, (IFARIOc-fog)
AdvT PGD(AdvT, CIFAR1OC-fog)
DANN PGD(AdvT, CIFARI0c-fogl
DANN
level
CIFAR 10c-fog
37.99%
47.00%
70.73%
76.32%
2
23.07%
39.84%
63 63.05%
77.00%
2
13.72%
30.07%
59.01% 0
75.56%
8.45%
21.41%
52.93%
74.41%
5
3.37%
10.58%
40.58%
62.79%
Table 5: Transfer attacks (f.o attack) with different corruption levels.
16

---

Under review as a conference paper at ICLR 2021
See Table for the results.
E.3 TRANSFER ATTACKS WITH & ATTACKS
Homogeneous (5 T')
Inhomogeneous (S +)
Data
Defender
Accuracy
Data
Defender
Accuracy
MNIST
DANN
99.19%
MNIST MNIST-M
DANN
68.37%
MNIST
AdvS
99.07%
MNIST -INIST-MNIST-M
AdvS
44.30%
CIFARIO
DANN
69.57%
CIFARIO-CIIAR 10c-fog
DANN
69.98%
CIFARIO
AdvS
70.97%
CIFARIO-<CIFAR 10c-F
AdvS
37.32%
Table 6: Transfer attacks (&2 attacks with 80/255) for all tasks.
See Table for the results. Plots of fixed point attacks See Figure 2 for the results.
MNIST
Rand Int Piooa Rand Inil
0.56
0.900
ADVT Int.
0.985
2 0.32
0.0
0.975
Rand Private Rand Int AdVT In
0.48
0,970
MNIST, MNIST-MNIST-M
FPA Attack on DANN for CIFARIO CIFAR1OC-FFg
FPA APtackk on DANN for CIEAR10
Randinnt
Prizate Rand
0.65
0.85
0.80
K05
Int
0.5
Private Rand
inst
10
12
16
18
10
12
IPA
(b) CIFARIO, OIFAR1O-CIFARR00-0g
Figure 2: Plots fixed point attacks.
E.4 DETAILS ON EXPERIMENT
Models. For MNIST, we use the original DANN architecture from Ganin et al. (2017) with slight modifications (e.g. adding batch normalization and dropout layers). For CIFAR10, use pre- activation ResNets from He et al. (2016) for the prediction branch of DANN, and for the domain prediction branch
the architecture from Sun et al. (2020) (convolution layer). This architec- ture slightly different from the the typical DANN architecture used for MNIST, which assumes vectorized feature and fully connected domain prediction branch.
17

---

Under review as a conference paper at ICLR 2021
Training Details. For our CIFARIO and CIFARIOC experiment, obtain both the baseline standard pretrained model and the adversarial pretrained model via the following optimization schemes: SGD with 150 epochs, multiple step learning rate [0.1. 0.01, 0.001] with milestone at epoch [75, 125]. momentum 0.9, and weight decay 0.0005, and batch size 128. The baseline adver- sarial pretrained model is trained with 7-step PGD. In evaluating the test time robustness for transfer attacks, we generate the adversarial samples with 20-step PGD. In evaluating the test time robustness for adaptive attacks, we evaluate with 7-step PGD.
For our MNIST and MNIST-M experiment, we obtain both the baseline standard pretrained model and the adversarial pretrained model via the following optimization schemes: ADAM with 100 epochs, learning rate 3 X 10 and batch size 128. The baseline adversarial pretrained model is trained with 40-step PGD. In evaluating the test time robustness for transfer attacks, we generate the adversarial samples with 100-step PGD. In evaluating the test time robustness for adaptive attacks, we evaluate with 40-step PGD.
The optimization scheme for DANN during test time adaptation (both transfer and adaptive attack experiments) follows as: ADAM with learning rate 0.004, batch size 128.
F PROOF OF THEOREM 1
We prove this theorem by a series of lemmas.
Lemma (Part (1)). In the semi-supervised minimax threat model, the learned model F I(T(D), D, D,U), U). by any algorithms T and r, must have a large error:
E {(L(E.Vn)
1 2 2 (1-
(5)
where the expectation is over the randomness of D, V and possible algorithm randomness.
Proof. This follows from Theorem 1 in Carmon et al. (2019b) (or equivalently Theorem in Schmidt et al. (2018)). The only difference of our setting from their setting is that we additionally have unlabeled data for the algorithm. Since the attacker can provide x' = x, the problem reduces to problem with at most n + I data points in their setting, and thus the statement (1) follows. O
Transductive learning algorithms (T, I):
To prove the statement (2), give concrete algorithms learning algorithms that achieves small test error on x'.
High-level structure of the learning algorithms. At the high level. the learning algorithms work as follows: At the training time we use part of the training data (denoted as D2 to train a pretrained model f)), and part of the training data (denoted the test time, upon receiving U, we use U to tune 0, and get two large-margin classifiers, 04 and 0_, which classify x' as +1 and 1, respectively. Finally, we check these two large margin classifiers on D2 (that's 's where D2 is used), and the one that generates smaller error wins and we classify x' into the winner class.
D1, reserved test-time adaptation). Then,
Detailed description. More specifically, the learning algorithms (T,I') work as follows:
1. Before game starts. Let m' K ng, m 10ng- We split the training set D into two subsets: D2 == {<;,,,)) y))m and D == {(rm 44x fm ) D2 will be used to train pretrained model at the training time, and Di will be used at the test time for adaptation.
2. Training time. T uses the second part D2 to compute a pretrained model, that is, a param- eter vector:
1 m m
0m
Ø
(6)
||0m 112
3. Test time. On input U, I uses D1 and U to perform adaptation. At the high level, it adapts the pre-trained # along the direction x', such that it also has a large margin on x', and also it makes correct predictions on D) with large margins. More specifically:
18

---

Under review as a conference paper at ICLR 2021
(a) First, I' constructs two classifiers, 0+ and 8_, such that 0+ classifies x' to be +1 with a large margin, and 8_ classifies x' to be -1 with a large margin. Specifically:
x' == x' /|1=>111 12
? = 12/1
(0) x²'
7+
0+ n+31,
0+ 0
(8)
|| 1x'
7- -Y (8) TT
0_ + 7 7-
0_ 8- 01100-112
||2||2
where 0+ and 0_ are viewed as the parameter vectors for linear classifiers. Note that 04 is constructed such that 0_x'/ ||>1||2 >/1|='112 1/2, and 0_ is such that 0 0_x'/ /||='112 > 11/2. (b) Finally, I' checks their large margin errors on Dj. Formally, let
n1 1/2
=
(10)
m
(rr = E (a.s)) Tyt T x S t),
(11)
erT. (0)
1y:00 xz t].
(12)
If (0+) S (f_) then sets F(z) := sgn(0 and classifies x' to +1; otherwise, sets F(x) == #f and classifies x' -1
Lemma 2 (Part (2)). In the maximin threat model, there is an absolute constant c. > O, such that for the T and described above, the adapted model F = T(T(D), D, has a small error:
B(((F Se-ck
(13)
Proof. Now, we have specified the algorithms and are ready to prove that w.h.p. F(x') is the correct
label y. By Lemma 3, y(err,(0-) - e0+)))+)) Z ca_ with probability 2 e - Then by the Hoeffding's inequality, Dí sufficiently large ensure y(err, err, (0_)) probability 2 1 2e K/12 This proves the statement (2).
yng
with
Lemma 3. There exists absolute constants c4 such that with probability > 1 e
y(err, err, (0+)) 2
C4
(14)
TTo
F.1
PROOF OF LEMMA 3
Without loss of generality, assume y +1. The proof for y follows the same argument.
Note that
(f), ((m x s
(15)
(N(z 0, 0, t) # -
(16)
Q
(17)
o 0
where
1 +00
Q(x)
c -///2 dt.
(18)
V2T
First, consider 0.
( 0
T0
( Q
Q (s), where
(19)
o
19

---

Under review as a conference paper at ICLR 2021
By Lemma 4, we have with probability 2 1 ca(//no)) 1/4 miaffm(d//mi 11/43
-1/2
/ A
no
nu
ng 1/8
1+ c1
(20)
o 0100 |0|
2
d
m
d
-1/2
" no
nu
72
1/8)
1 1 c1
(21)
a|00| 2
d
m
d
which gives
0 t
n2
1/8
V (v
120
170
-1/2
I c1
(22)
o 0100122 2
d
m
11 0 1
110
1/8
no
10
-1/2
010012
2
c1
d
d
+
m7
(23)
Since m
10mg and d >> no. we have
|s| =
S 1.
(24)
0110112
Next, we have
11 0+ 1
Q (s+), where 84 1. - t
(f1, Q
(25)
a a110+11 l12
0
p 0_
= =Q (s-), where 8_ == 1 100_
(f-, = Q
(26)
0110-112
0
We now check the sizes of 84 and 8_.
/ T 0+ -
1T 0 - t
S+
(27)
4 TTO, 04 p TT
(28)
((1 - 114112))
n + TT))
(29)
0110111
Then by definition and bounds in Claim 1,
2 + 40 S 42.
|84 s| 5 71
(30)
Since ss| is bounded by I, we know |s+| is also bounded by 43. Similarly, Is_ s| and thus |s_| are also bounded by some constants. Furthermore,
1
s+
(μ [u 1 TTT
(31)
1
1 0 + 1 7- T
1 T + n
p T T'
(32)
1|0111
10-112 ll2
By Claim 2, we have 110_ 12 = 110 112.. So
8+ - S_
a a110+112 lla
(7+ T To' -
(33)
1
(n+ 17- p TTx
(34)
0110+112
11 To.
(35)
"110+112"
Vd
(36)
2
(37)
4
20

---

Under review as a conference paper at ICLR 2021
Now are ready to bound the error difference:
err, (0_) exr, (0+) = Q(s - Q(++)
(38)
( 12
(39)
V2T
(s_ - s+) x min(e 2212 . 44/
(40)
22T
C1
(41)
/0p
for some absolute constant c4 0.
F.2 TOOLS
Claim 1. There exists absolute constant C3 >> 0, such that with probability 2 1 - em
oVd/4
S || 22 S 2o
(42)
2
1
1
0
no
m
< TT μ 20
V no
m
< 100,
(43)
-
-eVa/2 S 0T 1 < 2ddd,
(44)
d/2 4/25 1T s 3d/2
(45)
1
8
8
S 7 -
(46)
2 0
2 0
1 &c
1 8c + 2
S
1- S
(47)
2
0
σ
Proof. First, since x' - + oς + for Ç ~ (O0 I), with probability 2 1 - e for an absolute
constant d' > 0, we have:
vd/2 < [kll2 S 3vd/2,
(48)
| 2 ovd/2 ||u||2 - ||1|2 2 ovd/4,
(49)
| s 03vd/2+ + || + |1112 S 20
(50)
By Lemma 4, with probability 2 c k
1/2
ØT # < 20
ng
< 20
m
(51)
d
m
-1/2
0 4 2 2
1
n10
n10
0
m
(52)
d
4
Also, with probability
e cck
T0T<l < 2KK.
(53)
Finally,
10T vl < 10112|111100 Sevd.
(54)
Then
OT a' 0 (n + o6 +V)
(55)
$ 10 u| + 0 <|+ 10 ml
(56)
< $25
m V
+ 2Ko + evd
(57)
no
< 2cVd.
(58)
21

---

Under review as a conference paper at ICLR 2021
and
0 x''= = 0 af
oc +v)
(59)
2 0/2 Ko evd
(60)
2 -edd/2
(61)
For μ Jx,, we have with probability 2 1 - e
# N = (M + = o5 + v)
(62)
/ x' S 11/122 + 2K || + clllllavd 3d/2,
(63)
# x' 2 11112 2kol|al12 - cl d/2.
(64)
By definition:
1 =
1/
0 22/111 //|>2112
(65)
2
so
1
8c/0 S
77
- 2
+ 8e/o.
(66)
2
Similarly,
1
1
8E/O 1 7 5
+ 88/0.
(67)
2
2
Claim 2.
10+ = 10+1|==100111 =100-12-
(68)
Proof. We have by definition:
||0_ 11 ||0 + x
(69)
+ n/2 + 27 TT
(70)
| 1| = 110 + 2
(71)
1 + n + 2
(72)
Then
|0_ 112 110 112 n2 + 0T n2= 2n TT
(73)
(1- - 1) + 1+ + -0 1'(11-111111 - -1+)
(74)
= (n- - 1))1((1 + +,++ + +0 37
(75)
= (7- - --+)[-20 x' //|xol12 + 20 17
(76)
= O.
(77)
This completes the proof.
Lemma 4 (Paraphrase of Lemma I in Carmon et al. (2019b)). Let 6.
1 Em Y: There exist
02 absolute constants co2 C2 such that under parameter setting (1) and d n/ Co.
0²
pm (n²
12
no
no
n0
1/8
0m.))
12
>
(V
d
1 - C1
d
(78)
a 0 12
no
no
n10
1/8)
1 1 c1
(d)
(79)
)m
d
with probability 2 1 (ca(d/no)'/ 1/4 min (dd/mn 1/43
22


Reviews:
[{"review_id": "dwEwHK1pMk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies test-time adversarial robustness through a maximin framework and illustrate non-trivial robustness (under transfer attack) using domain adversarial neural network (DANN) to Linf-norm and unseen adversarial attacks. While I agree that test-time adaptation is an important and practical approach for adversarial robustness, the current version, in my opinion, does not deliver significantly novel insights, nor considering a reasonably practical threat model. My main concerns are detailed as follows.\n\n1. Impractical threat model: At test time, the maximin threat model (Def. 2) assumes the attacker to make move prior to the defender's action, which limits the attacker's ability and weakens the robustness evaluation. More importantly, it may create a false of security/robustness, as pointed out by (Athalye et al. 2018), that the robustness gain may actually come from information obfuscation and thus the results may fail to provide meaningful robustness evaluation. Although the authors mention the maximin threat model is a weaker (attack) model and part of the goal is to find an adaptation method that is \"good\" in this attack-move-first scenario, I couldn't see the practical utility and contributions form these results. Even in the test-time adaptation setting, the \"defender-move-first\" setting should be more practical. Better motivation and use cases are needed to justify why the considered setting is important.\n\n2. Due to the assumption of the considered maximin threat model, the experiments are limited to comparing accuracy on transfer attacks, which provide limited understanding of the true robustness of the victim model. Moreover, the baseline models in comparison are too weak and unfair. To have a fair comparison, the authors are suggested to compare robustness on robust models such as TRADES [R1] and adversarially trained models with unlabeled data [R2,R3], so that the baseline models also use unlabeled data. If DANN shows limited robustness against white-box attacks but stronger robustness against transfer attacks, one can only conclude that transfer attack is a weaker threat model, which is a known result. I do not see new insights from the reported results.\n\n[R1] https://arxiv.org/abs/1901.08573\n\n[R2] https://papers.nips.cc/paper/9298-unlabeled-data-improves-adversarial-robustness.pdf\n\n[R3] https://deepmind.com/research/publications/Are-Labels-Required-for-Improving-Adversarial-Robustness\n\n3. In addition to the issue of impractical threat model and lacking motivation, this paper contains too many high-level discussions accompanied with limited or even no empirical evidence. The theorem presented in the paper is a natural use of maximin inequality.  In my opinion, the current version requires significant revision. I suggest the authors carefully motivate the research goals (especially answering why the studied problems and settings are important), consolidate the claims on test-time robustness with convincing evidence, and make a broader connection to other test-time defenses other than DANN.", "extra_scores": {}, "extra_text": {}}, {"review_id": "G7_fyB26TDe", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explores adversarial robustness in a new setting of test-time adaptation. It shows this new problem of “test-time-adapted adversarial robustness” is strictly weaker than the “traditional adversarial robustness” when assuming the training data is available for the “test-time-adapted adversarial robustness”. The gap between the two problems is demonstrated by the simple DANN solution which has good “test-time-adapted adversarial robustness” but bad “traditional adversarial robustness”. The paper also explores the subcase of “test-time-adapted adversarial robustness” when assuming the training data is not available and provide some initial result. \n\nThe paper has clear strong points. It aims to tackle an important problem, the “test time adaptation allowed” extension for the “adversarial robustness”. The paper has a nice global picture and a clear position for this piece of work. I particularly like the way the author approaches the problem. They start from the most abstract and fundamental question, “test-time-adapted adversarial robustness” v.s. “Classic adversarial robustness”, or transductive learning v.s. Inductive learning in the setting of adversarial robustness. To proceed the thinking, they develop a good theoretical framework (the two threat models from definition 1 and definition 2) to formulate the two problems. And then consider a middle setting (definition 3) between the classic minimax and new maximin threat model.  Such frameworks help to develop theoretical understanding like one setting the strictly weaker than another (proposition 1). \n\nThe weak points of the paper are mainly about the presentation. The paper currently is very dense. Sometimes I feel the author may assume the reader has certain domain knowledge without explanations. For example, dataset “CIFAR10c-fog” appears very early in the introduction, but it is never clearly explained what it is, what is the main difference between it and CIFAR 10. After reading the paper the only impression I get is they not homogeneous. There also some places in the method, I can not understand,\nFor the maximin threat model, why the game is maximation over U instead of A (the left side of the equation from proposition 1).\nWhy there are A0 and A1 in the Adversarial semi-supervised minimax threat model? And why in this game, A0 and A1 are jointly maximizing L(\\tile{F},\\tilde{V}’)?\n\nMore question about experiments:\nFor FPA attacks, is there any baseline method we can compare the DANN with? Currently, I am not sure how to evaluate the performance of DANN. \nIn experiment (D), it says, “we also evaluate the accuracy of the adapted DANN models in the minimax threat mode”. But where are the results? \n\nI understand it is pretty hard to squeeze so many contents into limited 8 pages. Personally, I think it is helpful to cut off some content and make the main paper more clear, well organized, and strong. \n\nUnfortunately, I am not an expert in adversarial robustness. I did not check the technique and experiments deeply. My current score assumes no fatal flaws exist in the theory and experiment. My rating will be changed according to other reviewers’ comments and the author’s updates. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8vL_s5UM8xe", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores a new paradigm referred to as \"adversarial robustness of test-time adaptation\". The authors propose a new threat model called a \"maximin game\" instead of the widely-accepted \"minimax game\". In this new setup, the threat model becomes weaker as it first presents the unlabelled adversarial examples to the defender to get prepared. The authors also run some experiments showing that DANN seems to be a preferred method in this new setup. The paper is in general interesting with many novel definitions and discussions, but I do not see how this paper can benefit the community other than a set of new research directions with preliminary results. \n\n1. First of all, I consider the writings of the introduction not professional: the introduction is full of notations, definitions, and even datasets and results that are only elucidated later in the paper. This writing style posted a great challenge to understand and appreciate this paper. \n\n2. The paper interestingly introduces a set of new ideas/definitions. However, I wonder how these new research directions can benefit the community or industry. The authors explained that \"this question is also of practical importance since these 'new' solutions may possess desirable properties that good solutions in the minimax threat model may lack.\" and then the authors mentioned that \"one such property is that the defender solution is attack agnostic\". However, it seems the rest of this paper does not correspond to these remarks, e.g., \n * The experiments (in the main paper body) are within $\\ell_\\infty$ norm, so it seems there is no evidence about attack agnostic. \n * I did not find related discussions of other \"desirable properties\" later in this paper. \n * The authors also mentioned theoretical interest, yet I don't think, with the current contents, this paper can be appreciated as a theoretical paper. \nTherefore, I suggest the authors further clarify their contributions, with emphasis on how their contributions can benefit the community and the industry. The current presentation leaves the impression that the paper is a set of new definitions without real-world implications. \n\n3. The paper also says \"This threat model enables us to study whether a large test set can benefit a defender for adversarial robustness\", yet I do not see any experiments (in the main paper) that correspond to this discussion. The appendix seems lacking this discussion either. Similarly, this kind of argument or claims without validations makes the paper read like an arbitrary set of definitions. \n\n4. Following the \"attack agnostic\" argument above, Definition 2 suggests that the solution is also attack specific, as the defender needs to update the model according to the attacked samples. Although this update happens at test time, it happens before the evaluation. \n\n5. The experiments are only about DANN, and seemingly leaving the details of AdvS unexplained. Is the AdvS the same architecture but without the domain component? Then why not also report performances on S? Without these details, it's impossible to evaluate and appreciate the experimental results. \n\n6. While it seems the authors' focus is data-oblivious test-time adaptation, which, as I understand, does not allow the usage of labeling training data. Yet a major focus of the paper is DANN, which allows such usage. I'm aware that the authors have mentioned the differences in multiple places, but if the main focus is DANN, I suggest the authors rephrase the paper to be about the setup DANN builds upon, instead of targeting a broader domain but only supporting the claims with experiments limited to a specific setup. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "5umAQSOvvyu", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Contributions ###\n* The authors formalize test-time adaptation as a maximin threat model and contrast it with the \"threat model for classic adversarial robustness\" and the \"adversarial semi-supervised minimax threat model\".\n* The paper studies test-time adaptation via DANN and its robustness in the maximin and minimax threat model. DANN has larger robustness against the proposed attacks in the maximin threat model than in minimax, providing evidence that maximin  is a setting that is beneficial for the defender. However, it remains unclear if this would also be the case against stronger attackers.\n\n### Significance ###\n\nIncreasing robustness against adversarial robustness is a topic both relevant to basic research and also more applied research. Test-time adaptation is currently one of the most promising directions and this paper is thus very timely. Its formalization of a threat model for this domain can guide future research in this direction.\nThat being said, the main method investigated (DANN for test-time adaptation) seems less practical and significant since it requires large test batch sizes and also huge computational resources at inference time. \n\n### Originality ###\n\nThe paper has only limited novelty: while the formalization of test-time adaptation as maximin threat model is very thorough, it is also mostly straightforward. Also studying DANN as a test-time adaptation is an incremental idea once one assumes that inference is conducted on large batches. Nevertheless, I think papers like this that consolidate recent research directions into a coherent framework are valuable.\n\n### Clarity ###\n\nGenerally, the introduction and the part outlining the threat model are written very clearly and I enjoyed the thorough formalization. A few details could be improved (even though this is a bit nitpicking):\n* the introduction is very technical; it could be more high level and shorter and leaving the more technical parts to Section 2.\n* Threat models could mention that white-box knowledge of the attacker is assumed\n* it is not really clear why test inputs are assumed to be labeled in the threat models\n\nThe experimental part is less clear unfortunately:\n* since DANN is a central method in the experiments, it should be briefly summarized to make the paper self-contained\n* it is not really clear to me how specifically DANN is trained here (what is the objective?). Specifically, is the loss on the labeled data based on standard training or adversarial training? \n* generally, all training details (optimizer, learning rate, batch size, number of epochs etc.) are missing. the experiments cannot be reproduced in the current form.\n\n### Quality ###\n\nI have some serious doubts about the quality of the empirical evaluation.\nAs the authors state \"While we are not able to prove the existence of a defender solution\nthat separates maximin and minimax threat models, we design and implement experiments to\nprovide evidence that such a strategy may actually exist.\" Thus, one of the main hypothesis of the paper (maximin being a strictly weaker threat model than minimax) depends on the design of experiments. My main concern here is that the attackers used are not actually strong:\n* A transfer attack should clearly only be a baseline that should get surpassed by stronger adaptive attackers\n* However, the fixed point attack seems to be weaker than the transfer attack. In particular its effectiveness in the homogeneous setting for CIFAR-10 reduces with the number of iterations k. This does not seem to be a reliable way of evaluating DANN's maximin robustness\n* I would imagine a strong adaptive attack could be built by generating adversarial inputs  with the objective of minimizing the loss of DANN's domain label classifier. This would minimize the impact of test-time adaptation via DANN (because both domains would have already very similar representations). In any case, more effort should be spend on designing a strong attack against a DANN-based test-time adaptation.\n\nI think it is also confusing to add evaluations in the inhomogeneous setting. Clearly, DANN gives strong results here because dealing with domain shift is what it was designed for. However, the paper is on adversarial robustness and confounding this with domain shift is misleading (my suspicion is it was mainly added because DANN clearly outperforms adversarial training on the source data in this setting; to proof me wrong, the authors should add a better motivation for this setting or remove it). \n\n### Recommendation ###\n\nTo summarize, I think the paper provides a strong formalization of a threat model for test-time adaptation. However, it does not introduce novel approaches, lacks clarity in experimental details, and its empirical results are less trustworthy given that the proposed attacks are rather weak. I thus recommend rejecting the paper in its current form. I encourage the authors to address the current shortcomings; the paper clearly has potential in general.\n\n### Final Recommendation after Author Response ###\nI would like the authors for the very active and productive rebuttal period.  Actually, the current version has significantly deviated from the initially submitted version. I have read most of the paper a second time to take all changes into account. The authors considerably improved the paper and have addressed some of my concerns. I increase my score to 5. The reason for not increasing the score to 6 or 7 is mainly that clarity of the paper is lacking. To give two examples:\n * Presumably because of the many changes during the rebuttal, the organization, structure and the quality of the manuscript is not always sufficient: to name one of several examples, the core Theorem 1 comes without any reference to its proof or any proof sketch in the main paper (there exists a proof in the appendix, finding of which required scrolling through the entire appendix).\n * As stated in my initial review, since DANN is a central method in the experiments, it should be briefly summarized to make the paper self-contained and also the specific training\nI think the core issue with the submission is that there is simply too much content for one paper:\n * formalizing 3 threat models\n * a proof of separation of maximin and minimax\n * empirical evaluation  on two datasets (MNIST, CIFAR10), three attacks (transfer, two adaptive attacks), three defenses (DANN, AdvS, TTT), and two settings (homogenous, inhomogeneous)\nBecause of the page limit, a lot of details have been moved into the appendix, making the paper difficult to read. Moreover, even taking the appendix into account, details remain unclear such as how DANN was trained. Moreover, I do not find it convincing to move related work to the appendix; relating the current work to other work should be an integral part of the main paper.\n\nMy recommendation for the authors would be to strengthen the focus of the paper: I think the inhomogeneous setting does not contribute much and could be removed. Also I don't see much value in MNIST and the weak transfer attacks. Moreover, the parts on preliminaries and threat models is too long for a conference paper and could be shortened. \nI think if focus were improved and the main document became more self-contained, the quality of the work would be considerably improved. For the time being, I see the submission still marginally below the acceptance threshold.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies test-time adaptation in the context of adversarial robustness. The key idea is to use a maximin framework, which illustrates non-trivial robustness (under transfer attack) using domain adversarial neural network (DANN) to Linf-norm and unseen adversarial attacks. The approach is sound, well grounded, and quite logical. Results demonstrate the effectiveness.

However, there exists some limitations, for example, 1) The adaptive attack results are concerning. Comparing Table 1 and Table 3, with the adaptive attack (J-FPAM), the accuracy in the homogeneous setting is below that of adversarial training (Adv S) in Table 1, which somehow echoes my concern on not testing the transductive setting using strong attacks. It seems that adversarially trained models can better defend the adaptive attacks. 2) The paper says "This threat model enables us to study whether a large test set can benefit a defender for adversarial robustness", yet there is no any experiments in the main paper that correspond to this discussion. The appendix seems lacking this discussion either. 3) Due to the page limit, a lot of details have been moved into the appendix, making the paper difficult to read.

In the end, I think that this paper may not be ready for publication at ICLR, but the next version must be a strong paper if above limitations can be well addressed.
Decision: Reject