Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper explores adversarial robustness in a new setting of test-time adaptation. It shows this new problem of “test-time-adapted adversarial robustness” is strictly weaker than the “traditional adversarial robustness” when assuming the training data is available for the “test-time-adapted adversarial robustness”. The gap between the two problems is demonstrated by the simple DANN solution which has good “test-time-adapted adversarial robustness” but bad “traditional adversarial robustness”. The paper also explores the subcase of “test-time-adapted adversarial robustness” when assuming the training data is not available and provide some initial result. 

The paper has clear strong points. It aims to tackle an important problem, the “test time adaptation allowed” extension for the “adversarial robustness”. The paper has a nice global picture and a clear position for this piece of work. I particularly like the way the author approaches the problem. They start from the most abstract and fundamental question, “test-time-adapted adversarial robustness” v.s. “Classic adversarial robustness”, or transductive learning v.s. Inductive learning in the setting of adversarial robustness. To proceed the thinking, they develop a good theoretical framework (the two threat models from definition 1 and definition 2) to formulate the two problems. And then consider a middle setting (definition 3) between the classic minimax and new maximin threat model.  Such frameworks help to develop theoretical understanding like one setting the strictly weaker than another (proposition 1). 

The weak points of the paper are mainly about the presentation. The paper currently is very dense. Sometimes I feel the author may assume the reader has certain domain knowledge without explanations. For example, dataset “CIFAR10c-fog” appears very early in the introduction, but it is never clearly explained what it is, what is the main difference between it and CIFAR 10. After reading the paper the only impression I get is they not homogeneous. There also some places in the method, I can not understand,
For the maximin threat model, why the game is maximation over U instead of A (the left side of the equation from proposition 1).
Why there are A0 and A1 in the Adversarial semi-supervised minimax threat model? And why in this game, A0 and A1 are jointly maximizing L(\tile{F},\tilde{V}’)?

More question about experiments:
For FPA attacks, is there any baseline method we can compare the DANN with? Currently, I am not sure how to evaluate the performance of DANN. 
In experiment (D), it says, “we also evaluate the accuracy of the adapted DANN models in the minimax threat mode”. But where are the results? 

I understand it is pretty hard to squeeze so many contents into limited 8 pages. Personally, I think it is helpful to cut off some content and make the main paper more clear, well organized, and strong. 

Unfortunately, I am not an expert in adversarial robustness. I did not check the technique and experiments deeply. My current score assumes no fatal flaws exist in the theory and experiment. My rating will be changed according to other reviewers’ comments and the author’s updates. 

review_title: Interesting topic and solid analysis

final_score: rating: 3: Clear rejection
review_content: This paper studies test-time adversarial robustness through a maximin framework and illustrate non-trivial robustness (under transfer attack) using domain adversarial neural network (DANN) to Linf-norm and unseen adversarial attacks. While I agree that test-time adaptation is an important and practical approach for adversarial robustness, the current version, in my opinion, does not deliver significantly novel insights, nor considering a reasonably practical threat model. My main concerns are detailed as follows.

1. Impractical threat model: At test time, the maximin threat model (Def. 2) assumes the attacker to make move prior to the defender's action, which limits the attacker's ability and weakens the robustness evaluation. More importantly, it may create a false of security/robustness, as pointed out by (Athalye et al. 2018), that the robustness gain may actually come from information obfuscation and thus the results may fail to provide meaningful robustness evaluation. Although the authors mention the maximin threat model is a weaker (attack) model and part of the goal is to find an adaptation method that is "good" in this attack-move-first scenario, I couldn't see the practical utility and contributions form these results. Even in the test-time adaptation setting, the "defender-move-first" setting should be more practical. Better motivation and use cases are needed to justify why the considered setting is important.

2. Due to the assumption of the considered maximin threat model, the experiments are limited to comparing accuracy on transfer attacks, which provide limited understanding of the true robustness of the victim model. Moreover, the baseline models in comparison are too weak and unfair. To have a fair comparison, the authors are suggested to compare robustness on robust models such as TRADES [R1] and adversarially trained models with unlabeled data [R2,R3], so that the baseline models also use unlabeled data. If DANN shows limited robustness against white-box attacks but stronger robustness against transfer attacks, one can only conclude that transfer attack is a weaker threat model, which is a known result. I do not see new insights from the reported results.

[R1] https://arxiv.org/abs/1901.08573

[R2] https://papers.nips.cc/paper/9298-unlabeled-data-improves-adversarial-robustness.pdf

[R3] https://deepmind.com/research/publications/Are-Labels-Required-for-Improving-Adversarial-Robustness

3. In addition to the issue of impractical threat model and lacking motivation, this paper contains too many high-level discussions accompanied with limited or even no empirical evidence. The theorem presented in the paper is a natural use of maximin inequality.  In my opinion, the current version requires significant revision. I suggest the authors carefully motivate the research goals (especially answering why the studied problems and settings are important), consolidate the claims on test-time robustness with convincing evidence, and make a broader connection to other test-time defenses other than DANN.
review_title: Interesting framework, but requires better motivation and utility

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper explores a new paradigm referred to as "adversarial robustness of test-time adaptation". The authors propose a new threat model called a "maximin game" instead of the widely-accepted "minimax game". In this new setup, the threat model becomes weaker as it first presents the unlabelled adversarial examples to the defender to get prepared. The authors also run some experiments showing that DANN seems to be a preferred method in this new setup. The paper is in general interesting with many novel definitions and discussions, but I do not see how this paper can benefit the community other than a set of new research directions with preliminary results. 

1. First of all, I consider the writings of the introduction not professional: the introduction is full of notations, definitions, and even datasets and results that are only elucidated later in the paper. This writing style posted a great challenge to understand and appreciate this paper. 

2. The paper interestingly introduces a set of new ideas/definitions. However, I wonder how these new research directions can benefit the community or industry. The authors explained that "this question is also of practical importance since these 'new' solutions may possess desirable properties that good solutions in the minimax threat model may lack." and then the authors mentioned that "one such property is that the defender solution is attack agnostic". However, it seems the rest of this paper does not correspond to these remarks, e.g., 
 * The experiments (in the main paper body) are within $\ell_\infty$ norm, so it seems there is no evidence about attack agnostic. 
 * I did not find related discussions of other "desirable properties" later in this paper. 
 * The authors also mentioned theoretical interest, yet I don't think, with the current contents, this paper can be appreciated as a theoretical paper. 
Therefore, I suggest the authors further clarify their contributions, with emphasis on how their contributions can benefit the community and the industry. The current presentation leaves the impression that the paper is a set of new definitions without real-world implications. 

3. The paper also says "This threat model enables us to study whether a large test set can benefit a defender for adversarial robustness", yet I do not see any experiments (in the main paper) that correspond to this discussion. The appendix seems lacking this discussion either. Similarly, this kind of argument or claims without validations makes the paper read like an arbitrary set of definitions. 

4. Following the "attack agnostic" argument above, Definition 2 suggests that the solution is also attack specific, as the defender needs to update the model according to the attacked samples. Although this update happens at test time, it happens before the evaluation. 

5. The experiments are only about DANN, and seemingly leaving the details of AdvS unexplained. Is the AdvS the same architecture but without the domain component? Then why not also report performances on S? Without these details, it's impossible to evaluate and appreciate the experimental results. 

6. While it seems the authors' focus is data-oblivious test-time adaptation, which, as I understand, does not allow the usage of labeling training data. Yet a major focus of the paper is DANN, which allows such usage. I'm aware that the authors have mentioned the differences in multiple places, but if the main focus is DANN, I suggest the authors rephrase the paper to be about the setup DANN builds upon, instead of targeting a broader domain but only supporting the claims with experiments limited to a specific setup. 
review_title: A potentially interesting new research direction, but many questions need to be answered beforehand. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Contributions ###
* The authors formalize test-time adaptation as a maximin threat model and contrast it with the "threat model for classic adversarial robustness" and the "adversarial semi-supervised minimax threat model".
* The paper studies test-time adaptation via DANN and its robustness in the maximin and minimax threat model. DANN has larger robustness against the proposed attacks in the maximin threat model than in minimax, providing evidence that maximin  is a setting that is beneficial for the defender. However, it remains unclear if this would also be the case against stronger attackers.

### Significance ###

Increasing robustness against adversarial robustness is a topic both relevant to basic research and also more applied research. Test-time adaptation is currently one of the most promising directions and this paper is thus very timely. Its formalization of a threat model for this domain can guide future research in this direction.
That being said, the main method investigated (DANN for test-time adaptation) seems less practical and significant since it requires large test batch sizes and also huge computational resources at inference time. 

### Originality ###

The paper has only limited novelty: while the formalization of test-time adaptation as maximin threat model is very thorough, it is also mostly straightforward. Also studying DANN as a test-time adaptation is an incremental idea once one assumes that inference is conducted on large batches. Nevertheless, I think papers like this that consolidate recent research directions into a coherent framework are valuable.

### Clarity ###

Generally, the introduction and the part outlining the threat model are written very clearly and I enjoyed the thorough formalization. A few details could be improved (even though this is a bit nitpicking):
* the introduction is very technical; it could be more high level and shorter and leaving the more technical parts to Section 2.
* Threat models could mention that white-box knowledge of the attacker is assumed
* it is not really clear why test inputs are assumed to be labeled in the threat models

The experimental part is less clear unfortunately:
* since DANN is a central method in the experiments, it should be briefly summarized to make the paper self-contained
* it is not really clear to me how specifically DANN is trained here (what is the objective?). Specifically, is the loss on the labeled data based on standard training or adversarial training? 
* generally, all training details (optimizer, learning rate, batch size, number of epochs etc.) are missing. the experiments cannot be reproduced in the current form.

### Quality ###

I have some serious doubts about the quality of the empirical evaluation.
As the authors state "While we are not able to prove the existence of a defender solution
that separates maximin and minimax threat models, we design and implement experiments to
provide evidence that such a strategy may actually exist." Thus, one of the main hypothesis of the paper (maximin being a strictly weaker threat model than minimax) depends on the design of experiments. My main concern here is that the attackers used are not actually strong:
* A transfer attack should clearly only be a baseline that should get surpassed by stronger adaptive attackers
* However, the fixed point attack seems to be weaker than the transfer attack. In particular its effectiveness in the homogeneous setting for CIFAR-10 reduces with the number of iterations k. This does not seem to be a reliable way of evaluating DANN's maximin robustness
* I would imagine a strong adaptive attack could be built by generating adversarial inputs  with the objective of minimizing the loss of DANN's domain label classifier. This would minimize the impact of test-time adaptation via DANN (because both domains would have already very similar representations). In any case, more effort should be spend on designing a strong attack against a DANN-based test-time adaptation.

I think it is also confusing to add evaluations in the inhomogeneous setting. Clearly, DANN gives strong results here because dealing with domain shift is what it was designed for. However, the paper is on adversarial robustness and confounding this with domain shift is misleading (my suspicion is it was mainly added because DANN clearly outperforms adversarial training on the source data in this setting; to proof me wrong, the authors should add a better motivation for this setting or remove it). 

### Recommendation ###

To summarize, I think the paper provides a strong formalization of a threat model for test-time adaptation. However, it does not introduce novel approaches, lacks clarity in experimental details, and its empirical results are less trustworthy given that the proposed attacks are rather weak. I thus recommend rejecting the paper in its current form. I encourage the authors to address the current shortcomings; the paper clearly has potential in general.

### Final Recommendation after Author Response ###
I would like the authors for the very active and productive rebuttal period.  Actually, the current version has significantly deviated from the initially submitted version. I have read most of the paper a second time to take all changes into account. The authors considerably improved the paper and have addressed some of my concerns. I increase my score to 5. The reason for not increasing the score to 6 or 7 is mainly that clarity of the paper is lacking. To give two examples:
 * Presumably because of the many changes during the rebuttal, the organization, structure and the quality of the manuscript is not always sufficient: to name one of several examples, the core Theorem 1 comes without any reference to its proof or any proof sketch in the main paper (there exists a proof in the appendix, finding of which required scrolling through the entire appendix).
 * As stated in my initial review, since DANN is a central method in the experiments, it should be briefly summarized to make the paper self-contained and also the specific training
I think the core issue with the submission is that there is simply too much content for one paper:
 * formalizing 3 threat models
 * a proof of separation of maximin and minimax
 * empirical evaluation  on two datasets (MNIST, CIFAR10), three attacks (transfer, two adaptive attacks), three defenses (DANN, AdvS, TTT), and two settings (homogenous, inhomogeneous)
Because of the page limit, a lot of details have been moved into the appendix, making the paper difficult to read. Moreover, even taking the appendix into account, details remain unclear such as how DANN was trained. Moreover, I do not find it convincing to move related work to the appendix; relating the current work to other work should be an integral part of the main paper.

My recommendation for the authors would be to strengthen the focus of the paper: I think the inhomogeneous setting does not contribute much and could be removed. Also I don't see much value in MNIST and the weak transfer attacks. Moreover, the parts on preliminaries and threat models is too long for a conference paper and could be shortened. 
I think if focus were improved and the main document became more self-contained, the quality of the work would be considerably improved. For the time being, I see the submission still marginally below the acceptance threshold.
review_title: Review of "Test-Time Adaptation and Adversarial Robustness"

Meta-review: This paper studies test-time adaptation in the context of adversarial robustness. The key idea is to use a maximin framework, which illustrates non-trivial robustness (under transfer attack) using domain adversarial neural network (DANN) to Linf-norm and unseen adversarial attacks. The approach is sound, well grounded, and quite logical. Results demonstrate the effectiveness.

However, there exists some limitations, for example, 1) The adaptive attack results are concerning. Comparing Table 1 and Table 3, with the adaptive attack (J-FPAM), the accuracy in the homogeneous setting is below that of adversarial training (Adv S) in Table 1, which somehow echoes my concern on not testing the transductive setting using strong attacks. It seems that adversarially trained models can better defend the adaptive attacks. 2) The paper says "This threat model enables us to study whether a large test set can benefit a defender for adversarial robustness", yet there is no any experiments in the main paper that correspond to this discussion. The appendix seems lacking this discussion either. 3) Due to the page limit, a lot of details have been moved into the appendix, making the paper difficult to read.

In the end, I think that this paper may not be ready for publication at ICLR, but the next version must be a strong paper if above limitations can be well addressed.
Decision: Reject