Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper provided a new perspective on adversarial defense, i.e., optimizing adversarial defenses at inference. 

The authors proposed dynamic defenses that adapt the model and input during testing to defend against adversarial attacks. 
main_review: Pros:

1 The new perspective on defense at inference time indeed deserves some explorations. This perspective is interesting. 

2 comprehensive experiments on CIFAR-10/100 and ImageNet are reported; the experimental results look good. 

Cons:

1 Auto-attack is meant for evaluating a single & static model, which may not be applicable to evaluating dynamic defense. Therefore, authors may consider the adaptive attacks for the robustness evaluations. 

2 Inference-time defense may hurdle the speed of the model prediction. 

Some other comments:

The authors implicitly assume adversarial data comes from a different distribution from that of natural data. Could authors elaborate more about it?

* I may adjust my scores based on the authors' responses. 


#### Post rebuttal ####
The authors did not resolve my concerns. Please find my reasons as follows. 

Auto-attack is meant for evaluating a single & static model, which may not be applicable to evaluating dynamic defense. Therefore, using AA attacks for evaluating the dynamic defense is a factual error. 
Although there are some discussions of adaptive attacks on "dent", those adaptive attacks strategy still cannot convince me enough. 


limitations_and_societal_impact: see above.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes to defend deep networks by dynamically modifying the model and the input images. The model and the input are modified by optimizing the Shannon entropy with respect to the model and the input. 
main_review: Major Issues:
1) First of all, modifying the input [1,2] and dynamically modifying the model [3] for defensive purpose has been explored in previous works. The authors should cite these works and be more careful and precise with the claim of :
> Line 53 We demonstrate the **first** defense to optimize the model and input during testing for improved robustness.

2) The key design of this work is the proposal of minimizing the Shannon entropy, which confuses me a lot. How come can optimizing the current prediction help find the correct prediction? For an adversarial example that is wrongly classified, wouldn't optimizing the wrong prediction worsen the prediction. To be more clear, from my experience, for an example with label A, if adversarial attacks can modify the example to make the model predict B, and the dent method optimizes the entropy for the adversarial example, it is likely to be in favor of B rather than A. After briefly reading the provided code of cifar10a.py, I think I find a possible reason. In the line 28 of cifar10a.py, the adversarial examples are generated on a dynamic model rather than a static one. The dynamic model may hamper the attack to find a proper adversarial example. Thus, I have two questions:
* What is the accuracy of the attack in the line 28 and does the attack really succeed? This means what is the accuracy if adversarial examples are generated with dynamic model but predict by a static model? If the adversarial example generated on the dynamic model can not fool the static model, then this is an evidence that the attack in the code is not properly implemented.
* What will happen if we generate adversarial examples on a static model then predict these adversarial examples on dynamic one? I suspect that, in this setting, the dent method will be broken. BTW, to show the effectiveness, [1] also tested on both static and dynamic/randomized model. The author should follow the same routine.

Without clarifying these facts, it is very likely that the huge improvement is simply an illusion of incorrect attacks implementation. 
Also notice that, the author claims:
> Line154 : Our defense does not rely on shattered gradients

This claim is questionable. It seems to me that the dynamic process is not differentiable and may hamper a proper gradient computation. The author should adopt the BPDA in [4] to show it does not suffer from the obfuscated gradient. Also, the author claims that their method is deterministic and thus does not suffer from stochastic gradient problem (Line155). This is somewhat confusing. If the method is **deterministic**, then this method should not be claimed as **dynamic**. Please explain.

3) The evaluations in this work is questionable on several aspects. For robust models, the author omit two of the most important works: TRADES[5] and Adversarial Training[6]. Although the models tested in this paper are occasionally built up on these two methods, the author still should provide evaluations on these two important baselines for better comparison. Also, on Table 2, why only provide the detailed results on only two methods? Clearly the authors have the information for other models since they show it on Table1. The presented two methods Wong and Ding are either not the most representative nor have the best performance. 

4) As claimed by the author, the methods require batch predictions. What causes this deficit? To me, the modification of the model should not rely on batch information so much since it is stable after training. On Table 8, the author says:
> With static batch statistics (×), small batch sizes are better. With dynamic √batch statistics ( ), small batch sizes are worse.

What causes this phenomena?

Minor Issues:
1) For Line 29~32, I understand that adversarial robustness is related to domain adaptation. But how does this fact relate to the method in this work?
2) For Line 36, what does "adaptive attacks that are aware of its updates" refer to
3) For Line 135~137, please be more clear? Have dent solve the batch-wise deficiency?
4) For Line 158, what does "stale gradient" mean?
5) I recommend the author to write an algorithm to show their method more clear.
6) For Line 224, Table what?


The second point of the major issues is my major concern about this work. I will change my score if the author can answer my questions and provide solid proofs for their effectiveness.


[1] Mitigating adversarial effects through randomization.    ICLR 2018

[2] Randomization matters How to defend against strong adversarial attacks    ICML 2020

[3] Adversarial Robustness via Runtime Masking and Cleansing    ICML 2020

[4] Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples    ICML 2018

[5] Theoretically Principled Trade-off between Robustness and Accuracy    ICML 2019

[6] Towards Deep Learning Models Resistant to Adversarial Attacks
limitations_and_societal_impact: This work does not have obvious negative societal impact. I have listed the limitations in the main review part.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the problem of dynamic defenses agaisnt the potential adversarial attacks during the test time. The work introduces a new method derived from "defensive entroy minimization" approach that "fights back" and counter-optimize against attacks. In general, the proposed method updates the parameters of DNN slightly for every round of inferences, so as to balance the accuracy  and fuzziness of models. The basic idea was clear while experiment results demonstrated the potentials of proposed methods. During the experiments, authors aimed at establishing three major claims (1) Dynamic Defense of Adversarial Training, (2) Dynamic Defense of Nominal Training, and (3) Adaptive Attacks on Dent Updates, with sub-claims in details. Experiment results support these claims in part. Ablation studies also demonstrate the indispensibility of and contributed made made by every component.  
main_review: Typo: 224 "Table ?? in the supplement..."
The work was comprehensive and intuitive with details in experiments and comparisons. 
Supplementary materials provided necessary information to follow this work.  
limitations_and_societal_impact: My major concern on this paper is that how dent ensures the parameter update \Delta would not hurt the performance of neural networks. As was learned in some previous (e.g., lottery ticket theory), the modification to the weights might not affect the way that DNN behave. Or we can say that updating weights might not be the most efficient way to achieve the goal. Just be curious, have you explored the potentials of feature maps perturbation, where the behaviors of the DNN could be slightly changed with accuracy of prediction preserved?
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposed a new defense, namely, defensive entropy minimization (dent), to improve adversarially robustness. The proposed method alters testing but not the training phase, which is different from most existing defenses that are static and do not adapt during testing. The improvements of model robustness evaluated by different strong attacks and datasets have confirmed its effectiveness. 
main_review: Pros:
1. This paper is clearly organized and easy to follow.
2. The proposed method (dent) is interesting and reasonable.
3. Extensive experimental results on multiple datasets are provided and the effectiveness of the proposed method is confirmed.

Cons:
1. The performance of the proposed method (dent) is not stable as the static defense (e.g., adversarial training). When the batch size of the test data becomes smaller, the defense effect will be weakened. Since the adaptation is based on the test data, dent may not be able to handle the test scenarios (e.g., using small test batch size or hard dataset like ImageNet) which are not friendly to the estimation for the sample-wise parameters. Could the authors give more discussion or analysis and make some improvements to the method?
2. The relationship between this method and domain adaptation is not very clear. Could the authors give more explanation about it and how dent improves robust accuracy while maintaining natural accuracy?
3. Lack of further analysis or discussion on the performance using different datasets. More experimental results or discussion about dent's performance on the large dataset (i.e., ImageNet) are encouraged.
 
Minor Concerns/Questions:
1. It seems that the correct reference is missing in line 224. (The "Table ??" in line 224)
limitations_and_societal_impact: As aforementioned, could the authors further lessen the method's dependence on the batch size of test data? (or give more discussion?)
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes a novel test-time dynamic defense method, which adapts the model and input during testing by defensive entropy minimization (named dent). Given an input, the proposed method minimizes the entropy of the model prediction to adapt the input and updates parameters in batch normalization layers. Empirical results show that the proposed method improves the robustness of adversarially/nominally-trained models against various attacks including AutoAttack. 
main_review: Pros:
- The idea is novel and very interesting.
- The paper is easy to follow.
- Experimental results show strong robustness against existing attacks including AutoAttack.
- The paper considers adaptive attacks.

Cons:
- There is a big concern about vulnerability against a kind of data poisoning attacks at test time.

-------------
Overall, this work is interesting and impressive. The idea is novel and the strong robustness against AutoAttack provides useful information for researchers in this field.

However, there is a big concern about vulnerability against a class of adaptive attacks. The proposed method sequentially updates model parameters, and it is concerning that this type of defense methods could be vulnerable against a kind of data poisoning attacks at test time. Before attacks, between attack steps, and a part of each batch, attackers can feed malicious inputs (e.g., random noises) to make model parameters meaningless. The attackers can also craft inputs to broke model parameters.

Although this type of attacks is not used in existing attacks, it is because the parameters of the target model is typically fixed, and these attacks are meaningless against the fixed target models. In white box settings, while the purpose of attackers is to find an adversarial example with small perturbation, attackers can feed any inputs to the model for their purpose. Is the proposed defense robust against these attacks? It is important to present discussions and experiments for this type of attacks to show the robustness of the proposed method in white-box settings. 


I would be happy to raise my score in case these points are properly addressed.

Other comments:

- The word “step” is sometimes confusing. There are two kinds of steps in the paper: attack steps and dent steps. For example, Figure 1 shows three attack steps for static/dynamic defenses, while “Step” in Table 3 is dent steps (i.e., steps to update $\Delta$ and $\Sigma$). However, it seems no sufficient explanation about dent steps in the paper. 

- The drawback of dent+ is not clear. In line 148, the authors wrote “Dent adapts on batches rather than samples. Batch-wise adaptation stabilizes optimization”, but Table 1 shows that sample-wise Dent+ achieves better performance. I feel the statement lacks consistency. Readers may think that dent is not necessary as dent+ is better than dent.

- The difference in the optimizer for Dent and Dent+ is a bit weird. Why do the authors adopt AdaMod with large learning rate for Dent+, and how do you decide the optimizer? 

limitations_and_societal_impact: The authors have addressed limitations of the proposed method in Section 5.

The authors have not described negative societal impacts since they propose a defense method.
review_title: null

Meta-review: The paper proposed to dynamically modify the model and the input images during the test time (named "defensive entropy minimization"). While the idea is interesting, the true novelty needs to be carefully clarified given many missing related papers, because the test-time defense is in fact not new but one of the two main-stream approaches to adversarial robustness (the other is adversarial training, i.e., training-time defense by simulating the presumed attack without seeing the real attack). Besides the first 3 papers given by Reviewer XQWL, there is another paper closely related to test-time defense: Countering Adversarial Images using Input Transformations, ICLR 2018 (https://arxiv.org/abs/1711.00117). The paper cannot be accepted this time, and hope the authors can find the information helpful for their submission next time!
Decision: Reject