Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper introduces two properties of adversarial training observed from abundant empirical results. Based on the discoveries, the authors propose plausible explanations as well as new methods to gain higher adversarial robustness. The two properties are as follows

1. The batch normalization may negatively affect the adversarial robustness, where a training batch consists of a mixture of clean and adversarial examples. The authors observe that the parameters for batch normalization (BN) may be quite different between batches of clean and of adversarial examples, and conjecture the reason as that these two sets of examples are from two different domains. Therefore, the authors propose a few methods to boost the adversarial robustness: using different BN stats for clean and adversarial examples (but keeping the other parameters shared, which may seem useful only for fundamental study due to the requirement of clean/adversarial label for each example), using batch-unrelated normalization, and changing stat estimation for BN to reduce the difference between training and inference steps.

2. The deep networks for general image classification may be too shallow for adversarial training. Since the mixture of clean and adversarial examples may form a two domain distribution that is challenging to model, typical neural networks (even for ResNet-152 that has high depth for general image classification) may have too low capacity. The authors show that increasing the depth of neural network (up to and possibly beyond ResNet-638) results in even higher accuracy.

To my best understanding, the results and analysis of this paper are valid, and the proposed methods have shown gains in abundant experiments. Due to the significance of adversarial training and the new discoveries of this paper, I think the contributions are sufficient and would lean towards the paper being published / weak accept. However, below are my questions and concerns that I would like the authors to address.

1. For this paper, the adversarial examples are generated from a particular class of attacker, namely Projected Gradient Descent. I am curious how the conclusion could generalize and help the robustness if we have more than one adversarial attackers.

2. The paper focuses on adversarial robustness and seems to deprioritize clean image accuracy, which could seem to limit the scope for application purposes. Frankly, I agree that a fundamental understanding of adversarial robustness would be significant, and the authors discuss the problem of relatively lower clean image accuracy in Section 4.4. However, I might feel that clean image accuracy should be as significant for practical purposes, and it would be great if we can balance the trade-off between clean image accuracy and the adversarial robustness. The authors should feel free to correct me if I do not understand correctly.

3. There seems to be an interesting observation in Fig 1 for which I am curious. The accuracy for PGD-2000 in Fig 1 does not go monotonically with the ratio of clean images — the lowest accuracy is at 60 percent of clean images, which seems not to fully align with the argument that removing clean images will help robustness. Personally it would be great if the authors could share their insights of possible reasons.

4. For the second discovery that deeper networks help adversarial robustness, the red line (for adversarial robustness) in Fig 7 seems not converged yet at ResNet-638. If the computation resources allow, I am curious on the depth of ResNet at which the red line becomes flat, and this could be useful for headroom analysis on how good accuracy we can reach.

5. For the second discovery that deeper networks help adversarial robustness (Section 5), it seems Madry et al 2018 (Towards deep learning models resistant to adversarial attacks) also discusses model capacity vs the adversarial robustness. The mentioned paper does not seem to use deeper structure but uses other ways to increase capacity. The mentioned paper has been referred to in other sections of this work, however, it may be good to contrast in Section 5 on the conceptual novelty in this paper.

6. Typos: The title of Table 1, “MBN_{clean}/MBN_{clean}” would be “MBN_{clean}/MBN_{adv}”. There are some “, ,” (double commas) in the appendices. Comma after ‘(’ in appendix B3. Overall this paper is well written and easy to follow.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper studies adversarial training "at scale", i.e., on the ImageNet dataset. The paper makes two main contributions in this context:
- An in-depth investigation of the effect batch normalization (BN) has on adversarial robustness when the network is trained adversarially.
- Training increasingly deeper residual networks (up to 600 layers) and demonstrating that adversarial robustness still increases in this regime (unlike standard accuracy).

Overall I find the findings presented in the paper interesting and recommend accepting the paper. Experimenting with adversarial training on ImageNet is still hard for many academic groups due to the high computational cost. Hence the results of the paper may be useful for the wider robustness community. To achieve this goal, I strongly encourage the authors to release their models in a format that is easy to build on and experiment with for other researchers (e.g., PyTorch model checkpoints). Moreover, I find it interesting that very deep models on ImageNet can achieve increased adversarial robustness. To the best of my knowledge, these are the best robustness numbers published on ImageNet.

Further comments and questions:

- It would be good to know if BN also affects adversarial robustness on CIFAR-10 or other datasets.

- What happens when the width of the network is increased? Does this also help adversarial robustness?

- Section 4.1 states that "Adversarial training can be dated back to (Goodfellow et al., 2015), [...]". While the specific form of adversarial training for adversarial robustness in CNNs is indeed recent, it may be helpful for readers to provide additional context, e.g., min-max formulations have a long history in robust optimization and statistics.

- Section 4.4 states "Interestingly, we find that the accuracy on clean images can be significantly boosted from 62.3% to 73%.". It would be good to add context and state what accuracy the network achieves with standard training.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper reveals some interesting properties of neural networks when trained adversarially at ImageNet scale. The total cost of the experiments is quite impressive, therefore the results are valuable references. With extensive experiments, the authors reveals two intriguing properties of neural networks when trained adversarially, and devotes most of the paper to studying the first one, i.e., why networks with Batch Normalization cannot achieve high robustness when both clean and adversarial images are used for training. I am not fully convinced by this point, but I do think the second point is very interesting. Different from previous work arguing that more data is needed for improving adversarially robust generalization, this paper shows that adversarial robustness can be improved consistently just by making ResNet deeper. 

I tend to accept this paper for the valuable results and the discovery of the positive correlation between network capacity and adversarial robustness, but I am not fully convinced by the explanations for the problems with BN. I hope the authors could address my concerns before I can be more confident about my decision.

To me, it seems the correct way to do adversarial training is to only use adversarial samples, since we are trying to minimize the maximum risk inside a norm ball around the clean sample (Madry et al. 2018). All the experiments with clean images in the objective are consistently worse than training only on adversarial samples under the same setting in this paper. It therefore becomes not that important to study the effect of Batch Normalization on training with both clean and adversarial images.

Also, I am a little bit unconvinced that the running mean and variance of BN is the cause of bad performance for the mixed training. First, I want to know the standard and adversarial accuracies of the network with GN in the "100% adv + 0% clean" setting. It seems missing in the paper. Despite being much better than BN in the "100% adv + 100% clean" setting, it is not sure whether it is caused by the improvement in some other property (e.g., capacity) from GN. If the robust accuracy with "100% adv + 0% clean" is also much higher with GN than BN (seems unlikely though), then replacing GN with BN does not solve the problem and it is still the objective to blame.

I am also not fully convinced by the experiments with separated BN parameters for clean and adversarial samples. By doing so, the network is actually trained to approximate its behavior in the "100% adv + 0% clean" setting. Since the adversary is maximizing the loss, the gradient (of the conv layers) from adversarial samples will dominate the total gradient in the "MBN 100% adv + 100% clean" setting, and the "MBN_{adv}" network will be trained similarly as the network trained in the "100% adv + 0% clean" setting. This explains why the adversarial accuracy can be very close to "100% adv + 0% clean" but still lower.  Comparing results (under different ratios of clean samples) for networks without BN is very important and much more effective at explaining the phenomenon than trying to make BN work.

It would also be great if the authors could provide some curve showing the tendency of the running variance of BN. Sec 4.3 does make lots of sense from a practical aspect, i.e., fixing mean and var for training in the last 10 epochs could improve the results using same number of epochs, but what if we just train more epochs? Will the variance converge in just tens of epochs?

Finally, though it seems that deeper networks are more robust,  the robustness might be a misconception caused by gradient vanishing. Could the authors provide the average gradient norms on the correctly-classified images (remaining correct after attack) in the first step of PGD attacks for the models in Figure 7? If deeper networks indeed have much smaller gradient norm, could the authors try scaling the loss by some factor to make the attacks stronger?


===========================================================
I am not fully convinced by your explanations. Could you give the results for the gradients, or use the loss function from CW attack + PGD to report the robustness of the deeper models? I am not sure which type of residual connection you are using, but if you are using the "original" version as mentioned in this paper [1], then it does not necessarily avoid gradient vanishing, since if ReLU is deactivated, the gradient will be brought to zero. Another possible cause for the possible misconception of robustness might be caused by the saturated cross entropy loss, which could also give 0 gradients but could be verified by switching to the margin loss as in the CW attack. I have encountered such cases when I achieved non-zero robust accuracy (evaluated with PGD on cross entropy loss, and the robust accuracy is similar to your improvement from ResNet152 to ResNet638) on a naturally trained network as I increased the number of layers, but when I switched to CW loss (or margin loss), the robust accuracy goes to zero. Even for resnet, when you multiply the number of layers by ~4 as done in the paper, such phenomenon will actually happen.

[1] He, Kaiming, et al. "Identity mappings in deep residual networks." European conference on computer vision. Springer, Cham, 2016.

=====================================================================
Raising my score. Found a recent good paper based on the findings of this paper. 
review_title: Official Blind Review #1

Meta-review: This paper studies the properties of adversarial training in the large scale setting. The reviewers found the properties identified by the paper to be of interest to the ICLR community - in particular the robustness community. We encourage the authors to release their models to help jumpstart future work building on this study.
Decision: Accept (Poster)