Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes to explain a phenomenon that the increasing robustness for adversarial examples might lead to performance degradation on natural examples. The authors analyzed it from the following aspects: 

1)	Adversarial robustness reduces the variance of output at most layers in terms of reducing the standard deviation (STD) of singular values associated with a layer of NN. The authors provide the experiment to show that the stronger robustness for adversarial examples leads to smaller STD of singular values of parameter of layers. 

2)	The reduced norm variance can cause the margins concentrated around zeros. Specifically， the authors provide the relevant lemma to show the relationship between margin and singular vectors. Moreover，the authors also conducted the experiment to show that stronger robustness over adversarial examples can lead to zero concentration of margin. The authors think a small margin might cause shrinking the hypothesis space which might cause low generalization.


3)	The authors have derived a bound of generalization which is related to Instance-space Margin，and minimum singular values. This proved that strong robustness on adversarial examples might reduce the generalization. 


In general, this paper seems technically sound. It is good that to some theoretic analysis can be derived in particular a bound of generalization can be given. Moreover, some experiments were made trying to verify the theory. Despite interesting, there are still some major concerns regarding the paper：

1.	The authors mentioned that “it is widely observed that such methods would lead to standard performance degradation, i.e., the degradation on natural examples.” I am afraid that this may not be always the case. In some other related work (see C1), adversarial training can perhaps achieve the performance lift on both the adversarial examples and natural examples (if a trade-off parameter can be well specified). Some clarification or further discussion may be necessary regarding this point.

2.	Only PGD was used as the adversarial perturbations in the experimental part of this paper. It would be more convincing if the authors could perform analysis on different adversarial training methods, e.g. FGSM and even the unified gradient perturbations developed in C2. There are also more adversarial attacks in the literature.

3.	The authors stated that the sample concentration around decision boundaries smoothness sudden changes, which was verified by the accuracy degradation. This is ok but it would be better to visualize or quantifying directly whether this can indeed make the boundary smoother. One possible way is to plot the confidence when moving the points near the decision boundary to check whether the confidence changes smoothly. 

4.	Finally, this paper seems to be written in a hurry. The paper may need substantial improvement on the English writing. There are still quite a few typos and grammar errors in the paper; this makes the paper less attractive though it contains some theoretic merits.

In summary, it is good that a theoretical bound can be derived from the paper, but this paper's quality may need more enhancement particularly on its writing and experimental parts. 

C1:  Virtual Adversarial Training: a Regularization Method for Supervised and Semi-Supervised, Learning" http://arxiv.org/abs/1704.03976

C2: A Unified Gradient Regularization Family for Adversarial Examples C. Lyu, K. Huang, and H. Liang, ICDM 2015.


============
I have carefully read the response as  well as the revised paper. To me, the response has addressed those of my major concerns. I an inclined to increase my rating and would suggest to weak accept this paper.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: ===== Summary ===== 
The paper presents new theory to develop understanding about why adversarially robust neural networks show lower test performance compared to their standard counterparts despite being more robust to perturbations in the data. The main hypothesis is that the degradation in performance in adversarially robust networks is due to many samples being concentrated around the decision boundary, which makes the network less confident about its decisions. The paper studies this hypothesis by deriving a bound on the generalization error based on the margin between the samples in the training set and the decision boundary. The paper then presents empirical demonstrations that aim to illustrate the theoretical findings. 

Contributions:
1. Derive a generalization bound on the performance of adversarially robust networks  that depends on the margin between training examples and the decision boundary.
2. Provide empirical evaluations that aim to illustrate the theoretical results.

===== Review =====
The problem that the paper addresses is very significant to the robust optimization field and the study of adversarial robustness in neural networks. Thus, I believe that the results could represent a significant contribution. However, due to the way that the information is presented, it is difficult to validate the correctness of the theory and the insights from the paper. Consequently, I consider that the paper should be rejected.

===== Detailed Comments =====
- First, and foremost, the paper should be proof-read for English grammar and writing style. In its current form, it is difficult to follow the main argument of many of the paragraphs. This is exceedingly important because the main subject of the paper is already difficult to digest as is.

- In the related work section, a lot of previous work is referenced without any context about what the contribution of each of those papers is. Each of these papers should be mentioned along with their corresponding contributions. Otherwise, it is difficult to frame the paper within the context of the current literature. Moreover, not providing context makes it difficult to determine which parts of the paper are original and which are the result from previous work. 

- The first item in Section 1.1 is difficult to follow because there are many gaps in logic that are left to the reader to fill in. It is reasonable to expect the reader to fill in some of the details, but since the sole purpose of this section is to build intuition about what is about to be presented in the paper, then each step in the explanation should follow as seamlessly as possible.

- The motivation for studying the margins between the training set and the decision boundary is not clear until Section 5.2.1 where it is mentioned that this is a widely used tool in learning theory. This should be presented earlier since not every reader will be completely familiar with learning theory. Moreover, it would also be useful to provide some intuition about how margins relate to the confidence of a classifier.

- After presenting the main result of the paper — Theorem 4.1 — very little intuition is provided about each of the terms in the bounds. It would greatly increase the clarity of the result if each term was explained intuitively, so that the readers can gain the main insight of the paper before reading the proofs. This would also help motivate better the empirical evaluations in the following section. 

- The plots in FIgure 3 and Figure 4 are very difficult to understand and unintuitive. For Figure 3, the main reason the plots are difficult to read is because different colors are used for different networks. For Figure 4, it is difficult to understand what is happening because the tallest curves are plotted on top of the shortest ones. Hence, the information from the other curves is mostly lost. This seems like a very minor comment, but this graphs are not very complicated, so they should be easy to understand; yet it takes several minutes to take in what is happening in the graph.

- For the proof of Lemma 4.1it is not clear how to get from Equation (7) to the main result of the lemma even after referencing Theorem 3 of Sokolic et. al. (2017) and Jia et. al. (2019). This could also be my lack of expertise on the topic; however, since the proof is already in the appendix, the proof should not be sparse in the amount of detail that it provides. 

===== References =====
Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel R. D. Rodrigues. Robust Large Margin Deep Neural Networks. IEEE Transactions on Signal Processing, 65(16):4265–4280, aug 2017. ISSN 1053-587X. doi: 10.1109/TSP.2017.2708039.

Kui Jia, Shuai Li, Yuxin Wen, Tongliang Liu, and Dacheng Tao. Orthogonal Deep Neural Networks. Technical report, 2019. URL http://arxiv.org/abs/1905.05929.


review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper focuses on analyzing the regularization of adversarial robustness (AR) on neural networks (NNs). They establish a generalization error (GE) bound characterizing the regularization of AR, and identify two quantities: margin distributions and singular values of NNs' weight matrices. With empirical studies, they show that AR is achieved by regularizing NNs towards less confident solutions and making feature space changes smoother uniformly in all directions, which prevents sudden change wrt perturbations but leads to performance degradation.

The paper is well written with theoretically motivated experiments and detailed analysis. I'd suggest accepting the paper.

With proposed GE bound connecting 'margin' with AR radius via 'singular values of weight matrices of NNs', they present 3 key results with empirical experiments on CIFAR10/100 and Tiny-ImageNet.
1) AR reduces the variance of outputs at most layers given perturbations.
2) Empirically examples are concentrates around decision boundaries.
3) The samples concentration around decision boundaries smooths sudden perturbation change, but also degrades model performance.
The paper only shed light on their conjecture that the performance degradation comes from the indistinguishable changes induced by adversarial noise and by inter-class difference. It'd nicer to further analyze on how to obtain AR without sacrificing performance on natural examples.



review_title: Official Blind Review #1

Meta-review: The paper investigates why adversarial training can sometimes degrade model performance on clean input examples. 

The reviewers agreed that the paper provides valuable insights into how adversarial training affects the distribution of activations. On the other hand, the reviewers raised concerns about the experimental setup as well as the clarity of the writing and felt that the presentation could be improved. 

Overall, I think this paper explores a very interesting direction and such papers are valuable to the community. It's a borderline paper currently but I think it could turn into a great paper with another round of revision. I encourage the authors to revise the draft and resubmit to a different venue. 

 
Decision: Reject