Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work empirically studies for deep networks the relationship between (1) model robustness and (2) the decision surface. A novel metric is proposed, the Populated Region Set (PRS) metric, essentially the number of regions in decision space which have at least one training sample. The authors claim the metric has a "strong relationship" to robustness, as measured by correlation, and present a number of experiments to support their claim.
main_review: ##########################################################################

Pros: 
 
* Characterizing decision boundaries is an interesting topic. I think there is a lot of interesting empirical work to be done in this area.

* Figure 2 is interesting: the PRS ratio clearly decays to zero for network B. 

##########################################################################

Cons: 

* Why no PGD attacks in the robustness studies?

* Relationship is pretty weak for CNN-6 in Figure 4. Any thoughts why?

* In Section 6, the authors use the word "intensive" to describe t-SNE plots. What do the authors mean by this? For instance the authors write: "In Figure 6, we can identify that the model with the low PRS ratio has more intensive feature representation for each class." Is there a quantitative measure for this?


* Regression slopes are only visualized, not reported quantitatively. I would also like the see significance tests.

##########################################################################

Questions during rebuttal period: 


* "Robustness score", the area under the robustness curves, is okay to report, but to the best of my knowledge it is non-standard in the robustness community. Epsilon sweeps are included in the Supplementary.

How is it actually possible that you can compute your proposed measure? Why does the "curse of dimensionality" not apply?

* What is the relationship of your work and those which analyze the linear regions of decision surfaces? E.g Montufar et al. (NeuIPS 2014)
https://arxiv.org/abs/1402.1869


##########################################################################

Post rebuttal period: I thank the authors for their thorough response. Having their response and the other reviews, I still harbor some doubts about the validity and usefulness of the proposed PRS metric and its empirical evaluation. I have decided to keep my score marginally below acceptance.
summary_of_the_review: Interesting empirical work but needs improvement and clarification.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to understand the robustness of DNNs from the perspective of decision regions. Towards that, the authors introduce a new metric, the so-called Populated Region Set (PRS) whose ratio is later used to investigate the robustness of a selection of DNNs empirically. Based on the respective empirical evidence, the paper claims that the lower PRS ratio (roughly #decision regions with at least one training data point/size of training data) leads to enhanced robustness, better representation of test instances in the populated regions, and learns the sparse feature representation. The empirical evidence is collected using the models CNN, ResNet-18, and VGG-16 over datasets MNIST, F-MNIST, and CIFAR-10 and under perturbations with various $\epsilon$ and untargeted/targeted attacks.
main_review: The paper is well-written and clear. The newly introduced geometry metric, the so-called Populated Region Set (PRS) ratio seems promising and intuitive. Although I am not knowledgeable of the related work on the interaction between decision boundaries and model's robustness, I  find the type of analysis/comparisons made (which I indicated in the summary) insightful. 

The conclusions that are drawn related to the decision regions and based on the empirical evidence are consistent with what one may indeed intuit. As useful as I find its intellectual contribution, I am not fully convinced on the informativeness of the PRS ratio when it comes to more challenging settings. For instance, what happens when the number of classes is large? In the experiments, the datasets in consideration have only 10 classes each, which is limiting. As an immediate thought, I would expect that the PRS ratio is generally high in cases where the label space is large. What do the authors think about it?


_Typos:_ "We verify that the the size ...", "Traini accuracy" in Figure 2 first column

summary_of_the_review: The analysis of the interaction between the robustness of ML models and the Populated Region Set (PRS) ratio seems promising in understanding the geometrical interpretation of robustness. Although having an extension to the current experiments would make the use of such a metric more convincing, I am slightly leaning towards acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new metric, the size of the populated region set (PRS), as an explanation for models with similar clean accuracies reaching very different accuracies under adversarial attacks. PRS is the set of decision regions that have training examples in them. After introducing and defining populated regions show that PRS in the penultimate layer is inversely correlated with robust accuracy. This is shown to hold across MNIST, F-MNIST and CIFAR10 and for 3 different model architectures (simple CNN, VGG and ResNet). They use two example networks with high and low PRS to show how PRS evolves during training and that the higher resistance to adversarial attacks might stem from the neurons in the last layer having similar decision boundaries in input space. 
Test samples that also fall into the PRS are shown to be more robust and models with small PRS have more test samples in them than models with large PRS. Additionally, it is shown that models with small PRS produce more sparse features. 
main_review: Strength:
The idea of populated regions is interesting, and should be further investigated. Especially, how test samples that fall into them are more robust could be a good direction to study in relation to designing more robust models.
The authors investigate PRS from many different directions and show how it is correlated to other important properties across 3 datasets and 3 model types.

Weaknesses:
The paper has big problems with clarity. Two networks, one with a large PRS (network A) and one with a small PRS (network B) are used to explore how the size of the PRS is related to other properties, but neither in the main paper nor in the appendix is it explained how they differ from each other. Is it just a different random seed or are they trained differently? In figure 2 it looks like network B reaches a high train accuracy significantly faster than network A.

In figure 4, 8 and 10 we see scatter plots of lots of models trained but again it is not clear how they are different. The appendix lists clean accuracy for models in table 2,3 and 4 but it only shows 4 different batch sizes and 5 different random seeds. The scatter plots clearly have more than these 20 combinations per dataset.
Additionally, I don't understand how the confidence intervals for the accuracies in the appendix tables are calculated, since the accuracies are for a fixed random seed so each run should lead to the same result.

The paper never says what norm is used for the adversarial attacks. From the epsilon values used I assume it is l_2. Do the results hold for l_inf?

I'm also not sure how useful the PRS metric is. The authors give the example that it is hard to choose between models with similar clean accuracies. But in that case I don't see the advantage of using the PRS ratio instead of directly using robust accuracy. Since there is no explanation on why some models have different PRS ratio it also doesn't give any hints at how one could deliberately train models to achieve a lower PRS ratio and therefore be more robust.

After comparing FGSM and iterative attacks (both targeted and untargeted) in figure 3 they use FGSM (untargeted) for the rest of the paper because the difference between the networks is the biggest. I don't think this is a good choice since the comparison clearly shows it is the weakest attack. It would a lot more interesting to see if the results hold for even stronger attacks like PGD or auto-attack[1].

There are also a lot of grammatical errors in the paper which make it hard to read and understand. There are too many to list here so I would encourage the authors to have someone proofread the draft again.

Other comments & questions:

Since PRS size is correlated with robustness do models trained adversarially have very small PRS compared to when we use standard training?

On page 1: "We further observe that the model with the small size of PRS has the relatively parallel parameters compared to the large one in the final layer." 
I could only understand what is meant here after reading the section at the end of the paper.

Figure 2: You should make clear that the left and middle plot are only showing the PRS ratio of layer 8. 

Figure 6: Shouldn't the ratio of test samples with zero and non-zero gradient sum up to 1 for both networks? I don't understand what the bars show exactly. Additionally, instead of two images for Network B ist would be more interesting two show how a successful attack for Network A changes the 
logits.

Figure 13: I'm not sure what "more intensive feature representation" means here. I find it hard to make any qualitative claims from the plots. For CIFAR-10 you could even argue that the high PRS ratio shows better separation of the classes while they all melt together in the middle for the low PRS ratio model.

[1]"Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks"
Francesco Croce, Matthias Hein
ICML 2020
summary_of_the_review: While the concept is interesting the paper doesn't provide enough details about how the models were trained or about the adversarial attacks. I'm not convinced the newly introduced metric will be useful without a deeper evaluation, especially explaining why similar models differ drastically in PRS size. The paper is hard to read and understand as there are a lot of grammatical errors throughout.

Given all these factors I don't think this paper should be accepted.
review_title: null

Meta-review: This work proposes a concept called Populated Region Set (PRS) as a measure of robustness of deep neural networks (DNNs). The paper provides a suit of empirical results to demonstrate the strong correlation between the PRS ratio and adversarial robustness of DNNs. The authors made great efforts on addressing reviewers' concern, which is greatly appreciated. However, the theory of the work is a bit thin, and it leaves a number of outstanding issues unaddressed. For example, it is not clear the practical advantage of calculating PRS over the direct measure of robust accuracy. What new and better computational procedure can be constructed based on PRS? We encourage the authors keep improving their work for future submission.
Decision: Reject