Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper first studies the relationship of the decision regions induced by the penultimate layer of a deep net to adversarial robustness and then relies on empirical findings to propose strengthening adversarial robustness without adversarial training. 
strength_and_weaknesses: Strength
- Some empirical findings are quite interesting.
- If a model can really defend without adversarial training, it is really great. However, it seems not the case for Cifar100 as in the supplementary material.

Weaknesses
- The concept of decision boundary and decision region as defined in the paper is confusing and misleading.  Because the decision region is normally relevant one class and represents the date examples classified to this class by a deep net, while the decision boundary represent the boundaries of the decision regions. I believe it should be better if the authors use the terminology like activation patterns. 
- Some behavior experiments lack of details and descriptions, hence hard to follow. For example, cosine similarity matrix of the final layer on Network A/B: cosine similarity of what and what (e.g., representations or weights). Also, Figure 5a is hard to interpret. 
- Mathematical notions used is not solid. For example, the one to define $PRS(X, f, l)$: why do we need $\forall V_l \in \{-1, 1\}^D$; the one to define $MR_{l,c}$: if $DR \in PRS(X_c, f, l)$ then $DR \in X_c$, hence $DR \cap X_c = DR$.
- It is great if $L_{ham}$ and $L_{MVR}$ can help to defense for all datasets. But it seems that for Cifar100, it still needs adversarial training. 
clarity,_quality,_novelty_and_reproducibility: The empirical findings of this paper are pretty interesting to understand more about adversarial examples and robustness. Based on  the empirical findings, the paper proposes a method without adversarial training and adversarial examples to improve the robustness. Unfortunately, this seems not be able to generalize for all datasets. Additionally, the terminologies and mathematical notions used in this paper need to be revised for avoiding misleading.
summary_of_the_review: The empirical findings of this paper are pretty interesting. However, the proposed approach seems not able to generalize to all datasets because for Cifar100, it still needs adversarial training.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors introduced the concept of Populated Region Set (PRS) to characterize the complexity of DNNs and build the correction between low PRS ratio and high robustness of models via several experiments.
strength_and_weaknesses: Strength: 1. The concept of the Populated Region Set (PRS) is novel and interesting.
2. The experiments are convincing.  

Weaknesses: 1. Lack of theoretical analysis to verify the claims.
2. Some definitions are confusing. For example, in Definition 2 (Decision Region (DR)), if the i-th layer has \textbf{only one neuron}, what are $D_\ell$ and $|DR_{V_\ell}|$? From the definition, it seems that $D_\ell =1$ and $|DR_{V_\ell}|=2$ in this case, but this seems strange. Please give one small example for each definition to make things clear. 
clarity,_quality,_novelty_and_reproducibility: Clarity: This overall paper is clearly written and well organized. I find it easy to follow. However, some definitions are confusing. 
Quality: This paper is technically sound.
Novelty: The novelty of this paper is high.
summary_of_the_review: In summary, the novelty of this paper is high, and the concept of the Populated Region Set (PRS) is novel and interesting. However, my main concern is that doesn't provide theoretical analysis, which makes the contribution limited. Also, some definitions are confusing at this point.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper looks at he notion of _Populated Region Set (PRS)_ which are decision regions with at least one training example in them. The central message of the paper is that networks with lower number of PRS have better robustness.
strength_and_weaknesses: ## Strength

* The concept of PRS is interesting and well motivated.
* The design of the regulariser is also well motivated. 
* The figures and illustrations are easy to understand and the mathematical formalisations are minimal and kept to the point. Overall, the authors have tried to present a coherent story.

## Weaknesses
* The message of the paper is not borne by the experimental results. In particular, AA robust accuracy is always zero.
* It seems that certain networks have not been evaluated under AA and all AA results are relegated to the appendix.
clarity,_quality,_novelty_and_reproducibility: ## Clarity
* The illustration and structure of the paper aid the clarity of the paper.
* Section 1 contains terms that are not yet defined like PRS ratio, PRS inclusion ratio of test examples etc.
* However, the writing could be improved. It is not a major drawback but probably something to look at for future versions.

## Quality and Novelty
*  The authors have made a clear hypothesis and tried to validate with experiments. I find the way the flowchart of the paper quite good. The problem is whether the experiments bear the results and it does not.

## Reproducibility
* I do not have any major concerns.
summary_of_the_review: The main drawbacks of the paper  are the following

* The first red flag should have been that for $\epsilon=0.1$, a CNN network trained via standard training has 80% robust accuracy in Figure 3. This is the main result the authors use to justify that low PRS ratio increases robustness. However, a quick look at Appendix F shows that both Model A and Model B have less than 1% robust accuracy. This shows that it is not that low PRS ratio models are more robust just that, it is harder to find adversarial examples possibly due to ill-conditioning.

*Again Figure 6 shows a similar kind of behaviour as Figure and does not include measurements with AutoAttack.

*Finally table 2 also doesn't contain AA. In fact, even the PGD attack is fairly weak containing only 20 steps. This can also call into question the PRS regularisers are indeed more robust than ST and whether PRS regulariser+AT is more robust than AT.

I did find Appendix G quite interesting, as it does suggest a connection between PRS ratio and robustness but this is the only experiment that is not confounded by the possibility of weak attack.

Given the above, I am inclined to support rejection unless the authors convince me otherwise.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper prooses the novel concept of populated region set, and make connection between the proposed PRS ratio and the adversarial robustness of the model. The paper further propose PRS regualrization that can improve adversarial robustness without adversarial training.
strength_and_weaknesses: ## Strength
1. This paper provides novel concept on PRS and PRS ratio, and make connection between the proposed concept and adversarial robustness
2. Thorough experiments are conducted to emperically support the claim on the relationship between PRS and robustness
3. The proposed regualrization indicates the potential benefit PRS can bring to understanding and preventing adversarial attack

## Weakness
Some technical details of the proposed method are not clearly explained. For example:
1. How exactly is PRS computed given a DNN model? Is the computation method scalable to a larger/deeper model or a larger dataset?
2. How is the gradient of the Hamming distance regularization is Eq.(2) computed in the optimization?
3. As the model is being updated with the proposed regularization, will MRV change in the process? Does MRV need to be recomputed during the training process, and if so what would be the cost of the computation?
clarity,_quality,_novelty_and_reproducibility: The paper provides novel method and is written in good quality. Some technical details are not clearly explained.
summary_of_the_review: Generally the paper provides novel and interesting insight on understanding the adversarial robustness of DNN models. However the lack of technical details make it hard to assess the correctness and practicalness of the proposed method, Thus I would recommend a weak rejection for now.

## Post rebuttal
I believe the response from the author clearifies my doubts. Given the author response and revision I'm increasing my score.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the relationship of the decision regions in the penultimate layer deep NNs and deduces an approach to increase the robustness. The introduced concept uses the Populated Region Set (PRS) to characterize the complexity of DNNs to improve adversarial robustness against PGD. 
While the paper addresses in important topic and proposes an interesting analysis, the remaining criticism mostly focuses on the experimental evaluation - showing relatively strong improvements w.r.t. PGD but not against AA. As a result, it remains unclear whether the decision regions are sufficiently explored.  
justification_for_why_not_higher_score: The proposed method does not result in a sufficient improvement in robustness w.r.t. diverse attacks. Adversarial training is still required, e.g. for Cifar 100.
Decision: Reject