Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes to evaluate the robustness of the neural networks by extrapolating to the unseen classes. However, the authors only include evaluation for non-robust trained models, without considering the robust trained model, such as Madry et al. [1]. The conclusion is not convincing that the authors studied the robustness using only non-robust models, because it is well known that the accuracy for attacking non-robust model can be 100% (for CIFAR-10). It is useful if the authors study whether their method can be used to measure the robustness of the robust trained models.

1. The paper only evaluate robust accuracy on models without robust training.

2. The evidence in the paper does not support their point. The paper shows that there's link between adversarial robustness and the generalization ability of neural network to extrapolate to unseen classes. But could not support the claim in the abstract, that 

"The main idea lies in the fact that some features are present on unknown classes and that unknown classes can be defined as a combination of previous learned features without representation bias (a bias towards representation that maps only current set of input-outputs and their boundary)"

The extrapolation score does not prove this.

3. The attack accuracy in Table 1 is not convincing. In [1], the PGD for epsilon=0.3 is 100% for non-robust models and around 55% for robust models, the number in the Table matches non of these.

4. The writing of this paper is not clear and has gramma issues. For example, Table 1 and 2 miss information in Caption.

5. The author should cite "Adversarial Examples Are a Natural Consequence of Test Error in Noise", which also studied the generalization and adversarial robustness.

6. This evaluation measurement is not practical, it costs more computation than one evaluates a model directly using attacks. The measurement is also fragile under adversarial attacks, that one can feed in adversarial attacks to fool the score metrics, which is not convincing.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: Summary:
This paper aims at revealing the relationship between the quality of deep representations and the attack susceptibility of deep classification models. To this end, they propose the zero-shot test to investigate the "quality" of learned representations for unknown classes. Specifically, they leverage two kinds of quality metrics on data of unknown classes. The first one is based on clustering named Davies-Bouldin Index which measures the compactness of intra-cluster. The second one is based on the difference of soft-label histogram distributions with/without unknown classes during training, which may describe the generalization for unknown classes or bias towards known classes of learned features. Finally, with these two metrics, they rank the quality of different models and compare such ranking results with the attack robustness obtained by different attack techniques on CIFAR-10 dataset.

+Strengths:
1. This paper shows satisfactory related works about recent advances in adversarial attacks and defenses. 

-Weaknesses:
1. The writing of this paper contains many mistakes, especially for the issue of using singular and plural. I strongly recommend the authors to polish the paper carefully. Take the first two pages as an example, “results suggests” in Abstract, “Networks’s” in Introduction, “a DNNs”, “perturbations causes” in Sec1.1, “methods… which uses” in Sec.1.2, etc.
2. The idea of representations quality on unknown classes determining attack robustness is not reasonable. Since the adversarial attack is defined on the known classes, the reasons for that are in two aspects. The first is the training dataset. If the collected training data cannot represent/fulfil the whole continuous distribution/manifold of the categories or even biased, the models are of course easily fooled by unseen modes during test. The second is the mapping for classification is from high to low, which is naturally many to one. In a word, I think the quality of models on unknown classes is not directly related to the problem of adversarial attack.
3. This paper only conducts experiments on CIFAR-10 dataset, which is not convincing enough. It would be better to evaluate their method on more challenging benchmarks and also give more validation about their idea. E.g. on ImageNet, if enough number of categories has been seen for models, whether they would become more robust to adversarial attack. Evaluating features of more layers rather than the softmax outputs is also needed. 
4. What is amalgam proportion? Please explain it in detail or give a reference paper. Otherwise the readers cannot understand the motivation of the second metric in Sec3.2. Besides, Fig.2 contains little information and few captions for readers to understand their method. 
5. In Tab.1 and 2, the meanings of Attack Accuracy and Average Amount of Perturbations (typo in caption for Tab.2) are not introduced. Tab.5 shows 7 methods but its caption says “five”. Fig.4 is also confused, for AM, which histogram is with unknown classes and which one is without?
6. The title says "representation quality explain adversarial attacks". After reading this paper, I haven’t found the mechanism leading to the adversarial attacks of DNNs.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: === A. Summary ===

This paper proposes to evaluate the representation (here, the output of 9 probabilities for 9 classes at the last softmax output layer) of an image taken from an unseen category by two metrics:
- The first metric (DBI) essentially measures how compact the learned manifold is (i.e. how nearby are the softmax probability vectors of images from an unseen category). 
- The second metric basically measures how close a 9-D probability output vector ("soft labels" in the paper) of a classifier trained on the 9 classes is to a "ground-truth" which is the same 9-D vector but obtained when the classifier is trained on the full 10 CIFAR-10 classes.  

The authors computed these "representation measures" for a set of 7 networks (6 convnets + CapsuleNet) trained on CIFAR-10.
Then, a set of white-box adversarial attacks were performed on these 7 networks and the results (both how accurate the classifier is, and the amount of L2 perturbations required to change a label) were used as a "robustness measures".
The authors then claim that the ranking of the 7 classifiers based on the representation measures match the rankings derived from the robustness measures.

The paper attempts to find the correlation between the representation quality of a classifier with its adversarial robustness. This general direction is important and worth pursuing!
However, the claimed correlation is very weakly supported by the evidence in the paper.

=== B. Decision ===

Reject.

The aim of connecting the representation quality with adversarial robustness is interesting!
However, this paper has several major issues:

0. The paper uses the wrong citation format of ICLR (changing from LastName et al. 2019 --> (12) ). The paper maybe 8.5 to 9 pages long if using the original format. If this format violation was accepted, it would be unfair to other submissions and be a bad example this citation violation would be OK.
1. The paper has many claims (e.g. "DBI metric matches extremely well") that were not supported by evidence or clarified.
2. The paper claims to study classifiers' "representation quality". However, all the experiments were conducted on only the 10-class CIFAR-10 dataset and the "representations" are taken at the softmax layer as opposed to some mid-CNN feature layer (e.g. the popular fc7 in AlexNet trained on the 1000-class ImageNet). 
3. The key Definition in Sec. 2 that the paper hinges on has 0 references and is a debatable definition.

=== C. Suggestions for Improvement ===

- I'd advise the authors to perform careful literature review to identify where the gap to fill is before conducting the research. There is a ton of work that has been done in the intersection of zero-shot, adversarial examples, deep features, and classifier caliberation. 

For example, [1] has looked at how out-of-distribution samples can be represented by the deep features of a classifier:
[1] Bendale, A., & Boult, T. E. (2016). Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1563-1572).
- This work [2] has shown adding a "center loss" to force the deep features to be closer to the feature centroid (i.e. related to your DBI measure) helps improve the adversarial robustness. Indeed, the visualization in Fig. 3 is very interesting and is consistent with the result in [2].
[2] Agarwal, C., Nguyen, A., & Schonfeld, D. (2019, September). Improving Robustness to Adversarial Examples by Encouraging Discriminative Features. In 2019 IEEE International Conference on Image Processing (ICIP) (pp. 3801-3505). IEEE.

- The word "representation quality" here is confusing when it refers to the soft labels, which is more relevant to *caliberation* than quality of deep features as the title implies. And the paper misses this body of work in the related work. See here: https://nicholas.carlini.com/writing/2019/all-adversarial-example-papers.html
- Try to avoid "extremely" or "clearly" in claims (and in general scientific writing) because they tend to be overclaims and it is hard for readers to interpret those intensifiers. 
- I don't see how the rankings by DBI or AM are similar to the original rankings by adversarial measures at all (except for CapsNet). At least, reporting Pearson correlation (or any statistical similarity measure) would be more convincing.
- Section 2 is highly debatable and has no references for the "Definition".
- I'd perform this work on ImageNet, at least to follow the intuition in Sec. 2. With only 10 CIFAR-10 classes, it is unrealistic how to semantically represent a "frog" using the other 9 classes (truck, airplane, etc).

review_title: Official Blind Review #1

Meta-review: The reviewers found the aim of the paper interesting (to connect representation quality with adversarial examples). However, the reviewers consistently pointed out writing issues, such as inaccurate or unsubstantiated claims, which are not appropriate for a scientific venue. The reviewers also found the experiments, which are on simple datasets, unconvincing.
Decision: Reject