Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper proposes an alternative loss function for classification models that they claim leads to improved adversarial robustness even under strong adaptive attacks. It also attempts to analyze how the softmax cross-entropy loss discourages robustness by considering the problem in terms of local density in the pre-logit feature space.

Although as an emergency reviewer I haven't had time for a thorough verification, the overall idea seems sound, as do their theoretical and experimental results. I appreciate the careful analysis of the sample densities induced by each method; the N/L^2 vs. N/L result is especially nice. They also seem to follow best practices for evaluating adversarial defenses. Overall I do feel like the research topic of developing alternate loss functions and identifying pathologies with current popular loss functions is important and maybe insufficiently studied / publicized, so I think publishing this paper would be helpful for the field.

A few questions:
(1) If the MMC loss makes your final models so much more robust to small perturbations, why is there still such a large clean accuracy drop when combining your method with adversarial training in Figure 3a? I would have hoped that the tradeoff would have been less extreme. If you started adversarial training midway through the training process, or used a smaller perturbation size, would the tradeoff still be as large as with SCE models?
(2) It makes sense that the optimal method of choosing MM centers would be to place them at the vertices of simplexes when #dims = #classes, but if there was some other way of avoiding the degradation problem from Wen et al. 2016, would it ever make sense (especially when #dims < #classes) to allow some automatic slackening of the MM centers (i.e. allow them to move from their original positions, but at a heavy cost), in order to permit classes that are similar (e.g. different breeds of imagenet dogs) to cluster more closely together, or perhaps to allow for classes with different levels of dispersion? Something feels suboptimal about forcing class centers to be equidistant and identical. This isn't intended as a major criticism but it might be an interesting direction for future work.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This paper first shows some potential issues of softmax loss (i.e., cross-entropy loss with softmax function) and then propose the Max-Mahalanobis center (MMC) loss to encourge the intra-class compactness for better adversarial robustness.

The MMC loss is essentially minimizing the distance between the feature and the pre-fixed class center. Different from center loss, these centers are determined by minimizing the maximum inner product between any two class centers. Since the norm of these class centers are normalized to a constant. It is equivalent to angles. This acutally reminds me of a number of works in angular margin-based softmax loss. Just to name a few:

[1] Large-Margin Softmax Loss for Convolutional Neural Networks, ICML 2016
[2] SphereFace: Deep Hypersphere Embedding for Face Recognition, CVPR 2017
[3] Soft-margin softmax for deep classification, ICNIP 2017
[4] CosFace: Large Margin Cosine Loss for Deep Face Recognition, CVPR 2018
[5] ArcFace: Additive Angular Margin Loss for Deep Face Recognition, CVPR 2019

I think these works are closely related to what the authors aim to do, and therefore they should be discussed methodologically and compared empirically.

Besides that, I think it is also worth conducting an ablation study for how to determine these class centers. This paper considers to minimize the maximum inner product. There are a few papers listed below that explicitly discusses how to make the class centers uniformly spaced. The authors may consider to compare these methods for determining the class centers. 

[1] Learning towards Minimum Hyperspherical Energy, NeurIPS 2018 
[2] UniformFace: Learning Deep Equidistributed Representation for Face Recognition, CVPR 2019

For the experiments, the MMC loss indeed shows some advantages over the softmax loss. I am basically convinced by the experiments, although it can further strengthen the paper if the authors can conduct some evaluations on large-scale datasets like ImageNet.

I appreciate the authors provide many theoretical justifications, which is inspiring. Intuitively speaking, I can understand that shrinking the feature space (i.e., make feature distribution more compact) can improve the adversarial robustness. As a result, I think this paper is naturally motivated and is also theoretically sound. The experiments can be further improved.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper compares between SCE loss,  large-margin Gaussian Mixture (L-GM) loss and proposes the Max-Mahalanobis center (MMC) loss as an alternative to explicitly learn more structured representations and induce high-density regions in the feature space. Overall the paper is well written, with sufficient theoretical reasoning and experiments. However, the reviewer has the following concerns and questions,
The theoretical analysis depends largely on the Gaussian assumption and argues that when the loss is distributed as Gaussian, it seems to be not even a fair comparison since assuming L_{MMC} is gaussian is totally different from assuming L_{g-SCE} is Gaussian. Also in practice it is hard to justify whether certain loss function really behaves like a Gaussian distribution, which makes the application of the theorem more limited. In fact, if the samples are concentrated (which can be common in practice), is the proposed method still able to induce high density sample region?
The experiments give very competitive results for MMC loss. It would also be interesting to see if implementing other defenses or do an adversarial training would still make MMC loss much better than other loss (at least from the AT example, it seems that MMC does not perform uniformly better than SCE as before).
Are the experiment results sensitive to the choice of parameters C_MM and L?

I have read the author responses and I think they are quite solid. I have updated my score. 

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: [Post rebuttal: The rebuttal addresses most of my concerns. I have updated my score.]

This paper points out some limitation of the traditional softmax cross-entropy (SCE) loss, e.g., causing indirect and unexpected supervisory signals on the learned features so that the points with low loss values tend to spread over the space sparsely. To remedy this, the authors then proposes the max-mahalanobis center (MMC) loss. By analyzing the sample density, it is proved that the MMC loss has a sample density proportional to the number of data samples in each class. This means MMC loss induces higher feature densities than the SCE loss, thus is expected to be more robust under adverssarial attacks.

I find the paper fairly well written in general, and the arguments are well supported by the development of the theoretical results. There are, however, several questions raised by the reviewer.

One question is that by looking at the definition (8), it seems the MMC loss essentially tries to move data from different classes apart. This seems to be quite similar to the max margin loss. I would expect the authors differentiate the differences between these two. Consequently, I would also like to see empirical comparisons between these two losses.

Another question is even if you have Theorem 2, the transfer from Theorem 2 to robustness does not seem direct, e.g., how do you guarantee when the density form in eq.9 is better than that in eq.7 for robustness?

Then when looking back at the sample density definition in eq. (2). I wonder how the Vol is defined to guarantee eq.2 is a valid density function? e.g., to guarantee the integration equals to 1.

In terms of experiments, since it is claimed that the proposed loss adds little computation cost compared to the SCE loss, I think it is better to include running time comparison in the results.

The last two lines in page 8, what are unseen attacks? From the experiments, it seems that it means the PGD with different steps. I think the authors  should be careful to use the term, because some literature use the term *different attacks* to refer to attacks with different *norms*.

Second line below Table 5: it is said the MMC loss keeps state-of-the-art performance on clean data. However, by looking at Table 4, the state-of-the-art is obvious SCE. Why did you say that?

Others:
1. S_k,\tilde{k}^2 is undefined.
2. How do you solve the minmax problem below eq.8? I guess you use the algorithm in Appendix B.1? Please clarify.

review_title: Official Blind Review #1

Meta-review: This paper proposes an alternative loss function, the max-mahalanobis center loss, that is claimed to improve adversarial robustness. 

In terms of quality, the reviewers commented on the convincing experiments and theoretical results, and were happy to see the sample density analysis. 

In terms of clarity, the reviewers commented that the paper is well-written. 

The problem of adversarial robustness is relevant to the ICLR community, and the proposed approach is a novel and significant contribution in this area. 

The authors have also convincingly answered the questions of the authors and even provided new theoretical and experimental results in their final upload. 
Decision: Accept (Poster)