Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The work explores the problem of robustness and adversarial
attacks in NN. In a multiclass prediction setting the idea
is to use a taylor expansion of a loss coined CCKL which
is the KL divergence between predictions for pairs of samples
from different classes.

The papers seems to find a convoluted route to arrive
to something like this: when the Fisher information matrix
has a strong eigenvalue the model is not robust. In other words
it says that if the landscape close to convergence has
valleys, or fast changes, the model is not robust.
This appears quite obvious and related to previous similar
studies.

This statement is then empirically evaluated on CIFAR-10.

The mathematical derivations should be made more rigorous.
For example the paragraph on Cramer-Rao bound is very handwavy.

Typos

-  is found these  ->  is found that these 
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The paper is built around the theme of adversarial attacks. The papers titled "Towards..." are usually records of failed attempts of an attack on a grand challenge. The reviewer generally believes, that such records can be of a value.
In case of this paper an alternative loss function is proposed, based on information theory. The authors claim and support their claims with graphs that this loss tracks the accuracy much more faithfully than the "standard" cross entropy. A natural question would be therefore - what happens if we use this new measure as a training loss? Would it lead to more adversarially-robust models? Should evidence for that be supported for that the reviewer would be strongly in favour of accepting the paper. In the present shape the reviewer does not see why such measure should be of an interest to community. The impact of "inspiring the community to make interesting discoveries" is in the reviewers opinion not sufficient to justify publication at a venue of such importance as ICLF.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The authors propose a new performance metric, CCKL, as an alternative to standard cross-entropy. At a high level the goal of this objective is to ensure that the classifier is confidently separating samples from different classes. They decompose this objective into its Taylor expansion terms and argue that, since adversarial vulnerability is a local property, it only depends on the first of these terms. They finally propose training models of higher capacity as a way of relying more on higher-order terms of the objective and hence improving adversarial robustness.

I found the core idea potentially interesting. It is an attempt at rigorously studying the tension between local stability (necessary for robustness) and the fact that different classes should be confidently separated. However, I have several concerns regarding the theoretical derivation and experimental evaluation.

First of all, I did not find the proposed CCKL objective well motivated. Theorem 3.1 only shows that standard training improves a _lower bound_ on the proposed objective. However, this only implies that this objective is being optimized as a byproduct of standard cross-entropy optimization. It is unclear for instance if this is a good performance indicator or whether training with this objective will even lead to a good classifier (for instance the classifier could learn a permutation of the labels instead). Therefore, I do not find the theoretical and experimental evidence in favor of studying this proposed objective convincing.

Moreover, the experimental evidence presented is quite weak. While the authors compare various quantities related to standard and robust models, they do not perform the ablations necessary to understand whether these properties are causally related to robustness of simply a byproduct of the specific training scheme.

Finally, the conclusion that larger capacity can improve both model accuracy and robustness has already been made in prior work (Kurakin et al. 2016, Madry et al. 2017). The intuition that added capacity helps by allowing the model to leverage higher order terms of the loss is interesting but a rather incremental contribution.

Overall, while the ideas presented in the paper could be of interest, they are not rigorously established and rather serve as high-level intuition at this moment. Moreover, the writing of the paper could be significantly improved. It took me quite some time to distill the above intuition despite the fact that it is a rather elementary consequence of the Taylor expansion (I am not sure what the discussion about information geometry adds to this work). I would thus encourage the authors to continue working on these ideas and their presentation. However, for now, I will have to recommend rejection.
review_title: Official Blind Review #2

Meta-review: All reviewers suggest rejection. Beyond that, the more knowledgable two have consistent questions about the motivation for using the CCKL objective. As such, the exposition of this paper, and justification of the work could use improvement, so that experienced reviewers understand the contributions of the paper.
Decision: Reject