Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Update after author response:
I would like to thank the authors for the thoughtful response, and for addressing some of the concerns raised by the reviewers. The draft appears improved but my concerns about the novelty and interpretability of the work still stand, leading me to keep my assessment unchanged.
---------------------------

In this paper, the authors propose a general defense method against adversarial attacks by maximizing an approximate bound on the magnitude of distortion needed to force a misclassification. The authors note that this maximization can be achieved by increasing the margin between class clusters and by reducing the norm of the Jacobian of intermediate layers. Subsequently, they either directly adopt or introduce simple modifications to existing techniques to affect these two factors, showing the robustness of the combined method to several adversarial attacks on MNIST and CIFAR-10 datasets. 

As neural networks get deployed for increasingly critical applications, the issue of defense against adversarial attacks becomes progressively relevant. The paper does a good job of motivating a relatively simple approach to the problem based on an approximate bound, and pulls in from different existing methods to build a robust system. The strong points of the paper:
1. The paper is clearly written, and the approach is sensible. 
2. Fairly thorough empirical investigation under different threat models.
3. The proposed method performs consistently above the baselines for different experiments.

Here are some of my concerns:
1. The work is somewhat incremental and the novelty mostly lies in pulling a few different methods together that seem to work well in unison.
2. The two methods used for increasing the margin don’t actually optimize that objective directly. The Siamese Loss uses cosine distance as proxy and the variance reduction doesn’t guarantee increase in margin which is sensitive to outliers. Any improvement achieved thus appears to be an ill-understood side-effect. 
3. The definition of cluster distance (page 3) looks erroneous. 
4. The authors note that the proposal doesn’t work very well for a specific kind of attack (BIM) but don’t have clear recommendations for improvement. The tentative explanation of why this happens is also somewhat loose. 

In summary, I think the paper addresses an interesting problem even though the development is arguably incremental. However, since the unified approach is simple yet novel, and the results fairly promising, I am somewhat inclined to accept this paper.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: 
Summary
========
This paper proposes a defense against adversarial examples that detects perturbed inputs using kernel density estimation. The paper uses a combination of known (and often known to be broken) techniques, and does not provide a fully convincing evaluation.
I lean towards rejection of this paper.

Detailed comments
=================
The idea of increasing robustness by maximizing inter-class margins and minimizing intra-class variance is fairly natural, but the author's discussion of their approach (mainly in sections 1 and 2) is very hand-wavy and relies on a lot of general intuitions and unproven claims about neural networks.

For example, in the introduction, the authors claim:

"A trained deep classification model tends to organize instances into clusters in the embedding space, according to class labels. Classes with clusters in close proximity to one another, provide excellent opportunities for attackers to fool the model. This geometry explains the tendency of untargeted attacks to alter the label of a given image to a class adjacent in the embedding space as demonstrated in Figure 1a."

First, a t-SNE representation is just a 2D projection of high-dimensional data that is useful for visualization purposes, and one should be careful when extrapolating insights about the actual data from it. For example, distances in the 2D projection do not necessarily correspond directly to distances in the embedding space. 
The claim that untargeted attacks lead to a "nearby" cluster are hard to verify given just Figure 1. First, the colors of the labels between 1a and 1b do not seem to match (e.g., Dog is bright green in 1b but this color does not appear in 1a). If the other colors match, then this would seem to suggest that trucks (purple) often get altered to ships (orange). Yet, the two clusters are quite far apart in 1a. It seems hard to say something qualitative here. An actual experiment comparing distances in the embedding space and the tendency of untargeted attacks to move from one class to another would be helpful.
The color scheme in Figure 1b is also unclear. A color bar would help here at the very least.

These observations are then used to justify increasing cluster distance while minimizing cluster variance, but it would be nice to see a more formal argument relating these concepts to the embedding distance.

The technique proposed in Section 3.2. to reduce variance loss estimates each class' variance on each batch. Would  this still work for a dataset with a large number of classes (e.g., ImageNet)? For such a dataset, each class will be present less than once in expectation in each batch, which seems problematic.

The plots in Figure 2 don't give much of a sense of how the combination of the different proposed techniques is better than any individual technique. The evaluation compares PDM to RCE, but from Figure 2 one could guess that variance reduction alone (2c) performs very similarly to PDM (2e). An ablation study showing the contribution of each of the individual techniques would be helpful.

The evaluation section could be improved significantly. FGSM, JSMA, and to some extent BIM, are not recommended attacks for evaluating robustness. The gray-box and black-box threat model evaluations are also not the most interesting here. Instead, and following the recommendations of Carlini et al. (2019), the evaluation should:

- Propose an adaptive attack objective, tailored for the proposed defense in a white-box setting. The authors do this to some extent, by re-using the attack objective from Carlini & Wagner 2017, which targets KDE. It would still be good to provide additional explanations about how the hyperparameters for this attack were set.
- Optimize this objective using both gradient-based and gradient-free attacks
- As the proposed defense is attack-agnostic, I also suggest trying it out on rotation-translation attacks, as the worst-case attack can always be found by brute-force search

Other
=====
- The citations for adversarial training in the 2nd paragraph of the intro are unusual. Standard references here are for sure the first two below, and maybe some of the other three as is relevant to your work
    - Szegedy et al. 2013: "intriguing properties of neural networks"
    - Goodfellow et al. 2014: "Explaining and harnessing adversarial examples"
    - Kurakin et al. 2016: "Adversarial Machine Learning at Scale"
    - Madry et al. 2017: "Towards deep learning models resistant to adversarial attacks"
    - Tramer et al. 2017: "Ensemble Adversarial Training"
- The Taylor approximation in (1) does not seem to be well defined. The Jacobian of F is a matrix, so it isn't clear what evaluating that matrix at a point x means.
- The "greater yet similar" symbol (e.g., in equation (4)) should be defined formally.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: After rebuttal: my rating remains the same.
I have read other reviewers' comments and the response. Overall, the contribution of retraining and detection with previously explored kernel density is limited. 

=================
Summary: 
This paper proposes new regularization techniques to train DNNs, which after training, make the crafted adversarial examples more detectable. The general idea is to minimize the inter-class variance and maximize the intra-class distance, at some feature layer. This involves regularization terms: 1) SiameseLoss, an existing idea of contrastive learning known can increase inter-class margin; 2) reduce variance loss (RVL), a variance term on deep features, and 3) reverse cross entropy (RCE), a previously proposed term for detection purpose. The motivation behind seems intuitive and the empirical results demonstrate moderate improve in detection AUC, compared to one existing technique (e.g RCE).

My concerns:
1. The proposed technique requires retraining the networks to get a few percents of detection improvement. This is a disadvantage compared to standard detection approaches such as [1] and [2] which do not need to retain the network. I am surprised that these standard detection methods were not even mentioned at all. Retraining with fixed loss becomes problematic when the networks have to be trained using their own loss functions due to application-specific reasons. Moreover, the detection performance reported in this paper is not better than the one reported in [2] (ResNet, CIFAR-10, 95.84%) which do not need retraining.

2. There are already well-known margin-based loss functions, such as triplet loss [4], center loss [5], large-Margin softmax loss [6], and many others, which are not mentioned at all.

3. In terms of retraining-based detection, higher AUCs have been reported in [3] for a neural fingerprinting method.

4. Incorrect references to existing works. The second sentence in Intro paragraph 2: Metzen, et al, .... these are not adversarial training. Xu, et al. (feature squeezing) is not a randomization technique.

5. The "baseline" method reported in Table 2, is confusing. RCE is also a baseline? You mean conventional cross entropy (CE) training?

6. Some of the norms are not properly defined, which can be confusing in adversarial research. For example, from Equation (1) to (4). The "Frobenius norm used here" statement in Equation (3), don't know this F norm comes from.


[1] Characterizing adversarial subspaces using local intrinsic dimensionality. ICLR, 2018
[2] A simple unified framework for detecting out-of-distribution samples and adversarial attacks. NeurIPS, 2018
[3] Detecting Adversarial Examples via Neural Fingerprinting. arXiv preprint arXiv:1803.03870, 2018
[4] Facenet: A unified embedding for face recognition and clustering. CVPR, 2015.
[5] A Discriminative Feature Learning Approach for Deep Face Recognition. ECCV, 2016.
[6] Large-Margin Softmax Loss for Convolutional Neural Networks. ICML 2016.
review_title: Official Blind Review #3

Meta-review: A defense against of adversarial attacks is presented, which builds mostly on combining known methods in a novel way. While the novelty is somewhat limited, this would be fine if the results were unequivocally good and other parts of the problematic. However, reviewers were not entirely convinced by the results, and had a number of minor complaints with various parts of the paper.

In sum, this paper is not currently at a stage where it can be accepted.
Decision: Reject