Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary:
It was previously observed that models that were more robust to adversarial perturbation had more interpretable jacobian. The authors attempt to train for interpretable jacobian in order to improve the robustness of the model.
This is done by employing a GAN-like procedure where a discriminator attempts to distinguish between the transformed jacobian matrix (fake images, equivalent to generator) and real images.

Experiments indicates that this improves robustness compared to unprotected models and approximately similarly to models trained with adversarial training.

Comments:
* The motivation given for this line of research is the cost of adversarial training (2nd paragraph of Section 3)
No experimental comparison is given with regards to the time it takes to train a model with adversarial training, versus the time it takes to train a model with JARN. It is also important to note that this introduces additional complexity (needs to choose an architecture for the discriminator, tune proper learning rates, etc...), which is not mentionned.

* Why not test simpler jacobian regularization method as proposed by other papers (see below). Proposition 3 of Simon-Gabriel et al. shows that results similar to adversarial training can be obtained, and they don't need several iterations like adversarial training, nor do they need to train an additional discriminator like your method.

Opinion:
The paper provides an interesting proof of concept for a method, showing that it is feasible. It however doesn't make the the case for why it is a good idea. Discussion and comparison to very significant related work is missing and experimental measurement of any advantages of the proposed method vs. adversarial training is lacking. I think that these aspects should be improved before the paper is ready for publication.

Typos:
Line 11 in Algorithm 1 -> The label is wrong, i assume it's "Update the discriminator f_disc to maximize L_adv"

Related works that needs discussing:
- Drucker, Lecun 91, "Double backpropagation increasing generalization performance" for other regularizer on the jacobian, discusses generalization rather than robustness.
- Simon-Gabriel et al., "First-order Adversarial Vulnerability of Neural Networks and Input Dimension"

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a novel regularization strategy for improving the robustness of networks to adversarial noise. A term is added to the standard supervised cross-entropy loss that encourages the Jacobian of the network to itself be interpreted as a valid image. This "regularization" term is constructed by running the input-output Jacobian of the classification network through an "adapter network" and then in turn interpreting its output as a "generator" in a GAN setup. A separate discriminator network is training to distinguish real input images from these adapter-processed input-output Jacobians. The overall regularization is the standard minimax GAN loss applied to this generator/discriminator setup. The impetus for this stems from a previous observation that salient or interpretable input-output Jacobians naturally arise for networks that have undergone adversarial training to increase robustness.

Although this whole setup seems to be a little "Rube-Goldberg"-esque, I think there's some real sensible reasons for this sort of regularization to make intuitive sense. The input-output Jacobian characterizes how much the output (i.e. the logits) are affected by small changes to the input. The Jacobian, reinterpreted as living in the input image space (as the authors do), is a map of which input pixels have the strongest effect on the output of the network. If the Jacobian image looks like the underlying input image -- in particular, highlighting the labeled object -- this indicates that changing those pixels will result in the largest change on the network output. (This should be clear when looking at Figure 4 of the paper.) On the other hand, adversarial noise by definition leaves the underlying object alone (so that a human isn't aware of the perturbation) and modifies other pixels. Models that fall for such adversarial noise will not have salient Jacobians.

This is an amusing original idea, and I think this paper probably should be accepted to ICLR -- though I don't hold that position very strongly. However, I think the most interesting point is idea of Jacobian saliency, which is from prior work (Tsipras et al., 2018) that I haven't read. Therefore, I'm not sure how significant this paper is on it's own. Regardless, I would have liked to see more discussion in the paper of why Jacobian saliency should confer robustness (as I tried to do in the paragraph above), with perhaps some additional experiments designed around understanding whether this intuition (or something similar) is actually correct. There's some discussion of the theory behind the method in section 3.1, but it's not very intuitive to the situation at hand (non-linear neural networks), and I don't find it particularly informative.

Finally, some effort is spent arguing that this method is more computational efficient than adversarial training -- I wonder if that's still true when the all the complexity of GAN training is taken into account or how to consider that point when part of the conclusion is that their method is best when it is also combined with some amount of adversarial training.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: I think the main contribution of this paper is that it introduces a new way of robust training by encouraging Jacobian saliency. Previous research like Etmann et al 2019 and Tsipras et al 2018 showed that robustness leads to saliency. But surprisingly, this paper shows the other way, saliency map can also lead to robustness, which indicates a stronger connection between these two. In general, I like the intuition behind this paper, since it introduces a new perspective of robust training.

The training method proposed in this paper is still kind of preliminary, though. I suspect that training a GAN together with the classifier will cost even more time than min-max adversarial training or some certified robust training methods. It would be great if the authors can provide the training time comparison between JARN and some state-of-the-art robust training methods. Another concern is reproducibility since the training process of GAN is sensitive to hyperparameter selection. It would be better if the author can have some discussion on the training process to show that the reported performance of the defense is easy to reproduce instead of cherry-picking.  Also, there are too many works on robustness defense that have been proven ineffective (consider the works by Carlini). Since this is a completely new way of robust training and there is no certified guarantee, I suggest the authors refer [1] to evaluate the effectiveness of the defense more thoroughly to convince the readers that it really works. Especially, evaluation under adaptive attack is necessary.

I think this is a very interesting work. But since this method is completely new, more detailed information is needed to convince me that it really works. If it does work, I believe there must exist better ways to encourage Jacobian saliency than using a GAN.

[1] Carlini, Nicholas, et al. "On evaluating adversarial robustness." arXiv preprint arXiv:1902.06705 (2019).
review_title: Official Blind Review #2

Meta-review: This paper extends previous observations (Tsipars, Etmann etc) in relations between Jacobian and robustness and directly train a model that improves robustness using Jacobians that look like images. The questions regarding computation time (suggested by two reviewers, including one of the most negative reviewers) are appropriately addressed by the authors (added experiments). Reviewers agree that the idea is novel, and some conjectured why the paper’s idea is a very sensible one. We think this paper would be an interest for ICLR readers. Please address any remaining comments from the reviewers before the final copy.

Decision: Accept (Poster)