Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposed new regularizer for training robust models that can defend against evasion attack/adversarial examples. It looks to me there are two major novelties here, the authors suggest that 1) use dual norm on the gradient/jacobian as regularizer is tighter/better than the same norm or simply l2 norm; 2) the gradient/jacobian can be estimated by finite difference when moving the weight with a small step towards the gradient direction. 

I review the paper with a standard for empirical paper. Please kindly clarify the theoretical contributions if the authors thought the theory is novel and important. 

I donot think the empirical results are ready for publication. The final objective, when plugin finite difference into the norm regularizer, looks like logits squeezing https://openreview.net/forum?id=BJlr0j0ctX, or logits pairing https://arxiv.org/abs/1803.06373. Both methods are a little bit controversial. 

There are several issues in the experiments. Several important baselines are missing. The paper did not compare with any of the regularizer-based robust models. When considering efficiency and fast training, the authors also did not compare with recent fast method Shafahi et al. 2019 Free and Zhang et al. 2019 YOPO. 

When comparing with PGD adversarial training (Madry), in table 1, there is a more than 10% drop on robust accuracy for CIFAR-10 when \epsilon=8.

For l2 norm attack, how to interpret table 2? Why not provide accuracy under norm constraint like table 1? 

Some uncommon settings are used in the experiments such as ResNeXt-34 and attacking randomly selected 1000 images. 

Some relatively minor issues, could the authors elaborate on why the optimal value of max_v l( x+v) - c(v) is the squared dual norm of \grad l? 

Typo in title: scaleable -> scalable


========= after rebuttal =============
I thank the authors for detailed replies. I still cannot support paper because
(1) the authors emphasize the theoretical contribution and claims the bound are tighter. However, they did not directly compare with any certified robust methods, or previous bounds to support the argument. 
(2) The empirical results look suboptimal. The authors did not convince me why they sampled 1000 images for test for a small CIFAR-10 dataset. The proposed method is 10% less robust comparing to Madry's in table 1.  
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper revisits the method of gradient regularization, which regularizes the loss function by adding the norm of the input gradient, aiming to improve adversarial robustness. The standard gradient regularization is implemented with "double backpropagation", which could be time consuming for large networks. This paper proposes to replace the exact gradient by its discretization via finite difference, which is computationally more efficient comparing to "double backpropagation".  Experiments are conducted to show the effectiveness of the method in reducing training time. 

Overall, the paper is well written but the contribution is very limited and I find some of the experimental comparisons unfair.  Thus I do not support publication of the paper. Here are my detailed arguments.

1) The gradient regularization is not novel
The main contribution of the paper is to use the finite difference in the gradient regularization in order to improve the training time.  The method of gradient regularization is not new, hence the contribution is only the computational effectiveness. This would be fine if one could improve the state-of-the-art method's training time by a lot, but according to the experiments, the performance of adversarial robustness is far from the adversarial training, for example in CIFAR10 epsilon=8/255, there is a 12% drop, which is a huge gap. 

2) Unfair comparison in the experiments
In table 2, the performance of robustness with respect to the l2 norm is presented. However, in the baseline method of adversarial training, the used attack is in l_{infty} norm. This is unfair since l2 norm and l_{infty} norm has very different characteristics. Moreover, the gradient regularization uses l2 norm instead of l_{1} as in table 1, which makes the comparison unfair. 

Furthermore, the finite difference improves the training time versus the standard gradient regularization, but it does not imply that the performance will be the same. In particular, it would be better to include the performance of standard gradient regularization in table 1 and 2 as well. 

3) Comment on the motivation/theory
The theory part is fairly straightforward and it clearly shows that the norm of gradient (w.r.t input) itself is not sufficient to guarantee robustness. As a evidence, even under standard training (for example on MNIST), the gradient norm could be very small, in order of 10^{-4} but still have adversarial example with very small perturbation.  Thus, what we also need is to control how fast this gradient changes (Lipschitz constant or w-bound). However, the gradient norm regularization does not take into account how gradient changes. It is claimed that the finite discretization implicitly reduces how the gradient changes, however, I am not convinced by the argument since as long as we take h to be small, it is still a very local measure. An interesting question would be how h affect the performance and it is not discussed in the paper.  
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper provides new understandings on adversarial robustness from the perspective of input gradient regularization. Input gradient regularization hasn't been able to achieve comparable robustness to  adversarial training. Built upon existing works, this paper derives two minimum perturbation bounds (L-bound and w-bound) to explain this, as well as other defenses such as Lipschitz regularization and defensive distillation. Taking a step further, this paper proposes to use the finite difference to estimate the input gradients, which not only gives a nice property for reduced modulus of continuity (eg. the w-bound), but also makes the regulation scalable to large networks and datasets. I quite like the theoretical connections derived in this paper. Empirical evidences support their claims, and demonstrate indeed comparable robustness of input gradient regularization to adversarial training.

The empirical results can be strengthened by including the normal input gradient regularization baseline (using double backpropagation), at least on cifar-10. This is less likely to change the conclusions, but  would be interesting to see the comparisons.

Note that, there are already new progresses in adversarial training:
[1] Wang, Yisen, et al. "On the Convergence and Robustness of Adversarial Training." ICML, 2019.
[2] Zhang, Hongyang, et al. "Theoretically principled trade-off between robustness and accuracy." ICML, 2019.
[3] Carmon, Yair, et al. "Unlabeled data improves adversarial robustness." NeurIPS, 2019.
[4] Uesato, et al. "Are Labels Required for Improving Adversarial Robustness?" NeurIPS, 2019.

==========
After rebuttal:
Thanks for the new results. My rating remains the same.
review_title: Official Blind Review #3

Meta-review: (1) the authors emphasize the theoretical contribution and claims the bound are tighter. However, they did not directly compare with any certified robust methods, or previous bounds to support the argument. 
HM, not sure, need to check this

(2) The empirical results look suboptimal. The authors did not convince me why they sampled 1000 images for test for a small CIFAR-10 dataset. The proposed method is 10% less robust comparing to Madry's in table 1.  
Seems ok, understand authors response

1) The theoretical analysis are not terribly new, which is just a straightforward application of first-order Taylor expansion. This idea could be traced back to the very first paper on adversarial examples FGSM (Goodfellow et al 2014).
True

2) The novelty of the paper is to replace exact gradient (w.r.t input) by their finite difference and use it as a regularization. However, there is a misalignment between the theory and the proposed algorithm. The theory only encourages input gradient regularization, regardless to how it is evaluated, and previous studies have shown that this is not a very effective way to improve robustness. According to the experiments, the main empirical improvement comes from the finite difference implementation but the benefit of finite difference is not justified/discussed by the theory. Therefore, the empirical improvement are not supported by the theory. Authors have briefly respond to this issue in the discussion but I believe a more rigorous analysis is needed.  
This seems okay based on author response

3) Moreover, the empirical performance does not achieve state-of-the-art result. Indeed, there is a non-negligible  gap (12%) between the obtained performance and some well-known baseline. Thus the empirical contribution is also limited. 
Yea, for some cases
Decision: Reject