Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: Adversarial training generalizes data-dependent spectral norm regularization

This paper shows that, projected gradient descent based adversarial training is similar to the data-dependent spectral norm regularization, and under very restrictive condition, the authors show that this two methods are the same. Some experiments are conducted to support the theory.

Overall, I think this paper is marginal, while the experiments are not convincing. First, the relation between spectral normalization and adversarial training have been investigated by [1], while the fast computational of maximum singular value with power methods have also been proposed in [1]. The authors only give a data-dependent version of the spectral normalization based on the Jacobian of the neural networks, which I think is somewhat weak. The experiments are limited with specific settings that are not generally used in practice, which alleviate my confidence on this paper’s results. Also, the experiment section contain several not so important information. I think the authors should do far more experiments to support the main claim, while move these additional justification to the appendix.

Detailed comments:
1. I think the claim of theorem 1 is somewhat ambiguous. How to guarantee there exists such epsilon satisfies this condition? Is this the case we face in the real world? What will happen if \alpha is not sufficiently large? If we don’t use logits pairing and \ell_2 norm constraint, will the claim hold? I think the correlation behinds the spectral norm and adversarial training is well investigated and use this correlation as the intuition behinds work is enough. This theorem cannot convince me that the proposed methods have a strong theoretical basis.
2. Generally, the neural networks have a large number of parameters (~ millions) for image classification task. The global spectral norm regularization only needs to calculate the spectral norm of each layer’s weight matrix, whose computational cost is acceptable. However, to calculate the Jacobian and use the power methods, we will additionally do several forward pass and backward pass just as adversarial training. As a regularization technique, is this calculation tolerable? If this is some variant of the adversarial training, I don’t find the experiment results support the claim that it will outperform the adversarial training consistantly.
3. Why don’t use some standard neural network architecture like ResNet? As this results is not comparable to other existing work, I’m not sure if this result is meaningful. Also, are the comparisons fair? For example, the regularization coefficient of global spectral norm regularization and data-dependent spectral norm regularization are far more different. And the authors use only 1 iteration to calculate the singular value in global spectral norm regularization, why to do that? Also, what’s the result compared with \ell_p norm constraint adversarial training?
4. The evaluation of some assumption on the network is better moved to appendix, as this is only some sanity check, not the core contribution. More experiments with ResNet, WideResNet, MobileNet etc. on CIFAR100 and ImageNet are more convincing.
5. What’s the attack method in the main context?
6. I think the discussion in Appendix A.5 is somewhat confusing. If the authors want to argue that the network is locally linear so that we can approximate with linear regression, why should we use the power methods?

Still, I feel the contribution of this paper is somewhat weak. I don’t see any improvements of the proposed algorithms compared with the standard adversarial training, as well as the theoretical contribution like adversarial generalization. The experiments are not convincing, as the setting is different from the general setting the community used in adversarial training. I’m not familiar with the results in global spectral normalization and it’s possible that the global spectral normalization may have little gain in adversarial robustness, but in my opinion, the main contribution [1] is the generalization analysis of spectral normalized adversarial trained neural networks, which this paper lacks. On the empirical side, the computation efficiency and performance of the proposed algorithms don’t outperform adversarial training much. So I tend to reject this paper.


[1] Farnia, Farzan, Jesse Zhang, and David Tse. "Generalizable Adversarial Training via Spectral Normalization." International Conference on Learning Representations, 2019.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This largely theretical paper establishes a theoretical link between adversarial training and operator norm regularization
for DNNs. It is well written and structured, and it falls squarely within the the remit of the conference. The experimental apparatus is thorough and the derivations, proofs and the maths at large seem sound to me, even if I have not checked them in full detail. The study delivers a data-dependent variant of spectral norm regularization affecting large singular values of the DNN. It is proved to be equivalent to adversarial training based on a type of norm-constrained projected gradient ascent attack.
Results are novel and relevant and, in my opinion, they merit acceptance.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper studies the link between adversarial training and the proposed data-dependent operator norm regularization for ReLU network. Under specific conditions, in theory, the authors show the equivalence between the l2 PGD training and the regularization method. Empirical experiments are conducted to support the theory.

While this paper gives interesting observation on both theory and empirical study, I think this paper is not qualified for publishing in ICLR due to the following reasons: (1) limited theoretical results; (2) No significant improvement for practical algorithm;

Main argument:

The main theorem 1 seems to be weak as it is only valid for small perturbation region \epsilon and it is unclear how this assumption is consistent to the practice. It is also unclear how the assumption that \alpha \to \infty influences the practical algorithm.

It would be better to generalize the theorem to other \ell_p attack, instead of just \ell_2. 

Discussion of computational complexity of the proposed regularization method compared with PGD is missed.

The adversarial robustness is related to the (local) Lipschitz continuity and many other types of regularization decreases the (local) Lipschitz constant. Could you further give result that distinguishes the proposed norm?

It would be better to give improved algorithm for adversarial training based on the current result. The current contribution for further theoretical is too weak as the main theorem requires strong assumption. And I don’t see significant contribution to empirical algorithm.


Minor
Equ (10) seems not the typical one used and seems not the one studied later.

review_title: Official Blind Review #4

Meta-review: This paper shows an theoretical equivalence between the L2 PGD adversarial training and operator norm regularization. It gives an interesting observation and support it from both theoretical arguments and practical experiments. There has been a significant discussion between the reviewers and authors. Although the authors made efforts in rebuttal, it still leaves many places to improve and clarify, especially in improving the mathematical rigor of the  proof and experiments using state-of-the-art networks. 


Decision: Reject